Microparadigms: Chains of collective reasoning in publications about molecular interactions
 *Department of Biomedical Informatics,
 ^{†}Columbia Genome Center, and
 ^{‡}Center for Computational Biology and Bioinformatics, Columbia University, New York, NY 10032;
 ^{¶}Department of Statistics, Columbia University, New York, NY 10027; and
 ^{‖}Department of Genetics, Yale University, New Haven, CT 06520
See allHide authors and affiliations

Communicated by Sherman M. Weissman, Yale University School of Medicine, New Haven, CT, January 23, 2006 (received for review August 15, 2005)
Abstract
We analyzed a very large set of molecular interactions that had been derived automatically from biological texts. We found that published statements, regardless of their verity, tend to interfere with interpretation of the subsequent experiments and, therefore, can act as scientific “microparadigms,” similar to dominant scientific theories [Kuhn, T. S. (1996) The Structure of Scientific Revolutions (Univ. Chicago Press, Chicago)]. Using statistical tools, we measured the strength of the influence of a single published statement on subsequent interpretations. We call these measured values the momentums of the published statements and treat separately the majority and minority of conflicting statements about the same molecular event. Our results indicate that, when building biological models based on published experimental data, we may have to treat the data as highly dependentordered sequences of statements (i.e., chains of collective reasoning) rather than unordered and independent experimental observations. Furthermore, our computations indicate that our data set can be interpreted in two very different ways (two “alternative universes”): one is an “optimists’ universe” with a very low incidence of false results (<5%), and another is a “pessimists’ universe” with an extraordinarily high rate of false results (>90%). Our computations deem highly unlikely any milder intermediate explanation between these two extremes.
More than 5 million biomedical research and review articles have been published in the last 10 years. Automated analysis and synthesis of the knowledge locked in this literature has emerged as a major challenge in computational biology. Recent advances in automated text analysis have provided an opportunity for collecting and scrutinizing huge collections of published statements, offering a unique and previously inaccessible “bird’s eye” view of a large field. Among others, the GeneWays textmining project (1–3) recently made available for analysis millions of biological statements extracted from 78 contemporary research journals. By developing computational tools that allowed detailed statistical analysis of millions of statements extracted from scientific texts, we used these unique data to probe the largescale properties of the scientific knowledgeproduction process. We explicitly modeled both the generation of experimental results and the experimenters’ interpretation of their results and found that previously published statements, regardless of whether they are subsequently shown to be true or false, can have a profound effect on interpretations of further experiments and the probability that a scientific community would converge to a correct conclusion.
In this study, we focused on chronologically ordered chains of statements about published molecular interactions, such as “protein A activates gene B” or “small molecule C binds protein D.” Each chain comprises chronologically ordered positive and/or negative statements about the same pair of molecules; for brevity, we encode each such chain with a series of 0’s for the negative statements, and 1’s for the positive statements. For example, an imaginary chain of length 3 could include “protein A activates protein B” (1), “protein A does not activate protein B” (0) and “protein A activates protein B” (1) (see also Fig. 1). Discrepancies across published statements may arise because of variations in experimental conditions, errors in the conduct of the experiment, misinterpretation of results, or a combinations of these factors.
There is a well established term in economics, “information cascade” (4), which represents a special form of a collectivereasoning chain that degenerates into repetition of the same statement (4). Here we suggest a model that can generate a rich spectrum of patterns of published statements, including information cascades. We then explore patterns that occur in real scientific publications and compare them to this model.
Results and Discussion
Modeling Experiments and Publication Process.
There are numerous possible ways to evaluate dependencies across published statements. The simplest approach is to evaluate a correlation between the chronologically consecutive statements within the same chain of reasoning. We started with the simple correlation analysis and observed an overwhelmingly strong dependence between statements within a chain (correlation coefficient is 0.9857, with 842,720 pairs of neighboring statements studied; the corresponding P value is 0 for all practical purposes). However, this simple analysis is very hard to interpret, because the strong correlation across statements in a chain can be due to a number of different factors: because all statements within a chain have the same “true” value, because biological experiments have a very low error rate, because most of the published statements are not experimentally supported but are restatements of earlier statements, because published statements have large “momentums” that affect interpretation of later experiments, or because the general likelihood of randomly formulating a correct statement is high (if most of the molecules were capable of interacting in some way, most of the random statements of the form “A interacts with B” would be correct). Therefore, we designed a probabilistic model that allowed us to discriminate contributions of these factors while being simple and computationally tractable (see Model Box and Fig. 2 for explanation of the model assumptions and its application; see also Supporting Text, which is published assupporting information on the PNAS web site, for further indepth mathematical details). One of the benefits of formulating our model in probabilistic terms is the ability to quantify our confidence in the results of the analysis, given the model and its assumptions.
Patterns.
Our model can represent a rich spectrum of possible patterns in chains of published statements (see Fig. 3): all patterns that we propose here occur in the real chains, albeit some of them are rarer than others. The plots on the left side of Fig. 3 represent sequences of simulated positive and negative statements, viewed as white and gray cells, respectively; plots on the right side of the figure show the probability of reaching the correct answer (which is, of course, known only in a simulation) at each step of a reasoning chain for the chains shown to their left. All five patterns are characterized by distinct distributions of singledigit run lengths; for example, the runs of single digits are on average longest for the third pattern (Fig. 3 E and F) and shortest for the fourth pattern (Fig. 3 G and H).
The first pattern, shown in Fig. 3 A and B, corresponds to a complete statistical independence of published statements (“trust nobody” but yourself): scientists in this imaginary world do not read one another’s papers (momentums of all published statements are zero), and prior publications produce no bias in interpretation of experiments by a scientist. The probability of publishing a correct statement in this case is the same for all links in a reasoning chain (Fig. 3 B).
The second, third, and fourth patterns (Fig. 3 C and D, E and F, and G and H, respectively) illustrate three possible modes of dependence within a single reasoning chain. The third and fourth patterns correspond to extreme conformism (the superconformism pattern, indicating high concordance with the majority of statements), and anticonformism (the superanticonformism pattern, indicating a tendency to disagree with the majority of statements), respectively. Both patterns can result from published statements having large momentums: If the majority statements are heavier that the minority statements, the model produces the extreme conformism pattern, whereas if the minority statements are heavier, the resulting pattern is anticonformism. The superconformism pattern (Fig. 3 E and F) is a perfect example of an information cascade. Another pattern (anticonformism with an inferiority complex; Fig. 3 C and D) is a curious hybrid between the conformism and anticonformism patterns: The scientists in this hypothetical universe tend to follow the majority of published statements as long as there are no conflicts, but once the first conflicting statement is published, the same scientists tend to follow an anticonformist model, by joining the minority opinion and generating a stutterlike publication signature.
We call the fifth pattern, shown in Fig. 3 I and J, mild skepticism: The published statements here are dependent, because they have small, but positive, momentums, and the majority statements are heavier than the minority statements. This dependence is manifested by runs of zeros and ones longer than those observed in the independent model (Fig. 3 A and B), but the dependence is relatively weak. In this hypothetical world, scientists do read their peers’ articles and try to compare their own results to the published ones but tend to trust their own data more than the data published by their peers.
Patterns that resemble the mild skepticism were prevalent in our realworld data set (described below), but analysis revealed the presence of all five hypothetical patterns.
Data Analysis.
To estimate the momentums of published statements, we applied our computational tools to data stored in the geneways 6.0 database (3). To detect possible variations in behavior of statements about different types of molecular interactions, we divided interactions in the large data set into logical interactions (such as activate, regulate, and inhibit) and physical interactions (such as bind, phosphorylate, and methylate). This subdivision resulted in three distinct data sets: (i) the whole data set (all), and (ii) logical and (iii) physical interaction subsets.
Our first observation, based on computation, was that, because of the huge data set, we can clearly demonstrate that momentums of published statements are notably positive, but <0.1 (see Fig. 4 A and B). This result means that scientists are often strongly affected by prior publications in interpreting their own experimental data, while weighting their own private results (which have weight 1 under our model) at least 10fold as high as a single result published by somebody else.
The second observation was that, for all three data sets, the dominant statements were considerably “heavier” than the nondominant statements, revealing a tendency toward conformism (see Fig. 4; see also Appendix 1 and Data Sets 1 and 2, which are published as supporting information on the PNAS web site).
Our third and most striking finding emerged from the need to explain the observation that the published statements in our data set are predominantly positive (<5% of them are negative) and are highly correlated within chains. The estimated momentums of published statements are too small to wholly account for the high correlation, and a mechanical republication of the old statements (without experimental reevaluation) appears to be insufficient to explain the trend either (see Model Box and Fig. 4). Our stochastic analysis of the real data produced not a single, most likely explanation, but rather two sets of nearly equally probable “alternative universes” (see Fig. 4 C–E). These statistically derived “universes” reflect a conclusion that perhaps can be reached through a commonsense logical reasoning (such a derivation, however, would lack quantification of the confidence). One explanation is that the high agreement among published statements is due to a very low rate of experimental errors (optimists’ universe, where both falsepositive and falsenegative error rates are <0.05) and an overwhelming predominance of positive statements over negative ones among true statements. The alternative explanation posits a pessimists’ universe that is characterized by exceptionally errorprone experiments; both falsepositive and falsenegative error rates are significantly >0.9, and a randomly chosen positive statement is far more likely to be false than true. The statistical tools allow us to conclude that intermediate milder universes are very unlikely under our assumptions (see estimated marginal posterior distributions of parameter values in Fig. 4) and that both universes enjoy considerable support by data (see Fig. 5). This ambiguity is not due to the model’s weakness, but due to the lack of information about the actual proportion of positive versus negative statements that exist. In fact, our datashuffling experiments (described in detail in Supporting Text) showed that the twouniverse effect disappears for many types of reshuffled data. Furthermore, our model parameter estimates are sensitive to data randomization and to elimination of constant (singledigit) chains or variable (doubledigit) chains (see Supporting Text).
Optimum Parameter Values.
Our probabilistic model allows us to find optimum parameter values that maximize the probability that a given chain of scientific reasoning will converge to correct result. An evaluation of the optimum parameters under our model (see Model Box) indicated that the momentums of published statements estimated from real data are too high to maximize the probability of reaching the correct result at the end of a chain. This finding suggests that the scientific process may not maximize the overall probability that the result published at the end of a chain of reasoning will be correct.
A detailed analysis of a measure leading to improved probability of publishing correct results is outside of the focus of this study, but experience in the fields of physics (5) and structural biology (6) offers concrete steps (such as random and independent benchmarking of published results) that provide scientists with feedback about the true distribution of experimental errors. Another major question also remains open: In which of the two alternative universes discovered in our analysis are we living? Our results indicate that the optimistic and pessimistic realities are almost equally likely given currently available data.
Evaluating the quality of the published facts is more than a matter of pure academic curiosity: If the problem of convergence to a false “accepted” scientific result is indeed frequent, it might be important to focus on alleviating it through restructuring the publication process or introducing a means of independent benchmarking of published results.
Model Box
Our model is built on eight simple and intuitive assumptions. First, we assume that for every pair of substances, there is a general truth or rule: These substances either usually do or usually do not interact. The odds of encountering a negative rule (“A usually does not interact with B”) are not necessarily the same as the odds of encountering a positive rule (“C usually does interact with D”); we denote the corresponding probabilities by 1 − ρ and ρ, respectively. Second, each general rule may have an exception, with probability φ (e.g., proteins A and B interact in most cases, but do not interact when in tissue X). Third, we allow experiments to produce erroneous results: They produce falsenegative results with probability ν and falsepositive results with probability μ. Fourth, we assume an asymmetry in terms of ease of publication between negative and positive experimental results. Many experimentalists believe that it is more difficult to publish a negative result (“we were unable to demonstrate that A and B interact”) than to publish a positive result (“we demonstrated that A and B interact”), so the model allows negative results to be discarded, without publication, with probability 1 − η. Fifth, we assume that a published statement can be based on original experiments (with probability 1 − β_{i}) or can be a restatement of an earlier published statement (with probability β_{i}). We tested two formulations of the model: The simpler version assumes that β_{i} is constant, whereas, in the more complicated version of the model, β_{i} is increasing as the chain grows longer: β_{i} = 1 − i ^{−ψ}, ψ > 0. The more complicated formulation asserts that the chances that a scientist would experimentally reverify an old statement drop with the growth of the available evidence. We assume that the first statement in every chain is always supported by an experiment. Sixth, we allow an experimenter’s interpretation of her own data (and hence of her published result) to differ from the “unbiased” interpretation of the same data that an expert would have in the absence of prior publications. This model feature reflects our observation that, when reading about published experiments similar to their own, scientists build in their minds an equivalent of statistical prior distributions of experimental outcomes that they are using for interpreting their own experimental data. We assume that each published statement has a weight that is different for statements in reasoning chains where they are in the majority (α), are the minority (ι), and are of equal number (τ). For example, for the chain of reasoning 1, 0, 0, 1, 1, 1, every published positive statement would have weight α (because it is in the majority), whereas each negative statement would have weight ι. For the hypothetical chain 0, 1, 0, 1, the weight of each the statement would be equal (τ), because there are an equal number of zeros and ones. The weight of each published statement is nonnegative and reflects the importance of published statements in influencing both a researcher’s choice of experiments (and thus ultimately observed results) and her interpretation of the results. We set the subjective weight of the researcher’s own experiment to 1. Seventh, we assume that relationship among statements related to the same molecular interaction is adequately represented with a linear structure (a chain). Eighth, we assume that different chains are statistically independent.
In our model (Fig. 2 A), each chain of reasoning results from a combination of two processes: one determines the length of the chain, whereas the other specifies the arrangement of zeros and ones within the chain given that length. The first process is described in detail in Supporting Text. The second process (see Fig. 2 B and C) is responsible for generating a specific sequence of zeros and ones within a chain of a given length. In this study, we emphasize analysis of the second process.
Note that in our model, experimental data cannot be observed directly by the research community. They have to be inferred from publications. For that reason, we call the experimental results “hidden” (see Fig. 2 B and C) by analogy with the hidden states in the hidden Markov models.
To estimate the marginal posterior distributions for the parameters, we used the Metropoliscoupled Markov chain Monte Carlo technique (refs. 7 and 8; our implementation of the algorithm closely followed that of Altekar and colleagues, ref. 9), run on a cluster of 40 Intel processors. The value of parameter τ for these computations was assumed to be equal to the average of α and ι and the values of φ and η to 0 and 1, respectively. This assumption did not affect our findings regarding values of the other parameters, yet it greatly reduced computational complexity. Using the Metropoliscoupled Markov chain Monte Carlo technique, we estimated a full posterior distribution of parameters given data, P(parameters  data). We then divided the whole space of the permissible parameter values into “bad” and “good” neighborhoods [the error rate is very high in the bad neighborhood (>0.5) and low in the good neighborhood (<0.5)] and computed the posterior probabilities that the parameter values belong to each neighborhood. Our estimates of the marginal posterior distributions for major parameters are shown in Fig. 4. As long as in our computation we assumed noninformative prior parameter distributions, the mode of each estimated marginal density corresponds to the maximumlikelihood estimate of the parameter value; a narrow peak indicates a high degree of certainty in the estimate, whereas a wide peak indicates that the variance of the estimate is large.
The data set that we used for analysis included 2.5 million reasoning chains containing 3.3 million individual statements extracted from the geneways 6.0 database (1–3). We did our data analysis in two ways. In one version of the analysis, we used only one (most frequent) statement of each kind per article, whereas in the other version of analysis, we used all statements available in the database. The results of the two analyses are qualitatively indistinguishable, and we show here only results of the analysis of the former type.
Analysis of all three data sets under the constantβ model produced consistent estimates of β with the posterior mean close to 0.2 (the 95% credible interval for the largest dataset, all, was bound by 0.166 and 0.235). To determine whether this simple way of estimating restatements was reasonable, we manually analyzed 200 statements about molecular interactions in Drosophila melanogaster (these statements were randomly sampled from the flyspecific portion of the geneways 6.0 database). Among these 200 statements, 107 were based on original experiments, which gave us an estimate of β equal to 0.465 with a 95% confidence interval (0.394 and 0.637; see Supporting Text). Therefore, the simple constantβ model was falsified with the data. We were then able to estimate the value of the decay parameter (ψ) by using the manually collected data (ψ was equal to 0.445, 0.479, and 0.502 for datasets all, logical, and physical, respectively). Notably, however, when we compared estimation results under the simpler constantβ model with those under the more complicated decayingβ model, all of the major results reported here held for both computations, demonstrating the robustness of our model to estimates of mechanical restatements.
The probabilistic approach to analysis of data naturally allowed us to compare directly the relative plausibility of each alternative universe (by estimating the latter’s posterior probability, see Fig. 5). To define the bounds of universes, we divided the parameter space into eight equalsized subspaces, and we estimated the proportion of the posterior density associated with each subspace. [The subspaces were separated by three mutually orthogonal planes cutting axes (ρ, μ, ν) at ρ = 0.5, μ = 0.5, and ν = 0.5, respectively.] Because the number of informative (cold) Metropoliscoupled Markov chain Monte Carlo technique iterations in our analysis was enormous (3 × 10^{6}), the differences between posterior probabilities of the two universes are statistically significant.
We can draw several conclusions from this analysis of the relative plausibility of the alternative universes. For the largest combined data set (all), the most likely universe was the pessimists’ (posterior probability 0.73), followed by the optimists’ (posterior probability 0.27). A very similar picture is observed for the smaller data sets (Fig. 5), but for all practical purposes, both universes successfully explained reality.
Optimum Parameter Values.
Assuming that both experimental error rates (falsenegative and falsepositive) do not exceed 0.5, the optimum value for parameter ι is zero, whereas the optimum value of α depends on the length of the chain. The optimum values of α are close to 0.39, 0.29, and 0.21 for chains of length 3, 6, and 9, respectively (data not shown). As the chain grows longer, the optimum value of α becomes progressively smaller.
Supporting Information.
For more information, see Figs. 6–23, which are published as supporting information on the PNAS web site.
Figures and Tables
Acknowledgments
We thank Lynn Caporale, Murat Cokol, Lyn Dupré, Michael Krauthammer, Ani Nenkova, Paul Pavlidis, Valerie Reinke, James J. Russo, Rita Rzhetsky, and Tian Zheng for comments on the earlier version of the manuscript; Dennis Vitkup for suggesting the term “momentum of a statement”; and Ahmet Sinav for the artwork. This study was supported by grants from the National Institutes of Health (to A.R. and K.P.W.), the National Science Foundation, the Department of Energy, the Cure Autism Now Foundation, and the Defense Advanced Research Projects Agency (to A.R.), and the W. M. Keck Foundation (to K.P.W.).
Footnotes
 ^{§}To whom correspondence should be addressed. Email: andrey.rzhetsky{at}dbmi.columbia.edu

Author contributions: A.R. and K.P.W. designed research; A.R. and I.I. performed research; A.R., I.I., and J.M.L. analyzed data; and A.R. and K.P.W. wrote the paper.

Conflict of interest statement: No conflicts declared.

Freely available online through the PNAS open access option.
 © 2006 by The National Academy of Sciences of the USA
References
 ↵
 ↵
 ↵

↵
 Anderson L. R. ,
 Holt C. A.
 ↵

↵
 Venclovas C. ,
 Zemla A. ,
 Fidelis K. ,
 Moult J.

↵
 Gilks W. R. ,
 Richardson S. ,
 Spiegelhalter D. J.

↵
 Geyer C. J.
 Keramidas E. M.

↵
 Altekar G. ,
 Dwarkadas S. ,
 Huelsenbeck J. P. ,
 Ronquist F.