How can we make sound replication decisions?

Returning to our previous example, consider the development of a new vaccine that is to be deployed to billions of people. Different labs from across the world have identified several promising vaccine candidates. As with any vaccine, there will be questions of efficacy as well as—potentially harmful—side effects, which will depend upon the nature of the disease and individual health factors (age, preexisting conditions, etc.). Suppose that initial trials for a candidate vaccine appear to be quite promising, showing an immune response with minimal harm. Under emergency statutes, these initial trials follow an adaptive design (22, 23) based on relatively small samples (usually of healthy volunteers). After these initial trials, crucial questions about candidate vaccine safety and efficacy remain.

This example nicely illustrates the nonepistemic value of mitigating harm. How should we consider replications of the empirical clinical trials? Obtaining consistent, accurate results in these empirical replications would be the highest priority. If the results are inconsistent, or inaccurate, the attribution of serious adverse events could be misinformed, leading to real harms for many individuals and eventually an unfavorable benefit–risk ratio. When adjudicating among multiple candidate vaccines, unreliable estimates (e.g., about efficacy and/or harms) could cause an inferior vaccine to be mass-produced. Further, conflicting reports in the scientific literature could erode trust in the population intended to receive the vaccine, complicating deployment and uptake. For these very reasons, a Book of Truths attitude would be well-aligned for a scientist to have in this case. The planned replication studies need to provide a definitive result, at the level of a fact, if that is at all possible, and the literature itself needs to reflect this clarity.

For this example, ensuring accurate results would require considerable investments of money and time. Depending upon the efficacy of the drug and/or diversity of the target population, it may require studying many participants or patients, from different demographics and possibly different cultural backgrounds, to achieve the required accuracy and certainty that benefits far outweigh the harms. If the potential harm is serious and/or common enough, then it is worthwhile to spend the resources.

Effective replication tools in this regard are large-scale replications, which coordinate efforts across multiple sites. In the medical field, randomized controlled trial data can be collected in large, multicenter collaborations that are essentially direct replications per center (many labs). More recently, such collaborations are put in place to study more than one research question in platform trials (e.g., RECOVERY in COVID-19, and STAMPEDE in prostate cancer). The goal of the participating centers is not necessarily to refute or contextualize the results from a different center, but to increase the precision of the estimates such that valid conclusions can be drawn. Apart from these top–down collaborations, bottom–up collaborations have increased in popularity where multiple randomized controlled trials are designed separately but similarly (close to exact replications) and can be jointly analyzed in a prospective meta-analysis (24–26). Top–down collaborations are usually funded in advance, and therefore limited to a certain sample size. Bottom–up collaborations have decentralized funding and can easily inspire new research teams with new funding to join the effort. These replications mitigate harm by increasing the sample size and precision of the estimates, usually with the goal of reaching a conclusion earlier (to impact patient care as early as possible), or to not let the sacrifice of the patients in the trial go to waste in an inconclusive result.^‡

There is undeniable value in expanding access to discoveries (27), promoting education (28–30), and engaging the general public in research activities (31). This brings forth the challenge of effectively communicating science to nonexperts. The scientific literature is read by a wide range of people, including politicians, policymakers, journalists, and concerned individuals with a personal stake in the science, such as health concerns.

Scientific findings are often communicated as definitive and exacting when, in reality, science is uncertain, iterative, and messy. The misunderstanding of scientific concepts and empirical results is, unfortunately, quite common, especially when involving inherently probabilistic information (32–34). This can negatively influence laypeople’s attitudes and real-life decisions, such as those related to vaccination (35). We can think about the cause of misunderstanding as a mismatch between cognitive attitudes. It is problematic if a lay audience member, say a journalist, reports on a published study assuming a Book of Truths attitude when, in fact, the available evidence is mixed and the original authors themselves viewed the study more from a Book of Conversations attitude. This mismatch might lead the journalist to overclaim a study’s result when the intent of the study was not to be a definitive statement on the phenomenon in question, but, rather, an exploration of different experimental manipulations that contributes to the scientific conversation on the topic. As such, failures to replicate can be perceived as undermining the role of science as a reliable producer of knowledge instead of appreciating that replication is a mechanism of understanding uncertainty and scientific progress (36). It should be noted, nevertheless, that the problem often originates from scientists, their institutions, and their public communication channels that overhype results and circulate press releases that make extreme claims (37–40).

Replication can help address some of these issues. In the short term, it enables a prompt and comprehensive examination of new findings, thereby enhancing their reliability, and minimizing the risk of errors (41). In this context, prioritizing the replication of results that are immediately relevant to the public, for example, due to urgent requests or the influence it will have on policies, especially if these results are highly “surprising,” could be crucial, given their potential to impact a large number of individuals. To consider a concrete example, one may wonder whether a quicker disproof of the fraudulent claim regarding a causal link between vaccines and autism may have alleviated vaccine hesitancy and the conspiracy theories that fueled it for decades. More generally, the negative consequences of public policy built upon poor, or grossly incomplete, research also relates to the nonepistemic value of mitigating harm. Replicability might also contribute to clarifying the conditions under which an effect holds (42), thereby better informing the actions of institutions responsible for translating scientific findings into practical regulations.

Communicating the nuances of replications to nonexpert audiences could be improved by the implementation of new reporting practices within peer-reviewed journals. For example, in addition to the summary of results directly usable by the general public, the level of support for the author’s conclusions based on their data could be reported, with an emphasis on contextualizing this strength of evidence with prior studies, preferably in a systematic review (10). One could also provide more detailed limitation sections in the paper and include this information in the abstract. This is particularly relevant for studies that will inform decisions that have the potential to impact a large number of individuals, such as policy decisions (e.g., refs. 43 and 44). Also, linking replications to the original research (45) could help audiences to understand whether the results are disputed, and this, in the long run, may help educate on the overall progress of science. Improving the science literacy of lay audiences is an ambitious task but could enable more informed individual and collective decision-making (46, 47).

Could the scientific publication system accommodate both a “Book of Conversations” and a “Book of Truths” attitude? This could be accomplished by an explicit, clearly visible distinction between publications that serve the goal to communicate new findings and ideas to other scientists, and publications that are intended to report a finding or conclusion that is more firmly established and ready for public consumption. For instance, Lewandowsky and Oberauer (48) proposed that original studies are published as “provisional” before having been replicated, with an invitation to other researchers to replicate it. After successful replication, the report is promoted to archival status, including the replicators as coauthors. After a failure to replicate, the publication is withdrawn and replaced by a public record of the replication failure. These records aim to reduce potential public interest and simultaneously enable reuse, for instance, by later meta-analyses. Although, as we later discuss in this dialog, it can be a highly nontrivial task to define what constitutes a successful (or failed) replication attempt. A version of this proposal is implemented by the Journal of Artificial Intelligence Research (45), where instead of replacing original studies that are not reproduced by a record of replication failure, articles that are reproduced receive a badge that states this explicitly. Also, reports describing the replications are published alongside the original articles, whether they replicated or not, to better contextualize the effects being investigated and allow for data reuse (45). Such a differentiated publication system would make the different levels of evidence for published findings more transparent for lay audiences and could help communicate to the public how science works.

Public credibility and trust is a value worth considering when making replication decisions. Acting responsibly as researchers—which involves, among other things, conducting original research with transparency, honesty, and accountability—plays a crucial role in the public’s perception of our credibility as a community. It follows, then, that decisions regarding replications, should be made responsibly, with an eye to continuing to earn (or, for some areas, to earn back, in the wake of replication concerns, see refs. 6, 49, and 50) and maintain the public’s trust. Increasing public credibility naturally aligns with a Book of Truths attitude, with its focus on establishing repeatable findings, which helps establish public trust. A person holding the Book of Conversations attitude is less concerned with what the public finds credible, as the literature is viewed primarily as a conversation among scientists themselves (insiders). This can become problematic if the knowledge gained from studies is not clearly conveyed to the appropriate audiences.

One useful tool that can help improve public credibility is the registered replication report format (51). When replication studies are conducted as a registered report, the plans for the study, including sampling and analysis strategies, are registered and peer-reviewed before the study is conducted. Although the registered replication report format does not guarantee this, it can improve the likelihood that a replication study will be methodologically sound, and that researcher degrees of freedom and bias will be limited. When coupled with prospective meta-analysis designs, this can yield more accurate estimates of effect sizes, which can help combat the “winner’s curse,” where initial studies report inflated effect sizes, only to have those effect size estimates dramatically lowered upon replication (52). More consistent, accurate effect size estimates can help build credibility and trust with the public.

Transparency and communication with other scientists is a nonepistemic value that aligns with both of our cognitive attitudes, but for different reasons. From a Book of Truths perspective, transparency enables direct replication by providing scientists with all necessary experimental details. From a Book of Conversations attitude, transparency and communication to other scientists is the primary role of the peer-reviewed literature.

Providing transparency and accurately communicating all relevant aspects of a study is integral for replication efforts, even more so when real benefits and harms are at stake. The Reproducibility Project: Cancer Biology (53–55) demonstrated major difficulties in replicating preclinical research. A contributing factor to this was that methods were insufficiently documented for the majority of studies, and they simply could not be completed. Additionally, reanalyses in the biomedical sciences have shown failed outcome reproducibility (i.e., the original results could not be confirmed by rerunning the original analysis scripts on the original data) and occasionally major differences between published and reanalyzed results for pivotal commercial trials (56), likely due to insufficient transparency in analysis plans and methods.

There are two main components of replication to consider here. The first is to be able to recreate the study itself and successfully rerun all the analyses. This requires not only an adequate, detailed description of the methods, but also the sharing of the raw data and of the specific code used in any and all reported analyses. The second component is to be able to use the presented information on methods, including all the processes undertaken and the analyses conducted, in order to design one or more new studies that use the same methods, processes, and analytical pathways. Regardless, the ability to meaningfully perform exact, similar, or diverse replications depends on the ability to understand what happened in the original study.

Notions of transparency and reproducibility can lie on a continuum. Within the field of computational science, Peng (57) describes a “reproducibility spectrum” where, at one end, the publication contains no code at all, and at the other, everything is provided to fully reproduce the results, which includes all of the code, data, linked executables and dependencies, etc. The reproducibility study type can be distinguished based on which documentation it relies on, such as the article describing the original study, code, and data, while the degree to which the study is reproduced can be characterized by whether the result obtained is exactly the same, different but leads to the same conclusions, or if an alternative analysis leads to the same conclusion (58). All else equal, the more data and code that are available and easy to use, the better for reproducibility purposes (59). Data and code sharing is also indispensable for evaluating new models under alternative specifications.

Yet, making results fully reproducible^§ is not without costs. Within the field of computer science, making computational experiments available can be as easy as creating a Docker container (60) (i.e., an executable package, including the code, system tools, dependencies, and settings necessary to run the experiment), but producing quality software that is open-source, fully documented, and can be reused and extended is a time-consuming and costly endeavor. It is not only time-consuming to produce code, the hours spent maintaining it and answering questions regarding documentation can prevent the scientists from working on new research. Costs in the form of money could also be huge. Studying the emergent behaviors of large language models (61) requires access to the largest models, which currently are the commercial ones and cost more than 100 million United States dollars to train. It is worth mentioning that these commercial models are neither open nor transparent.

In light of these potential costs, one perspective that scientists can take is that “natural selection” will determine which results are important enough to warrant the additional costs required for full reproducibility. Using computer science as an example, if a computational method is shown to be effective, naturally, improved versions of the code will be developed and released, typically by other labs. On the flip side, requiring this effort for the initial publication may hinder scientific progress, in particular, for the long tail of scientists with limited resources. The idea is to require transparency in what was done, but not necessarily full reproducibility. This perspective aligns well with a Book of Conversations attitude, as the literature is viewed as a transparent exchange of promising ideas among scientists, with the understanding that not all research results are expected to be fully reproducible at face value.

Moving the computer science literature in a direction where all computational products are available and fully reproducible would require considerable time and effort. Consider, for example, the Association for Computing Machinery Special Interest Group on Management of Data Conference Reproducibility Evaluation effort in which authors of accepted papers were invited to submit their computational experiments for evaluation of whether all results (e.g., figures, tables) in the paper could be easily reproduced from the code and data that were made available. Over the past 15 y of this program, only around 25 to 30% of the authors submitted their experiments for evaluation. In contrast, by lowering the bar and requesting authors to just make their code and data available, over 65% of the accepted papers made their artifacts available at Very Large Data Bases Endowment 2021 (62). A similar effort at the NeurIPS conference increased code sharing from below 50% to nearly 75% in one year (63). Making the code available, though, might not be enough, as a study found that less than 50% of code was executable even after communicating with the original authors (64).

Regardless of how we view these reform policies, there are now many tools, platforms, and repositories that can help in methodological documentation and transparency. For example, in the biological and biomedical sciences, decades of commitment have led to major advances in reproducibility practices. This commitment has spurred a multifaceted effort to improve methods and infrastructure and encompass reporting guidelines (e.g., refs. 65, 66, 67), or highly standardized and specialized data repositories (e.g., ref. 68).

Theory development is the endeavor by scientists to find explanations for empirical phenomena—empirical regularities that can be observed repeatedly across time and situations. By “explanation,” we usually mean some set of theoretical assumptions that, if true, render the phenomena in question substantially more likely than if they were not true. To be worth explaining, a phenomenon must not only be stable over time—as demonstrable by direct replication under nearly identical conditions—but also general across many situations (69), and robust across several different methods for observing it (70). For example, theories of memory aim to explain the shape of the forgetting curve (71) because it is observed across many kinds of memory contents, types of memory tests, and person populations. A theorist aiming to find an explanation for a phenomenon needs to know as much as possible about its scope of generality because a candidate explanation needs a corresponding scope. It follows that, to advance theory development, empirical research needs to prioritize establishing the generality of an empirical regularity through conceptual replication as much as establishing its robustness through direct replication (69, 72).

The epistemic concerns that relate to the development of theory derive in large part from those associated with the underlying phenomena—a theoretical framework is not made better if it explains phenomena that are fragile to variations in the experimental design or study sample. From this perspective, a Book of Truths attitude more naturally aligns here as a theory is difficult to develop if one does not know for certain what phenomena to explain. Publication bias presents a major threat to establishing phenomena and its boundary conditions. The occurrence of publication bias may affect theory development in two ways: first in the establishment of explananda, and second, in the testing of new predictions against the empirical literature. If a theory predicts a phenomenon not usually considered in a given context, and experiments so far have failed to detect that phenomenon, but publication of these null results have been suppressed, then false theories may remain in the literature unchallenged; it may even be buttressed when false positive results occur later. Replication tools intended to limit or eliminate publication bias naturally align here, such as registered replication reports (51).

However, Feest (73) challenges the usefulness of direct and conceptual replications by arguing that they offer very limited information about the nature of an effect to be explained. She argues that for research paradigms that are of limited “conceptual scope,” that is, the causal relationships between the independent and dependent variables are not well understood, it becomes nearly impossible to know what variables to hold constant (for a direct replication) or systematically manipulate (for a conceptual replication)—see also refs. 74 and 75. The crux of the problem is that if we carry out a replication of either kind and observe the “same” effect, we have no idea whether it truly is the same effect being observed or whether it is something completely different. We often do not know which experimental variables are core to the effect and which are auxiliary. In other words, the effect of an independent variable on a dependent variable is often confounded by interactions with background factors, either held constant in the replication study or varied in ways not accounted for by the researchers, leading to aggregation fallacies (e.g., ref. 76). Feest illustrates her points with the “Mozart effect”—a result first reported by Rauscher at al. (77) whereby listening to a Mozart sonata temporarily improved participants’ scores on a spatial reasoning test. A considerable amount of subsequent empirical work identified that: i) this temporary improvement was much smaller than originally identified (78) and ii) the effect was attributable to the arousal in mood one may experience when listening to upbeat music, but the stimulus need not be musical at all. Both the sonata, specifically, and the music itself, more generally, played an auxiliary role to the effect.^¶

Feest (73) proposes to solve this problem by considering experiments designed to evaluate effects not as replications per se, but as systematic “explorations” of the variable space. This perspective better aligns with a Book of Conversations attitude as the focus shifts from establishing the “truth” of an effect via direct replication to building out a series of results that systematically inform us about the behavior of various dependent and independent variables, a notion echoed by ref. 80. One could debate whether this is conceptual replication in a different guise, but the intent and subsequent interpretation are perhaps different (for thoughtful discussions, see also refs. 81–86). While compatible with multiple values and attitudes, metastudies (87), which systematically randomize the values of multiple independent variables when designing experiments, are particularly well-suited here. Modeling frameworks that help delineate the logical structure of replication experiments would also be beneficial (88).

We need to ask which study promises a larger gain of information for our research question when prioritizing between two empirical studies that have about the same cost. If the aim is to establish a phenomenon, we should ask how much information we gain by corroborating or debunking the hypothesis that it is real. On the other hand, if the aim is to test a theory, we should instead gauge the information gained from a study of the credibility of the theory. The two aims can lead to different evaluations. For establishing a phenomenon, the choice is between a direct replication of a first study supporting the hypothesis, or a conceptual replication assessing its generality across some dimensions. For theory testing, the choices are different. A good theory predicts not only a single phenomenon but several phenomena, and therefore offers multiple avenues for testing the theory. After researchers have run a first study yielding support for one prediction of the theory of interest, a second study testing another prediction of the theory often promises a larger information gain than a direct or a conceptual replication of the first study (72). However, see Davis-Stober and Regenwetter (89) and Heck (90) for an argument regarding the inherent challenges of interpreting evidence aggregated across studies that each test different predictions of a theory.

Theories are typically grounded in multiple sources of evidence and are intended to account for various phenomena. Therefore, the direct replication of a single experiment, even if successful, does not necessarily offer strong support for the overall theory. Converging evidence obtained through different methodologies, and sometimes by different research teams (i.e., triangulation), may be more informative, especially when considering different predictions of the theory. Specifically, if diverse pieces of evidence confirm various predictions of the theory, the theory is strongly supported. The more diverse these pieces of evidence are—either with regard to their content or the methodology that has produced them—the stronger the support for the theory when they converge. Even unsuccessful replications can be informative because they might contribute to a better understanding of the theory. However, one should worry that allegiance and publication biases may generate a literature of reported scientific findings that all seem to support a theory, but the entire literature may be spurious. Even the most squarely refuted findings continue sometimes to be heavily cited, often without attention paid to the refutations or with excuses from the supporters of the theory who are unwilling to let go (91, 92).

The aim of testing a theory has implications for the role of preregistration. Oberauer and Lewandowsky (72) distinguish between preregistration of hypotheses and of analysis plans, which serve different aims. Preregistration of hypotheses is advocated as a means to distinguish between a priori predictions and post hoc interpretations of findings. When a study aims to test a strong theory that unambiguously predicts a particular outcome, then the theory itself already serves the role of preregistering the hypotheses. In the best case, the theory is a formal model that we can run to compute the predictions for a study. With strong theories, the preregistration document merely serves to document the hypotheses that follow from the theories. When the theory to be tested is weak, so that what it predicts depends strongly on rather arbitrary auxiliary assumptions, then preregistration can serve to make the auxiliary assumptions explicit. Preregistering such assumptions, however, does not make them less arbitrary. Therefore, in the case of weak theories, empirical confirmation of preregistered hypotheses does not provide more support for the theory than confirmation of an alternative, not preregistered hypothesis that could be derived from the theory with different, equally arbitrary auxiliary assumptions. Hence, preregistration of hypotheses is a good opportunity to make theoretical assumptions explicit, but it adds nothing to the conclusions we can draw from the results of a study for the theories we aim to test by it (72).

A second aspect of preregistration is to commit to an analysis plan before seeing the data. This serves to reign in problematic research practices such as p-hacking, where many analysis paths are tested but only those that show support for a theory, or are otherwise desirable for the researchers, are reported in a paper (93). Usually, showing an effect is more informative, and more publishable, than not showing it, and therefore, p-hacking is likely to artificially inflate the size of reported effects. Preregistration of an analysis path may help to reduce researchers’ freedom to report cherry-picked results from a large number of analysis paths. Evidence that this is effective comes from a study of published effect sizes in psychology (94), which showed that studies with preregistration had considerably smaller effect sizes than studies without preregistration.

For the aim of theory testing, preregistration is probably most effective in the form of a registered report: Study design and method, hypotheses to be tested, and analysis plan are not only registered publicly but submitted for peer review. After passing review, the publication outlet commits to publishing the study regardless of whether the outcome supports or challenges the theory. This provides an opportunity for proponents and opponents of the tested theory to agree on reasonable auxiliary assumptions for deriving predictions, on a diagnostic design, and adequate analysis methods. Such a negotiation reduces not only the freedom of the research team to choose the auxiliary assumptions and analytical paths that suit their goals but also the freedom of their opponents to dismiss the evidence. In some areas of medicine, such as clinical trials, there are already many thousands of protocols that have been published upfront in peer-reviewed journals and there are also large numbers of detailed statistical analysis plans that are published upfront before a trial is run (95–98). Yet, even with detailed prespecified analysis plans, deviations are very common (99).

C.P.D.-S., A.S., B.A., S.H.C., T.M.E., S.M.F., A.F., J.F., J.P.A.I., K.O., F.P., S.R., D.J.S., J.t.S., K.T., D.v.R., J.V., and O.E.G. wrote the paper.

The authors declare no competing interest.