Skip to main content
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Accessibility Statement
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian
  • Log in
  • My Cart

Main menu

  • Home
  • Articles
    • Current
    • Special Feature Articles - Most Recent
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • List of Issues
  • Front Matter
  • News
    • For the Press
    • This Week In PNAS
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Editorial and Journal Policies
    • Submission Procedures
    • Fees and Licenses
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Accessibility Statement
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian

User menu

  • Log in
  • My Cart

Search

  • Advanced search
Home
Home

Advanced Search

  • Home
  • Articles
    • Current
    • Special Feature Articles - Most Recent
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • List of Issues
  • Front Matter
  • News
    • For the Press
    • This Week In PNAS
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Editorial and Journal Policies
    • Submission Procedures
    • Fees and Licenses

New Research In

Physical Sciences

Featured Portals

  • Physics
  • Chemistry
  • Sustainability Science

Articles by Topic

  • Applied Mathematics
  • Applied Physical Sciences
  • Astronomy
  • Computer Sciences
  • Earth, Atmospheric, and Planetary Sciences
  • Engineering
  • Environmental Sciences
  • Mathematics
  • Statistics

Social Sciences

Featured Portals

  • Anthropology
  • Sustainability Science

Articles by Topic

  • Economic Sciences
  • Environmental Sciences
  • Political Sciences
  • Psychological and Cognitive Sciences
  • Social Sciences

Biological Sciences

Featured Portals

  • Sustainability Science

Articles by Topic

  • Agricultural Sciences
  • Anthropology
  • Applied Biological Sciences
  • Biochemistry
  • Biophysics and Computational Biology
  • Cell Biology
  • Developmental Biology
  • Ecology
  • Environmental Sciences
  • Evolution
  • Genetics
  • Immunology and Inflammation
  • Medical Sciences
  • Microbiology
  • Neuroscience
  • Pharmacology
  • Physiology
  • Plant Biology
  • Population Biology
  • Psychological and Cognitive Sciences
  • Sustainability Science
  • Systems Biology
Letter

The preregistration revolution needs to distinguish between predictions and analyses

Alison Ledgerwood
PNAS November 6, 2018 115 (45) E10516-E10517; first published October 19, 2018; https://doi.org/10.1073/pnas.1812592115
Alison Ledgerwood
aDepartment of Psychology, University of California, Davis, CA 95616
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: aledgerwood@ucdavis.edu

This Letter has a Reply and related content. Please see:

  • Reply to Ledgerwood: Predictions without analysis plans are inert - October 19, 2018
  • The preregistration revolution - March 12, 2018
  • Article
  • Info & Metrics
  • PDF
Loading

Nosek et al. (1) recently joined others in advocating for “widespread adoption of preregistration” as a tool for advancing science. The language they use in making this important argument, however, creates unnecessary confusion: Like many others discussing these issues, they seem to conflate the goal of theory falsification with the goal of constraining type I error. This masks a crucial distinction between two types of preregistration: preregistering a theoretical, a priori, directional prediction (which serves to clarify how a hypothesis is constructed) and preregistering an analysis plan (which serves to clarify how evidence is produced).

Indeed, philosophers of science have identified elements of both how a hypothesis is constructed and how evidence is produced that are important for scientifically valid inference (2⇓–4). We can distill these to two key, separable questions:

  • i) Have these data influenced my theoretical prediction? This question is relevant when researchers want to test existing theory: Rationally speaking, we should only adjust our confidence in a theory in response to evidence that was not itself used to construct the theoretical prediction in question (3). Preregistering theoretical predictions can help researchers distinguish clearly between using evidence to inform versus test theory (3, 5, 6).

  • ii) Have these data influenced my choice of statistical test (and/or other dataset-construction/analysis decisions)? This question is relevant when researchers want to know the type I error rate of statistical tests: Flexibility in researcher decisions can inflate the risk of false positives (7, 8). Preregistration of analysis plans can help researchers distinguish clearly between data-dependent analyses (which can be interesting but may have unknown type I error) and data-independent analyses (for which P values can be interpreted as diagnostic about the likelihood of a result; refs. 1 and 9).

Put differently, preregistration of theoretical predictions helps researchers know how to correctly calibrate their confidence that a study tests (versus informs) a theory, whereas preregistration of analysis plans helps researchers know how to correctly calibrate their confidence that a specific finding is unlikely to be due to chance.

Conflating theoretical predictions and analyses is problematic for multiple reasons. First, it implies, erroneously, that preanalysis plans can only help control type I error when research is in a prediction-making/theory-testing phase (e.g., theory Z predicts a gender difference in trait X; ref. 1). In fact, preanalysis plans can also be useful in the question-asking/discovery/theory-building phase (e.g., is there a gender difference in trait X?). Second, it may lead people to preregister the wrong things (e.g., a researcher attempting to control type I error records careful predictions but omits or only loosely specifies a preanalysis plan). Third, it increases misunderstandings and backlash against preregistration as scientists discuss these issues in everyday life (e.g., students erroneously infer that their results are more robust if they correctly guess them ahead of time; skeptics understandably argue that recording one’s prediction ahead of time has no effect on type I error).

If we want clear communication, productive debates, and effective strategies for advancing science, we must first pull apart our tangled terminology. Preregistering theoretical predictions enables theory falsifiability. Preregistering analysis plans enables type I error control.

Acknowledgments

I thank Brian Nosek, Simine Vazire, Paul Eastwick, and Roger Giner-Sorolla for their very helpful comments on earlier versions of this letter.

Footnotes

  • ↵1Email: aledgerwood{at}ucdavis.edu.
  • Author contributions: A.L. wrote the paper.

  • The author declares no conflict of interest.

Published under the PNAS license.

View Abstract

References

  1. ↵
    1. Nosek BA,
    2. Ebersole CR,
    3. DeHaven AC,
    4. Mellor DT
    (2018) The preregistration revolution. Proc Natl Acad Sci USA 115:2600–2606.
    OpenUrlAbstract/FREE Full Text
  2. ↵
    1. Mayo DG
    (1991) Novel evidence and severe tests. Philos Sci 58:523–552.
    OpenUrlCrossRef
  3. ↵
    1. Mayo DG,
    2. Spanos A
    1. Worrall J
    (2010) Error, tests, and theory confirmation. Error and Inference: Recent Exchanges on Experimental Reasoning, Reliability, and the Objectivity and Rationality of Science, eds Mayo DG, Spanos A (Cambridge Univ Press, New York), pp 125–154.
  4. ↵
    1. Rubin M
    (2017) When does HARKing hurt? Identifying when different types of undisclosed post hoc hypothesizing harm scientific progress. Rev Gen Psychol 21:308–320.
    OpenUrl
  5. ↵
    1. Popper K
    (1962) Conjectures and Refutations (Basic Books, New York).
  6. ↵
    1. van’t Veer AE,
    2. Giner-Sorolla R
    (2016) Pre-registration in social psychology—A discussion and suggested template. J Exp Soc Psychol 67:2–12.
    OpenUrl
  7. ↵
    1. Gelman A,
    2. Loken E
    (2014) The statistical crisis in science. Am Sci 102:460–465.
    OpenUrlCrossRef
  8. ↵
    1. Simmons JP,
    2. Nelson LD,
    3. Simonsohn U
    (2011) False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci 22:1359–1366.
    OpenUrlCrossRefPubMed
  9. ↵
    1. de Groot AD
    (2014) The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. 1969. Acta Psychol (Amst) 148:188–194.
    OpenUrlCrossRefPubMed
PreviousNext
Back to top
Article Alerts
Email Article

Thank you for your interest in spreading the word on PNAS.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
The preregistration revolution needs to distinguish between predictions and analyses
(Your Name) has sent you a message from PNAS
(Your Name) thought you would like to see the PNAS web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
The preregistration revolution needs to distinguish between predictions and analyses
Alison Ledgerwood
Proceedings of the National Academy of Sciences Nov 2018, 115 (45) E10516-E10517; DOI: 10.1073/pnas.1812592115

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Request Permissions
Share
The preregistration revolution needs to distinguish between predictions and analyses
Alison Ledgerwood
Proceedings of the National Academy of Sciences Nov 2018, 115 (45) E10516-E10517; DOI: 10.1073/pnas.1812592115
Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Mendeley logo Mendeley
Proceedings of the National Academy of Sciences: 115 (45)
Table of Contents

Submit

Sign up for Article Alerts

Article Classifications

  • Social Sciences
  • Psychological and Cognitive Sciences
  • Biological Sciences
  • Ecology

Jump to section

  • Article
    • Acknowledgments
    • Footnotes
    • References
  • Info & Metrics
  • PDF

You May Also be Interested in

Abstract depiction of a guitar and musical note
Science & Culture: At the nexus of music and medicine, some see disease treatments
Although the evidence is still limited, a growing body of research suggests music may have beneficial effects for diseases such as Parkinson’s.
Image credit: Shutterstock/agsandrew.
Scientist looking at an electronic tablet
Opinion: Standardizing gene product nomenclature—a call to action
Biomedical communities and journals need to standardize nomenclature of gene products to enhance accuracy in scientific and public communication.
Image credit: Shutterstock/greenbutterfly.
One red and one yellow modeled protein structures
Journal Club: Study reveals evolutionary origins of fold-switching protein
Shapeshifting designs could have wide-ranging pharmaceutical and biomedical applications in coming years.
Image credit: Acacia Dishman/Medical College of Wisconsin.
White and blue bird
Hazards of ozone pollution to birds
Amanda Rodewald, Ivan Rudik, and Catherine Kling talk about the hazards of ozone pollution to birds.
Listen
Past PodcastsSubscribe
Goats standing in a pin
Transplantation of sperm-producing stem cells
CRISPR-Cas9 gene editing can improve the effectiveness of spermatogonial stem cell transplantation in mice and livestock, a study finds.
Image credit: Jon M. Oatley.

Similar Articles

Site Logo
Powered by HighWire
  • Submit Manuscript
  • Twitter
  • Facebook
  • RSS Feeds
  • Email Alerts

Articles

  • Current Issue
  • Latest Articles
  • Archive

PNAS Portals

  • Anthropology
  • Chemistry
  • Classics
  • Front Matter
  • Physics
  • Sustainability Science
  • Teaching Resources

Information

  • Authors
  • Editorial Board
  • Reviewers
  • Librarians
  • Press
  • Site Map
  • PNAS Updates

Feedback    Privacy/Legal

Copyright © 2021 National Academy of Sciences. Online ISSN 1091-6490