Skip to main content
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Accessibility Statement
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian
  • Log in
  • My Cart

Main menu

  • Home
  • Articles
    • Current
    • Special Feature Articles - Most Recent
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • List of Issues
  • Front Matter
  • News
    • For the Press
    • This Week In PNAS
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Editorial and Journal Policies
    • Submission Procedures
    • Fees and Licenses
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Accessibility Statement
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian

User menu

  • Log in
  • My Cart

Search

  • Advanced search
Home
Home

Advanced Search

  • Home
  • Articles
    • Current
    • Special Feature Articles - Most Recent
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • List of Issues
  • Front Matter
  • News
    • For the Press
    • This Week In PNAS
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Editorial and Journal Policies
    • Submission Procedures
    • Fees and Licenses

New Research In

Physical Sciences

Featured Portals

  • Physics
  • Chemistry
  • Sustainability Science

Articles by Topic

  • Applied Mathematics
  • Applied Physical Sciences
  • Astronomy
  • Computer Sciences
  • Earth, Atmospheric, and Planetary Sciences
  • Engineering
  • Environmental Sciences
  • Mathematics
  • Statistics

Social Sciences

Featured Portals

  • Anthropology
  • Sustainability Science

Articles by Topic

  • Economic Sciences
  • Environmental Sciences
  • Political Sciences
  • Psychological and Cognitive Sciences
  • Social Sciences

Biological Sciences

Featured Portals

  • Sustainability Science

Articles by Topic

  • Agricultural Sciences
  • Anthropology
  • Applied Biological Sciences
  • Biochemistry
  • Biophysics and Computational Biology
  • Cell Biology
  • Developmental Biology
  • Ecology
  • Environmental Sciences
  • Evolution
  • Genetics
  • Immunology and Inflammation
  • Medical Sciences
  • Microbiology
  • Neuroscience
  • Pharmacology
  • Physiology
  • Plant Biology
  • Population Biology
  • Psychological and Cognitive Sciences
  • Sustainability Science
  • Systems Biology
Colloquium Paper

An empirical analysis of journal policy effectiveness for computational reproducibility

View ORCID ProfileVictoria Stodden, Jennifer Seiler, and Zhaokun Ma
PNAS March 13, 2018 115 (11) 2584-2589; first published March 12, 2018; https://doi.org/10.1073/pnas.1708290115
Victoria Stodden
aSchool of Information Sciences, University of Illinois at Urbana–Champaign, Champaign, IL 61820;
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Victoria Stodden
  • For correspondence: victoria@stodden.net
Jennifer Seiler
bDepartment of Statistics, Columbia University, New York, NY 10027
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Zhaokun Ma
bDepartment of Statistics, Columbia University, New York, NY 10027
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  1. Edited by David B. Allison, Indiana University Bloomington, Bloomington, IN, and accepted by Editorial Board Member Susan T. Fiske January 9, 2018 (received for review July 11, 2017)

  • Article
  • Figures & SI
  • Info & Metrics
  • PDF
Loading

Article Figures & SI

Tables

    • View popup
    Table 1.

    Responses to emailed requests (n = 180)

    Type of responseCountPercent, %
    Did not share data or code:
    Contact another person2011
    Asked for reasons2011
    Refusal to share127
    Directed back to supplement63
    Unfulfilled promise to follow up53
    Impossible to share32
    Shared data and code6536
    Email bounced32
    No response4626
    • View popup
    Table 2.

    ICERM implementation criteria for articles deemed likely to reproduce (n = 56)

    ICERM criteriaPercent compliant,%
    A precise statement of assertions to be made in the paper.100
    Full statement (or valid summary) of experimental results.100
    Salient details of data reduction & statistical analysis methods.91
    Necessary run parameters were given.86
    A statement of the computational approach, and why it constitutes a rigorous test of the hypothesized assertions.8
    Complete statements of, or references to, every algorithm used, and salient details of auxiliary software (both research and commercial software) used in the computation.80
    Discussion of the adequacy of parameters such as precision level and grid resolution.79
    Proper citation of all code and data used, including that generated by the authors.79
    Availability of computer code, input and output data, with some reasonable level of documentation.77
    Avenues of exploration examined throughout development, including information about negative findings.68
    Instructions for repeating computational experiments described in the article.63
    Precise functions were given, with settings.41
    Salient details of the test environment, including hardware, system software, and number of processors used.13
    • View popup
    Table 3.

    ICERM archiving criteria for articles deemed likely to reproduce (n = 56)

    ICERM criteriaPercent compliant,%
    Data documented to clearly explain what each part represents.91
    Data archived with significant longevity expected.82
    Data location provided in the acknowledgements.70
    Authors have documented use and licensing rights.66
    Software documented well enough to run it and what it ought to do.57
    The code is publicly available with no download requirements.54
    There was some method to track changes/to the software, as well as some certainty that the code is securely archived.50
    • View popup
    Table 4.

    Classification of reproducibility effort (n = 22)

    ClassificationPercent, %
    Impossible to reproduce (missing essential code, data,or methodology)5
    Nearly impossible to reproduce (specialized hardware,intense computation requirements, sensitive data,human study, or other unavoidable reasons)14
    Difficult to reproduce because of unavoidable inherent complexity (e.g., requiring 300 million Markov chain Monte Carlo steps on each dataset, or needing months to do runs)14
    Reproducible with substantial tedious effort (e.g.,individual download of a large number of datasets,hand coding of data into a new format, i.e., from an image, many archiving steps required)5
    Reproducible with substantial intellectual effort (e.g.,methods well defined but required some knowledge of jargon or understanding of the field; or down the rabbit hole references to past articles required to reproduce; etc.)5
    Could reproduce with fairly substantial skill and knowledge (e.g., required GPU programing abilities to run code that wasn’t given; translating complex models into MATLAB code; pseudo code with functions not detailed described in text into code; missing scripts)23
    Reproducible after tweaking (e.g., missing parameters required fiddling to find, missing modified code lines, missing arguments required for differing architecture; missing minor method step)5
    Minor difficulty in reproducing (e.g., installing a specialized library, converting to a different computational system)18
    Straightforward to reproduce with minimal effort14
    • View popup
    Table 5.

    Changes in disclosure practices

    Disclosure practice2009–2010,%2011–2012,%
    Citations to data and/or code in references2529
    Data location given in acknowledgements2948
    Code location given in acknowledgements45
    • View popup
    Table 6.

    Materials availability via inspection

    Materials availability2009–2010,%2011–2012,%
    Most or all relevant data locations given5275
    Most or all relevant software locations given4354
    Some data and software locations given2545
    All major software and data locations given1525
    Code, scripts, parameters, documentation1012
    No supporting materials available41
PreviousNext
Back to top
Article Alerts
Email Article

Thank you for your interest in spreading the word on PNAS.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
An empirical analysis of journal policy effectiveness for computational reproducibility
(Your Name) has sent you a message from PNAS
(Your Name) thought you would like to see the PNAS web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Journal policy for computational reproducibility
Victoria Stodden, Jennifer Seiler, Zhaokun Ma
Proceedings of the National Academy of Sciences Mar 2018, 115 (11) 2584-2589; DOI: 10.1073/pnas.1708290115

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Request Permissions
Share
Journal policy for computational reproducibility
Victoria Stodden, Jennifer Seiler, Zhaokun Ma
Proceedings of the National Academy of Sciences Mar 2018, 115 (11) 2584-2589; DOI: 10.1073/pnas.1708290115
Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Mendeley logo Mendeley
Proceedings of the National Academy of Sciences: 115 (11)
Table of Contents

Submit

Sign up for Article Alerts

Article Classifications

  • Physical Sciences
  • Computer Sciences
  • Social Sciences
  • Social Sciences

Jump to section

  • Article
    • Abstract
    • Results
    • Methods
    • Conclusion
    • Acknowledgments
    • Footnotes
    • References
  • Figures & SI
  • Info & Metrics
  • PDF

You May Also be Interested in

Surgeons hands during surgery
Inner Workings: Advances in infectious disease treatment promise to expand the pool of donor organs
Despite myriad challenges, clinicians see room for progress.
Image credit: Shutterstock/David Tadevosian.
Setting sun over a sun-baked dirt landscape
Core Concept: Popular integrated assessment climate policy models have key caveats
Better explicating the strengths and shortcomings of these models will help refine projections and improve transparency in the years ahead.
Image credit: Witsawat.S.
Double helix
Journal Club: Noncoding DNA shown to underlie function, cause limb malformations
Using CRISPR, researchers showed that a region some used to label “junk DNA” has a major role in a rare genetic disorder.
Image credit: Nathan Devery.
Steamboat Geyser eruption.
Eruption of Steamboat Geyser
Mara Reed and Michael Manga explore why Yellowstone's Steamboat Geyser resumed erupting in 2018.
Listen
Past PodcastsSubscribe
Multi-color molecular model
Enzymatic breakdown of PET plastic
A study demonstrates how two enzymes—MHETase and PETase—work synergistically to depolymerize the plastic pollutant PET.
Image credit: Aaron McGeehan (artist).

Similar Articles

Site Logo
Powered by HighWire
  • Submit Manuscript
  • Twitter
  • Facebook
  • RSS Feeds
  • Email Alerts

Articles

  • Current Issue
  • Special Feature Articles – Most Recent
  • List of Issues

PNAS Portals

  • Anthropology
  • Chemistry
  • Classics
  • Front Matter
  • Physics
  • Sustainability Science
  • Teaching Resources

Information

  • Authors
  • Editorial Board
  • Reviewers
  • Librarians
  • Press
  • Site Map
  • PNAS Updates

Feedback    Privacy/Legal

Copyright © 2021 National Academy of Sciences. Online ISSN 1091-6490