PNAS Profile of NAS member Angelika Amon, winner of the 2019 Vilcek Prize in Biomedical Science
Image courtesy of Samara Vise (Koch Institute for Integrative Cancer Research, Cambridge, MA.)
New Research In
Physical Sciences
Featured Portals
Articles by Topic
Biological Sciences
Featured Portals
Articles by Topic
Edited by David B. Allison, Indiana University Bloomington, Bloomington, IN, and accepted by Editorial Board Member Susan T. Fiske January 9, 2018 (received for review July 11, 2017)
Article Figures & SI
Tables
Type of response Count Percent, % Did not share data or code: Contact another person 20 11 Asked for reasons 20 11 Refusal to share 12 7 Directed back to supplement 6 3 Unfulfilled promise to follow up 5 3 Impossible to share 3 2 Shared data and code 65 36 Email bounced 3 2 No response 46 26 ICERM criteria Percent compliant,% A precise statement of assertions to be made in the paper. 100 Full statement (or valid summary) of experimental results. 100 Salient details of data reduction & statistical analysis methods. 91 Necessary run parameters were given. 86 A statement of the computational approach, and why it constitutes a rigorous test of the hypothesized assertions. 8 Complete statements of, or references to, every algorithm used, and salient details of auxiliary software (both research and commercial software) used in the computation. 80 Discussion of the adequacy of parameters such as precision level and grid resolution. 79 Proper citation of all code and data used, including that generated by the authors. 79 Availability of computer code, input and output data, with some reasonable level of documentation. 77 Avenues of exploration examined throughout development, including information about negative findings. 68 Instructions for repeating computational experiments described in the article. 63 Precise functions were given, with settings. 41 Salient details of the test environment, including hardware, system software, and number of processors used. 13 ICERM criteria Percent compliant,% Data documented to clearly explain what each part represents. 91 Data archived with significant longevity expected. 82 Data location provided in the acknowledgements. 70 Authors have documented use and licensing rights. 66 Software documented well enough to run it and what it ought to do. 57 The code is publicly available with no download requirements. 54 There was some method to track changes/to the software, as well as some certainty that the code is securely archived. 50 Classification Percent, % Impossible to reproduce (missing essential code, data,or methodology) 5 Nearly impossible to reproduce (specialized hardware,intense computation requirements, sensitive data,human study, or other unavoidable reasons) 14 Difficult to reproduce because of unavoidable inherent complexity (e.g., requiring 300 million Markov chain Monte Carlo steps on each dataset, or needing months to do runs) 14 Reproducible with substantial tedious effort (e.g.,individual download of a large number of datasets,hand coding of data into a new format, i.e., from an image, many archiving steps required) 5 Reproducible with substantial intellectual effort (e.g.,methods well defined but required some knowledge of jargon or understanding of the field; or down the rabbit hole references to past articles required to reproduce; etc.) 5 Could reproduce with fairly substantial skill and knowledge (e.g., required GPU programing abilities to run code that wasn’t given; translating complex models into MATLAB code; pseudo code with functions not detailed described in text into code; missing scripts) 23 Reproducible after tweaking (e.g., missing parameters required fiddling to find, missing modified code lines, missing arguments required for differing architecture; missing minor method step) 5 Minor difficulty in reproducing (e.g., installing a specialized library, converting to a different computational system) 18 Straightforward to reproduce with minimal effort 14 Disclosure practice 2009–2010,% 2011–2012,% Citations to data and/or code in references 25 29 Data location given in acknowledgements 29 48 Code location given in acknowledgements 4 5 Materials availability 2009–2010,% 2011–2012,% Most or all relevant data locations given 52 75 Most or all relevant software locations given 43 54 Some data and software locations given 25 45 All major software and data locations given 15 25 Code, scripts, parameters, documentation 10 12 No supporting materials available 4 1
Journal policy for computational reproducibility
Sign up for Article Alerts
Jump to section
You May Also be Interested in
Simulations of future California climate suggest that under high CO2 conditions, by the late 21st century, increasing temperatures could lead to reduced snowpack, drier summers, and increased fire risk, independent of changes in winter precipitation.
Image courtesy of Alan H. Taylor.
Researchers report a system for obtaining information about planetary motions approximately 200 million years ago, and the system could help constrain models of Solar System evolution and provide tests of gravitational models.
Image courtesy of Paul E. Olsen.
Compared with female co-twins not exposed to a male co-twin inside the uterus, female co-twins exposed to a male co-twin had 15.2% higher probability of dropping out of high school, 8.6% lower earnings at age 32, and 5.8% fewer children, according to a study.
Image courtesy of iStock/Avril Morgan.
In the last few years, astronomers have started to get glimpses into the planet nurseries formed from newborn stars, revealing important clues as to exactly how planets form.
Image credit: Shutterstock/Neo Edmund.
More Articles of This Classification
Related Content
Cited by...
- A comprehensive analysis of the usability and archival stability of omics computational tools and resources
- On the prevalence of uninformative parameters in statistical models applying model selection in applied ecology
- Pixel: a digital lab assistant to integrate biological data in multi-omics projects
- Identifying and Overcoming Threats to Reproducibility, Replicability, Robustness, and Generalizability in Microbiome Research
- Sackler Colloquium on Improving the Reproducibility of Scientific Research: Reproducibility of research: Issues and proposed remedies