An empirical analysis of journal policy effectiveness for computational reproducibility

New Research In

Edited by David B. Allison, Indiana University Bloomington, Bloomington, IN, and accepted by Editorial Board Member Susan T. Fiske January 9, 2018 (received for review July 11, 2017)

Table 1.

Responses to emailed requests (n = 180)

Type of response	Count	Percent, %
Did not share data or code:
Contact another person	20	11
Asked for reasons	20	11
Refusal to share	12	7
Directed back to supplement	6	3
Unfulfilled promise to follow up	5	3
Impossible to share	3	2
Shared data and code	65	36
Email bounced	3	2
No response	46	26

Table 2.

ICERM implementation criteria for articles deemed likely to reproduce (n = 56)

ICERM criteria	Percent compliant,%
A precise statement of assertions to be made in the paper.	100
Full statement (or valid summary) of experimental results.	100
Salient details of data reduction & statistical analysis methods.	91
Necessary run parameters were given.	86
A statement of the computational approach, and why it constitutes a rigorous test of the hypothesized assertions.	8
Complete statements of, or references to, every algorithm used, and salient details of auxiliary software (both research and commercial software) used in the computation.	80
Discussion of the adequacy of parameters such as precision level and grid resolution.	79
Proper citation of all code and data used, including that generated by the authors.	79
Availability of computer code, input and output data, with some reasonable level of documentation.	77
Avenues of exploration examined throughout development, including information about negative findings.	68
Instructions for repeating computational experiments described in the article.	63
Precise functions were given, with settings.	41
Salient details of the test environment, including hardware, system software, and number of processors used.	13

Table 3.

ICERM archiving criteria for articles deemed likely to reproduce (n = 56)

ICERM criteria	Percent compliant,%
Data documented to clearly explain what each part represents.	91
Data archived with significant longevity expected.	82
Data location provided in the acknowledgements.	70
Authors have documented use and licensing rights.	66
Software documented well enough to run it and what it ought to do.	57
The code is publicly available with no download requirements.	54
There was some method to track changes/to the software, as well as some certainty that the code is securely archived.	50

Table 4.

Classification of reproducibility effort (n = 22)

Classification	Percent, %
Impossible to reproduce (missing essential code, data,or methodology)	5
Nearly impossible to reproduce (specialized hardware,intense computation requirements, sensitive data,human study, or other unavoidable reasons)	14
Difficult to reproduce because of unavoidable inherent complexity (e.g., requiring 300 million Markov chain Monte Carlo steps on each dataset, or needing months to do runs)	14
Reproducible with substantial tedious effort (e.g.,individual download of a large number of datasets,hand coding of data into a new format, i.e., from an image, many archiving steps required)	5
Reproducible with substantial intellectual effort (e.g.,methods well defined but required some knowledge of jargon or understanding of the field; or down the rabbit hole references to past articles required to reproduce; etc.)	5
Could reproduce with fairly substantial skill and knowledge (e.g., required GPU programing abilities to run code that wasn’t given; translating complex models into MATLAB code; pseudo code with functions not detailed described in text into code; missing scripts)	23
Reproducible after tweaking (e.g., missing parameters required fiddling to find, missing modified code lines, missing arguments required for differing architecture; missing minor method step)	5
Minor difficulty in reproducing (e.g., installing a specialized library, converting to a different computational system)	18
Straightforward to reproduce with minimal effort	14

Table 6.

Materials availability via inspection

Materials availability	2009–2010,%	2011–2012,%
Most or all relevant data locations given	52	75
Most or all relevant software locations given	43	54
Some data and software locations given	25	45
All major software and data locations given	15	25
Code, scripts, parameters, documentation	10	12
No supporting materials available	4	1