Despite myriad challenges, clinicians see room for progress.
Image credit: Shutterstock/David Tadevosian.
Edited by David B. Allison, Indiana University Bloomington, Bloomington, IN, and accepted by Editorial Board Member Susan T. Fiske January 9, 2018 (received for review July 11, 2017)
Type of response | Count | Percent, % |
Did not share data or code: | ||
Contact another person | 20 | 11 |
Asked for reasons | 20 | 11 |
Refusal to share | 12 | 7 |
Directed back to supplement | 6 | 3 |
Unfulfilled promise to follow up | 5 | 3 |
Impossible to share | 3 | 2 |
Shared data and code | 65 | 36 |
Email bounced | 3 | 2 |
No response | 46 | 26 |
ICERM criteria | Percent compliant,% |
A precise statement of assertions to be made in the paper. | 100 |
Full statement (or valid summary) of experimental results. | 100 |
Salient details of data reduction & statistical analysis methods. | 91 |
Necessary run parameters were given. | 86 |
A statement of the computational approach, and why it constitutes a rigorous test of the hypothesized assertions. | 8 |
Complete statements of, or references to, every algorithm used, and salient details of auxiliary software (both research and commercial software) used in the computation. | 80 |
Discussion of the adequacy of parameters such as precision level and grid resolution. | 79 |
Proper citation of all code and data used, including that generated by the authors. | 79 |
Availability of computer code, input and output data, with some reasonable level of documentation. | 77 |
Avenues of exploration examined throughout development, including information about negative findings. | 68 |
Instructions for repeating computational experiments described in the article. | 63 |
Precise functions were given, with settings. | 41 |
Salient details of the test environment, including hardware, system software, and number of processors used. | 13 |
ICERM criteria | Percent compliant,% |
Data documented to clearly explain what each part represents. | 91 |
Data archived with significant longevity expected. | 82 |
Data location provided in the acknowledgements. | 70 |
Authors have documented use and licensing rights. | 66 |
Software documented well enough to run it and what it ought to do. | 57 |
The code is publicly available with no download requirements. | 54 |
There was some method to track changes/to the software, as well as some certainty that the code is securely archived. | 50 |
Classification | Percent, % |
Impossible to reproduce (missing essential code, data,or methodology) | 5 |
Nearly impossible to reproduce (specialized hardware,intense computation requirements, sensitive data,human study, or other unavoidable reasons) | 14 |
Difficult to reproduce because of unavoidable inherent complexity (e.g., requiring 300 million Markov chain Monte Carlo steps on each dataset, or needing months to do runs) | 14 |
Reproducible with substantial tedious effort (e.g.,individual download of a large number of datasets,hand coding of data into a new format, i.e., from an image, many archiving steps required) | 5 |
Reproducible with substantial intellectual effort (e.g.,methods well defined but required some knowledge of jargon or understanding of the field; or down the rabbit hole references to past articles required to reproduce; etc.) | 5 |
Could reproduce with fairly substantial skill and knowledge (e.g., required GPU programing abilities to run code that wasn’t given; translating complex models into MATLAB code; pseudo code with functions not detailed described in text into code; missing scripts) | 23 |
Reproducible after tweaking (e.g., missing parameters required fiddling to find, missing modified code lines, missing arguments required for differing architecture; missing minor method step) | 5 |
Minor difficulty in reproducing (e.g., installing a specialized library, converting to a different computational system) | 18 |
Straightforward to reproduce with minimal effort | 14 |
Disclosure practice | 2009–2010,% | 2011–2012,% |
Citations to data and/or code in references | 25 | 29 |
Data location given in acknowledgements | 29 | 48 |
Code location given in acknowledgements | 4 | 5 |
Materials availability | 2009–2010,% | 2011–2012,% |
Most or all relevant data locations given | 52 | 75 |
Most or all relevant software locations given | 43 | 54 |
Some data and software locations given | 25 | 45 |
All major software and data locations given | 15 | 25 |
Code, scripts, parameters, documentation | 10 | 12 |
No supporting materials available | 4 | 1 |