An empirical analysis of journal policy effectiveness for computational reproducibility
See allHide authors and affiliations
Edited by David B. Allison, Indiana University Bloomington, Bloomington, IN, and accepted by Editorial Board Member Susan T. Fiske January 9, 2018 (received for review July 11, 2017)

Online Impact
Article Information
- Published in issue March 13, 2018.
- Published first March 12, 2018.
Article Versions
- Previous version (March 12, 2018 - 12:08).
- You are viewing the most recent version of this article.
Author Information
- aSchool of Information Sciences, University of Illinois at Urbana–Champaign, Champaign, IL 61820;
- bDepartment of Statistics, Columbia University, New York, NY 10027
Edited by David B. Allison, Indiana University Bloomington, Bloomington, IN, and accepted by Editorial Board Member Susan T. Fiske January 9, 2018 (received for review July 11, 2017)
Footnotes
- ↵1To whom correspondence should be addressed. Email: victoria{at}stodden.net.
Author contributions: V.S. designed research; V.S., J.S., and Z.M. performed research; V.S., J.S., and Z.M. analyzed data; and V.S. and J.S. wrote the paper.
The authors declare no conflict of interest.
This paper results from the Arthur M. Sackler Colloquium of the National Academy of Sciences, “Reproducibility of Research: Issues and Proposed Remedies,” held March 8-10, 2017, at the National Academy of Sciences in Washington, DC. The complete program and video recordings of most presentations are available on the NAS website at www.nasonline.org/Reproducibility.
This article is a PNAS Direct Submission. D.B.A. is a guest editor invited by the Editorial Board.
Data deposition: We have created a repository at https://github.com/ReproducibilityInPublishing/Science-2018 containing supplemental material, including the code and email templates that were used in the survey.
Cited By...
This article has been cited by the following articles in journals that are participating in Crossref Cited-by Linking.
- Patrick D. Schloss, Jacques RavelmBio 2018 9 3
- Olivier Klein, Tom E. Hardwicke, Frederik Aust, Johannes Breuer, Henrik Danielsson, Alicia Hofelich Mohr, Hans IJzerman, Gustav Nilsonne, Wolf Vanpaemel, Michael C. Frank, Michéle Nuijten, Simine VazireCollabra: Psychology 2018 4 1
- Serghei Mangul, Lana S. Martin, Brian L. Hill, Angela Ka-Mei Lam, Margaret G. Distler, Alex Zelikovsky, Eleazar Eskin, Jonathan FlintNature Communications 2019 10 1
- Serghei Mangul, Lana S. Martin, Eleazar Eskin, Ran BlekhmanGenome Biology 2019 20 1
- Shawn J. Leroux, Andrew J. KrollPLOS ONE 2019 14 2
- Rachael V. Gallagher, Daniel S. Falster, Brian S. Maitner, Roberto Salguero-Gómez, Vigdis Vandvik, William D. Pearse, Florian D. Schneider, Jens Kattge, Jorrit H. Poelen, Joshua S. Madin, Markus J. Ankenbrand, Caterina Penone, Xiao Feng, Vanessa M. Adams, John Alroy, Samuel C. Andrew, Meghan A. Balk, Lucie M. Bland, Brad L. Boyle, Catherine H. Bravo-Avila, Ian Brennan, Alexandra J. R. Carthey, Renee Catullo, Brittany R. Cavazos, Dalia A. Conde, Steven L. Chown, Belen Fadrique, Heloise Gibb, Aud H. Halbritter, Jennifer Hammock, J. Aaron Hogan, Hamish Holewa, Michael Hope, Colleen M. Iversen, Malte Jochum, Michael Kearney, Alexander Keller, Paula Mabee, Peter Manning, Luke McCormack, Sean T. Michaletz, Daniel S. Park, Timothy M. Perez, Silvia Pineda-Munoz, Courtenay A. Ray, Maurizio Rossetto, Hervé Sauquet, Benjamin Sparrow, Marko J. Spasojevic, Richard J. Telford, Joseph A. Tobias, Cyrille Violle, Ramona Walls, Katherine C. B. Weiss, Mark Westoby, Ian J. Wright, Brian J. EnquistNature Ecology & Evolution 2020 4 3
- David B. Allison, Richard M. Shiffrin, Victoria StoddenProceedings of the National Academy of Sciences 2018 115 11
- Stephen M. Powers, Stephanie E. HamptonEcological Applications 2019 29 1
- Adrian-Tudor Pãnescu, Vasile MantaScience & Technology Libraries 2018 37 3
- Marcin Miłkowski, Witold M. Hensel, Mateusz HoholJournal of Computational Neuroscience 2018 45 3
- J. Kyle Medley, Kiri Choi, Matthias König, Lucian Smith, Stanley Gu, Joseph Hellerstein, Stuart C. Sealfon, Herbert M. Sauro, Andreas PrlicPLOS Computational Biology 2018 14 6
- Peter W. Hildebrand, Alexander S. Rose, Johanna K.S. TiemannTrends in Biochemical Sciences 2019 44 11
- Serghei Mangul, Thiago Mosqueiro, Richard J. Abdill, Dat Duong, Keith Mitchell, Varuni Sarwal, Brian Hill, Jaqueline Brito, Russell Jared Littman, Benjamin Statz, Angela Ka-Mei Lam, Gargi Dayama, Laura Grieneisen, Lana S. Martin, Jonathan Flint, Eleazar Eskin, Ran BlekhmanPLOS Biology 2019 17 6
- Yang-Min Kim, Jean-Baptiste Poline, Guillaume DumasGigaScience 2018 7 7
- Paloma Ruiz-Benito, Giorgio Vacchiano, Emily R. Lines, Christopher P.O. Reyer, Sophia Ratcliffe, Xavier Morin, Florian Hartig, Annikki Mäkelä, Rasoul Yousefpour, Jimena E. Chaves, Alicia Palacios-Orueta, Marta Benito-Garzón, Cesar Morales-Molino, J. Julio Camarero, Alistair S. Jump, Jens Kattge, Aleksi Lehtonen, Andreas Ibrom, Harry J.F. Owen, Miguel A. ZavalaEcological Modelling 2020 416
- James H. Stagge, David E. Rosenberg, Adel M. Abdallah, Hadia Akbar, Nour A. Attallah, Ryan JamesScientific Data 2019 6 1
- Chad A. Greene, Kaustubh Thirumalai, Kelly A. Kearney, José Miguel Delgado, Wolfgang Schwanghart, Natalie S. Wolfenbarger, Kristen M. Thyng, David E. Gwyther, Alex S. Gardner, Donald D. BlankenshipGeochemistry, Geophysics, Geosystems 2019 20 7
- Jessica L. Couture, Rachael E. Blake, Gavin McDonald, Colette L. Ward, Jelte M. WichertsPLOS ONE 2018 13 7
- Daniel S QuintanaeLife 2020 9
- John D. Blischak, Peter Carbonetto, Matthew StephensF1000Research 2019 8
- Gabriel A. Vieira, Francisco ProsdocimiPeerJ 2019 7
- Robert Forkel, Johann-Mattis List, Simon J. Greenhill, Christoph Rzymski, Sebastian Bank, Michael Cysouw, Harald Hammarström, Martin Haspelmath, Gereon A. Kaiping, Russell D. GrayScientific Data 2018 5 1
- Kate A. Field, Paul C. Paquet, Kyle Artelle, Gilbert Proulx, Ryan K. Brook, Chris T. DarimontPLOS Biology 2019 17 4
- Daniel C. Kirouac, Brian Cicali, Stephan SchmidtCPT: Pharmacometrics & Systems Pharmacology 2019 8 4
- Olavo B Amaral, Kleber Neves, Ana P Wasilewska-Sampaio, Clarissa FD CarneiroeLife 2019 8
- Dan Sholler, Karthik Ram, Carl Boettiger, Daniel S KatzBig Data & Society 2019 6 1
- Jaqueline J Brito, Jun Li, Jason H Moore, Casey S Greene, Nicole A Nogoy, Lana X Garmire, Serghei MangulGigaScience 2020 9 6
- B. Bond-LambertyJournal of Geophysical Research: Biogeosciences 2018 123 4
- Mark Yarborough, Robert Nadon, David G KarlineLife 2019 8
- Tom E. Hardwicke, Stylianos Serghiou, Perrine Janiaud, Valentin Danchev, Sophia Crüwell, Steven N. Goodman, John P.A. IoannidisAnnual Review of Statistics and Its Application 2020 7 1
- Antica Culina, Ilona van den Berg, Simon Evans, Alfredo Sánchez-TójarPLOS Biology 2020 18 7
- Santiago SchnellBulletin of Mathematical Biology 2018 80 12
- Paul N. Schofield, Ulrike Kulka, Soile Tapio, Bernd GroscheInternational Journal of Radiation Biology 2019 95 7
- Thu-Mai Christian, Amanda Gooch, Todd Vision, Elizabeth Hull, Cassidy Rose SugimotoPLOS ONE 2020 15 3
- Derek M Isaacowitz, Majse Lind, Laura P SandsInnovation in Aging 2019 3 4
- Daniel Sui, Peter KedronAnnals of the American Association of Geographers 2020
- Pepijn Obels, Daniël Lakens, Nicholas A. Coles, Jaroslav Gottfried, Seth A. GreenAdvances in Methods and Practices in Psychological Science 2020 3 2
- Shelley Stall, Leah McEwen, Lesley Wyborn, Nancy Hoebelheinrich, Ian BrunoData Intelligence 2020 2 1-2
- Daniel C. KirouacClinical Pharmacology & Therapeutics 2020 107 4
- Open Up – the Mission Statement of the Control of Impulsive Action (Ctrl-ImpAct) Lab on Open ScienceChristina B. Reimer, Zhang Chen, Carsten Bundt, Charlotte Eben, Raquel E. London, Sirarpi VardanianPsychologica Belgica 2019 59 1
- Daniel Nüst, Edzer PebesmaAnnals of the American Association of Geographers 2020
- David E. Rosenberg, Yves Filion, Rebecca Teasley, Samuel Sandoval-Solis, Jory S. Hecht, Jakobus E. van Zyl, George F. McMahon, Jeffery S. Horsburgh, Joseph R. Kasprzyk, David G. TarbotonJournal of Water Resources Planning and Management 2020 146 6
- Jeffery S. Horsburgh, Richard P. Hooper, Jerad Bales, Margaret Hedstrom, Heidi J. Imker, Kerstin A. Lehnert, Lea A. Shanley, Shelley StallWIREs Water 2020 7 3
- Travis C. Tai, James P. W. RobinsonFrontiers in Environmental Science 2018 6
- Nikolas I. Krieger, Adam T. Perzynski, Jarrod E. Dalton, Vasilis J. PromponasPLOS ONE 2019 14 7
- Matteo Pellegrini, Serghei MangulEmerging Topics in Life Sciences 2019 3 4
- Thomas Denecker, William Durand, Julien Maupetit, Charles Hébert, Jean-Michel Camadro, Pierre Poulain, Gaëlle LelandaisPeerJ 2019 7
- Christophe Hurlin, Christophe PerignonSSRN Electronic Journal 2019
- Peter Kedron, Wenwen Li, Stewart Fotheringham, Michael GoodchildInternational Journal of Geographical Information Science 2021 35 3
- David M. Liu, Matthew J. SalganikSocius: Sociological Research for a Dynamic World 2019 5
- Richard C. Becker, Vlad Cotarlan, Sakthi SadayappanJournal of Thrombosis and Thrombolysis 2019 47 3
- Tom E. Hardwicke, Manuel Bohn, Kyle MacDonald, Emily Hembacher, Michèle B. Nuijten, Benjamin N. Peloquin, Benjamin E. deMayo, Bria Long, Erica J. Yoon, Michael C. FrankRoyal Society Open Science 2021 8 1
- Wei Jeng, Shih-Hung Wang, Hung-Wei Chen, Po-Wei Huang, Yu-Jen Chen, Hsu-Chun Hsiao, He DebiaoPLOS ONE 2020 15 11
- Neal D. Goldstein, Ghassan B. Hamra, Sam HarperEpidemiology 2020 31 2
- Clinical relevance assessment of animal preclinical research (RAA) tool: development and explanationKurinchi S. Gurusamy, David Moher, Marilena Loizidou, Irfan Ahmed, Marc T. Avey, Carly C. Barron, Brian Davidson, Miriam Dwek, Christian Gluud, Gavin Jell, Kiran Katakam, Joshua Montroy, Timothy D. McHugh, Nicola J. Osborne, Merel Ritskes-Hoitinga, Kees van Laarhoven, Jan Vollert, Manoj LaluPeerJ 2021 9
- Juan A. Añel, Diego P. Montes, Javier Rodeiro Iglesias2020
- David N. Borg, Joshua J. Bon, Kristin L. Sainani, Brenton J. Baguley, Nicholas J. Tierney, Christopher DrovandiSports Medicine 2020 50 8
- Juan A. Añel, Michael García-Rodríguez, Javier RodeiroGeoscientific Model Development 2021 14 2
- Witold M. HenselEuropean Journal for Philosophy of Science 2020 10 3
- Pajau Vangay, Josephine Burgin, Anjanette Johnston, Kristen L. Beck, Daniel C. Berrios, Kai Blumberg, Shane Canon, Patrick Chain, John-Marc Chandonia, Danielle Christianson, Sylvain V. Costes, Joan Damerow, William D. Duncan, Jose Pablo Dundore-Arias, Kjiersten Fagnan, Jonathan M. Galazka, Sean M. Gibbons, David Hays, Judson Hervey, Bin Hu, Bonnie L. Hurwitz, Pankaj Jaiswal, Marcin P. Joachimiak, Linda Kinkel, Joshua Ladau, Stanton L. Martin, Lee Ann McCue, Kayd Miller, Nigel Mouncey, Chris Mungall, Evangelos Pafilis, T. B. K. Reddy, Lorna Richardson, Simon Roux, Justin P. Shaffer, Jagadish Chandrabose Sundaramurthi, Luke R. Thompson, Ruth E. Timme, Jie Zheng, Elisha M. Wood-Charlson, Emiley A. Eloe-Fadrosh, Vanni BuccimSystems 2021 6 1
- Marco A. Janssen, Calvin Pritchard, Allen LeeEnvironmental Modelling & Software 2020 134
- Matthew Lau, Thomas F. J.-M. Pasquier, Margo SeltzerJournal of Open Source Software 2020 5 46
- Andrea K Thomer, Karen M WickettBig Data & Society 2020 7 1
- Ana Trisovic, Katherine Mika, Ceilyn Boyd, Sebastian Feger, Mercè CrosasData 2021 6 2
- Saman Razavi, Anthony Jakeman, Andrea Saltelli, Clémentine Prieur, Bertrand Iooss, Emanuele Borgonovo, Elmar Plischke, Samuele Lo Piano, Takuya Iwanaga, William Becker, Stefano Tarantola, Joseph H.A. Guillaume, John Jakeman, Hoshin Gupta, Nicola Melillo, Giovanni Rabitti, Vincent Chabridon, Qingyun Duan, Xifu Sun, Stefán Smith, Razi Sheikholeslami, Nasim Hosseini, Masoud Asadzadeh, Arnald Puy, Sergei Kucherenko, Holger R. MaierEnvironmental Modelling & Software 2021 137
- Mathieu Bernard, Roland Thiolliere, Amanda Saksida, Georgia R. Loukatou, Elin Larsen, Mark Johnson, Laia Fibla, Emmanuel Dupoux, Robert Daland, Xuan Nga Cao, Alejandrina CristiaBehavior Research Methods 2020 52 1
Article usage
Citation Manager Formats
Article Classifications
- Physical Sciences
- Computer Sciences
- Social Sciences
- Social Sciences