Measuring the effectiveness of scientific gatekeeping

Kyle Siler; Kirby Lee; Lisa Bero

doi:10.1073/pnas.1418218112

New Research In

Edited by Susan T. Fiske, Princeton University, Princeton, NJ, and approved November 18, 2014 (received for review September 21, 2014)

This article has a Letter. Please see:

Mistake index as a surrogate of quality in scientific manuscripts - March 10, 2015

See related content:

Science should prevent mistakes, not corrections
- Mar 10, 2015

Significance

Peer review is an institution of enormous importance for the careers of scientists and the content of published science. The decisions of gatekeepers—editors and peer reviewers—legitimize scientific findings, distribute professional rewards, and influence future research. However, appropriate data to gauge the quality of gatekeeper decision-making in science has rarely been made publicly available. Our research tracks the popularity of rejected and accepted manuscripts at three elite medical journals. We found that editors and reviewers generally made good decisions regarding which manuscripts to promote and reject. However, many highly cited articles were surprisingly rejected. Our research suggests that evaluative strategies that increase the mean quality of published science may also increase the risk of rejecting unconventional or outstanding work.

Abstract

Peer review is the main institution responsible for the evaluation and gestation of scientific research. Although peer review is widely seen as vital to scientific evaluation, anecdotal evidence abounds of gatekeeping mistakes in leading journals, such as rejecting seminal contributions or accepting mediocre submissions. Systematic evidence regarding the effectiveness—or lack thereof—of scientific gatekeeping is scant, largely because access to rejected manuscripts from journals is rarely available. Using a dataset of 1,008 manuscripts submitted to three elite medical journals, we show differences in citation outcomes for articles that received different appraisals from editors and peer reviewers. Among rejected articles, desk-rejected manuscripts, deemed as unworthy of peer review by editors, received fewer citations than those sent for peer review. Among both rejected and accepted articles, manuscripts with lower scores from peer reviewers received relatively fewer citations when they were eventually published. However, hindsight reveals numerous questionable gatekeeping decisions. Of the 808 eventually published articles in our dataset, our three focal journals rejected many highly cited manuscripts, including the 14 most popular; roughly the top 2 percent. Of those 14 articles, 12 were desk-rejected. This finding raises concerns regarding whether peer review is ill-suited to recognize and gestate the most impactful ideas and research. Despite this finding, results show that in our case studies, on the whole, there was value added in peer review. Editors and peer reviewers generally—but not always—made good decisions regarding the identification and promotion of quality in scientific manuscripts.

Footnotes

↵¹To whom correspondence should be addressed. Email: ksiler{at}gmail.com.

Author contributions: K.S., K.L., and L.B. designed research; K.S., K.L., and L.B. performed research; K.S. analyzed data; and K.S. wrote the paper.
The authors declare no conflict of interest.
This article is a PNAS Direct Submission.
Data deposition: Our data are confidential and securely stored at the University of California, San Francisco.
This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1418218112/-/DCSupplemental.

View Full Text