While some sustainable development efforts will reduce emerging disease risk, others may increase it—suggesting important lessons for the coronavirus outbreak and beyond.
Image credit: Pixabay/sasint.
New Research In
Physical Sciences
Featured Portals
Articles by Topic
Biological Sciences
Featured Portals
Articles by Topic
Edited by Susan T. Fiske, Princeton University, Princeton, NJ, and approved February 5, 2018 (received for review August 23, 2017)
Significance
Scientific grant peer reviewers must differentiate the very best applications from comparatively weaker ones. Despite the importance of this determination in allocating funding, little research has explored how reviewers derive their assigned ratings for the applications they review or whether this assessment is consistent when the same application is evaluated by different sets of reviewers. We replicated the NIH peer-review process to examine the qualitative and quantitative judgments of different reviewers examining the same grant application. We found no agreement among reviewers in evaluating the same application. These findings highlight the subjectivity in reviewers’ evaluations of grant applications and underscore the difficulty in comparing the evaluations of different applications from different reviewers—which is how peer review actually unfolds.
Abstract
Obtaining grant funding from the National Institutes of Health (NIH) is increasingly competitive, as funding success rates have declined over the past decade. To allocate relatively scarce funds, scientific peer reviewers must differentiate the very best applications from comparatively weaker ones. Despite the importance of this determination, little research has explored how reviewers assign ratings to the applications they review and whether there is consistency in the reviewers’ evaluation of the same application. Replicating all aspects of the NIH peer-review process, we examined 43 individual reviewers’ ratings and written critiques of the same group of 25 NIH grant applications. Results showed no agreement among reviewers regarding the quality of the applications in either their qualitative or quantitative evaluations. Although all reviewers received the same instructions on how to rate applications and format their written critiques, we also found no agreement in how reviewers “translated” a given number of strengths and weaknesses into a numeric rating. It appeared that the outcome of the grant review depended more on the reviewer to whom the grant was assigned than the research proposed in the grant. This research replicates the NIH peer-review process to examine in detail the qualitative and quantitative judgments of different reviewers examining the same application, and our results have broad relevance for scientific grant peer review.
Footnotes
- ↵1To whom correspondence should be addressed. Email: epier{at}wisc.edu.
Author contributions: A.K., C.E.F., and M.C. designed research; A.K., J.R., C.E.F., and M.C. performed research; E.L.P. and A.F. coded data with input from M.J.N.; E.L.P. and M.B. analyzed data; and E.L.P. and M.B. wrote the paper with input from all coauthors.
The authors declare no conflict of interest.
This article is a PNAS Direct Submission.
This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1714379115/-/DCSupplemental.
Published under the PNAS license.
Low agreement among reviewers evaluating the same NIH grant applications
Sign up for Article Alerts
Article Classifications
- Social Sciences
- Social Sciences
Jump to section
You May Also be Interested in
QnAs with NAS member and bioengineer Sangeeta Bhatia.
Image courtesy of Howard Hughes Medical Institute/Scott Eisen.
An ancient bead-making settlement on a Florida coastal island was crucial to pre-Columbian economies.
Image courtesy of Terry E. Barbour and Kenneth E. Sassaman.
A beetle preserved in Burmese amber suggests early evidence of insect pollination of flowering plants.
Image courtesy of Bo Wang.
A study explores the pervasiveness of plastics in fish nurseries and suggests plastic accumulation in larval fish nurseries may affect marine ecosystems.
Image courtesy of Jonathan L. Whitney.