Scientific versus public debates: A PNAS case study

Unfortunately, scientific communication and the resulting public use of research often do not reflect the painstaking and sometimes imperfect process of peer review, and in a hyperpartisan landscape invalid conclusions can acquire a tendentious life of their own. Here we review the process by which a controversial PNAS paper was published, the correction to errors that were uncovered after publication, and how the scientific process missed the original error but ultimately led to a Correction.*

We’re providing this explanation now because the July 2019 paper (1), which purportedly focused on the degree of racial bias documented in officer-involved shootings, has garnered new attention in light of the recent tragedy involving George Floyd and the massive protests that have followed. A June 2, 2020, Wall Street Journal op-ed (2) cited the PNAS article but failed to provide context or note the original paper’s unjustified supposition. Elucidating the genesis of this PNAS paper, and its misstatements, should help to set the record straight. We think doing so is essential so that future scholarship—and future efforts in science communication related to data on police activities—can avoid similar pitfalls on these sensitive and important issues.

Scientific publication operates via a peer review process in which submitted papers are assessed by knowledgeable specialists who critically evaluate the validity of the research, the reliability of the data, and the soundness of the conclusions drawn from them. As in any human endeavor, the peer review process is not perfect, and sometimes ill-founded results and conclusions are published. However, publication in a peer-reviewed journal subjects the findings to even wider critical scrutiny by scientists who are free to challenge the paper’s methods and conclusions and correct the scientific record.

Last July, PNAS published an article titled “Officer characteristics and racial disparities in fatal officer-involved shootings” by David J. Johnson, Trevor Ness, Nicole Burkel, Carley Taylor, and Joseph Cesario (1). A PNAS Editorial Board member edited the paper and two experts in the field positively reviewed it. Soon after the paper was published, PNAS received several Letters pointing out apparent errors in the study. The Letters included one by the political scientists Dean Knox and Jonathan Mummolo (3) and another by psychologists Ulrich Schimmack and Rickard Carlsson (4). Following standard procedures, and after a round of revision, the Letters were sent to the original authors to solicit a Reply, and PNAS published both the Letters and a Reply by two of the original authors (Johnson and Cesario) in January 2020 (5).

The Letters pointed to a glaring mistake—not in the body of the paper, but in the significance statement the authors published as part of the article. Every PNAS research article is accompanied by a significance statement, which is required to be “written at a level understandable to an undergraduate educated scientist outside their field of specialty.” As described in the Information for Authors, “the primary goal of the significance statement is to explain the relevance of the work in broad context to a broad readership” (https://www.pnas.org/page/authors/format).

In this case, the significance statement erroneously asserted that “white officers are not more likely to shoot minority civilians than nonwhite officers.” Unfortunately, the Johnson et al. article did not contain the data to address this question, and the analyses did not allow the authors to conclude this, as both Letters to the Editor strongly argued. This one sentence is the glaring error in the paper. But in retrospect, Johnson et al. also framed their literature review and interpreted their findings in ways that clearly did not correspond to the data they had analyzed and presented.

Johnson and colleagues presented the paper as if it were part of a literature that examines whether the police harm Blacks at a higher rate than Whites, and whether police officers are racially biased in fatal shootings. The literature they cited is concerned with police behavior, but the data they analyzed are not informative about any aspect of police behavior. They only presented data on the race of people fatally shot by the police, which have nothing to say about the likelihood that an officer will engage in a fatal shooting or how this likelihood might vary across suspect characteristics.

Johnson et al. agreed with their critics that the data they presented are, in fact, uninformative about racial bias in policing. The analysis included no information on nonfatal police encounters or the racial composition of the settings in which fatal encounters occur. The paper was about the racial composition of the victim pool, not a contribution to the debate about racial bias in police encounters. To calculate the likelihood that a White officer fatally shot a Black suspect we need to know how many Black suspects were approached and how many of these approaches led to a fatal shooting. To determine bias, we would then compare this figure to how many White suspects were shot after being approached by a White officer. Johnson et al. only had data on the number fatally shot, not the number approached.

After PNAS published the Letters and the authors’ Reply, the Editorial Board concluded that while the record had been set straight, grasping the entirety of the correction and its progression through multiple steps was too hard for readers to comprehend without reading all of the elements of the correspondence. PNAS then asked one of the original reviewers to examine the article again to determine if there were other errors. This reviewer found that the significance statement was the main error. The authors were then asked for an official Correction of the original article, and it was published on April 13, 2020 (see Correction in ref. 1). On the PNAS website, the original article, the two critiques, a Correction, and the author’s Reply are highlighted before the abstract.

Even though the original mistake was corrected and a responsible reader would see that the paper does not speak to differing rates of White or Black officers killing Black civilians, a Wall Street Journal op-ed published on June 2, 2020, by Heather Mac Donald cited the paper prominently in a piece titled “The Myth of Systemic Police Racism” (2). Coming after the police killing of George Floyd and the national outpouring of anguish and protest over a long line of deaths of African Americans in police encounters, the op-ed prompted renewed attention to the original article and the ensuing debate, and PNAS received emails asking for a retraction of the original article.

In response to these requests, the Editorial Board took a further step and asked two National Academy of Sciences members who are experts on crime and statistics to examine the original article. Both agreed that the Johnson et al. significance statement was incorrect and that the Correction was warranted. They also agreed that the authors poorly framed the article, the data examined were poorly matched, and that the analysis only addressed the racial composition of the victims of fatal police shootings in the year 2015. The paper itself does not contain fabricated data or serious statistical errors warranting a retraction. It does, unfortunately, address a question with much less public policy relevance than originally claimed.

PNAS is grateful for all of the social scientists involved in debating, reviewing, and ultimately correcting the error that did occur during the publication process. Social scientists, editors, reviewers, and readers are all human beings who make mistakes. Science progresses because of the multiple checks and balances entailed in the process of peer review, publication, data sharing, and replication.

The process could have more rapidly identified the need for a Correction, and the original review should have caught the disparities between the framing, the data, and the conclusions. The problem that exists now, however, is outside the realm of science. It has to do with the misinterpretation and partisan political use of a scientific article after its publication. The tendentious misappropriation of scientific findings is a challenge scientists increasingly face in today’s polarized media environment, and this example underscores the duty of scientists not only to adhere to the precepts of the scientific method, but to communicate their results to the public clearly, accurately, and impartially—and to engage those debates in which their findings are misconstrued, misinterpreted, or misused for partisan purposes.