Language from police body camera footage shows racial disparities in officer respect

Study 1: Perceptions of Officer Treatment from Language.

We first test whether human raters can reliably judge respect from officers’ language, and whether these judgments reveal differences in officer respect toward black versus white community members.

Respect is a complex and gradient perception, incorporating elements of a number of correlated constructs like friendliness and formality. Therefore, in this study, we ask participants to rate transcribed utterances spoken by officers along five conceptually overlapping folk notions related to respect and officer treatment. We randomly sampled 414 unique officer utterances (1.1% of all usable utterances in the dataset) directed toward black (N = 312) or white (N = 102) community members. On each trial, participants viewed the text of an officer utterance, along with the driver’s utterance that immediately preceded it. All proper names and places were anonymized, and participants were not told the race or gender of the driver. Participants indicated on four-point Likert scales how respectful, polite, friendly, formal, and impartial the officer was in each exchange. Each utterance was rated by at least 10 participants.

Could participants reliably glean these qualities from such brief exchanges? Previous work has demonstrated that different perceivers can arrive at similar judgments from “thin slices” of behavior (22). In a similar vein, participants showed consistency in their perceptions of officer language, with reliability for each item ranging from moderate (Cronbach’s α = 0.73) to high (α = 0.91) agreement (see SI Appendix, Annotator Agreement). These results demonstrate that transcribed language provides a sufficient and consensual signal of officer communication, enough to gain a picture of the dynamics of an interaction at a given point in time.

To test whether participant ratings uncovered racial group differences, we averaged scores across raters to calculate a single rating on each dimension for each utterance, then built a linear mixed-effects regression model to estimate the fixed effect of community member race across interactions, controlling for variance of a random effect at the interaction level. Officer utterances directed toward black drivers were perceived as less respectful [b = −0.23, 95% confidence interval (−0.34, −0.11)], polite [b = −0.23 (−0.35, −0.12)], friendly [b = −0.24 (−0.36, −0.12)], formal [b = −0.16 (−0.30, −0.03)], and impartial [b = −0.26 (−0.39, −0.12)] than language directed toward white drivers (Fig. 1). These differences persisted even when controlling for the age and sex of the driver (see SI Appendix, Model Outputs for Each Rated Dimension).

• Download figure
• Open in new tab
• Download powerpoint

Fig. 1.

(Left) Differences in raw participant ratings between interactions with black and white community members. (Right) When collapsed to two uncorrelated components, Respect and Formality, we find a significant difference for Respect but none for Formality. Error bars represent 95% confidence intervals. PC, principal component.

Given the expected conceptual overlap in the five perceptual categories we presented to the participants, we used principal component analysis to decompose the ratings into their underlying components. Two principal components explained 93.2% of the variance in the data (see SI Appendix, Principal Component Analysis (PCA) Loadings for loadings). The first component, explaining 71.3% of the variance and composed of positive loadings on the impartial, respectful, friendly, and polite dimensions with some loading on the formal dimension, we characterize as Respect, broadly construed. The second, explaining 21.9% of the variance and composed primarily of a very high positive loading on the formal dimension and a weak negative loading on the friendly dimension, we characterize as Formality. This component captures formality as distinct from respect more generally, and is likely related to social distance.

Standardizing these factor scores as outcome variables in mixed-effects models, we find that officers were equal in Formality with white and black drivers [β = −0.01 (−0.19, 0.16)], but higher in Respect with white drivers [β = 0.17 (0.00, 0.33)] (Fig. 1).

Study 1 demonstrates that key features of police treatment can be reliably gleaned from officer speech. Participant ratings from thin slices of police–community interactions reveal racial disparities in how respectful, impartial, polite, friendly, and formal officers’ language to community members was perceived. Such differences were driven by differences in the Respect officers communicated toward drivers rather than the Formality with which officers addressed them.

Study 2: Linguistic Correlates of Respect.

The methods of study 1 (human coding of 414 individual utterances), although effective at discovering racial disparities in officer respect toward community members in our dataset, cannot offer a general solution to the analysis of body camera data. One problem is scale: Each year, on the order of 26 million vehicle stops are made (5). Furthermore, using only a small sample of individual utterances makes it impossible to study how police treatment varies over officers, or how the interaction progresses across time in each stop.

In this study, we therefore develop computational linguistic models of respect and formality and tune them on the 414 individual utterances; in study 3, we apply these models to our full dataset of 36,738 utterances. Our method is based on linguistic theories of respect that model how speakers use respectful language (apologizing, giving agency, softening of commands, etc.) to mitigate “face-threatening acts.” We use computational linguistic methods (e.g., refs. 23⇓⇓–26) to extract features of the language of each officer utterance. The log-transformed counts of these features are then used as independent variables in two linear regression models predicting the perceptual ratings of Respect and Formality from study 1.

Our model-assigned ratings agree with the average human from study 1 about as well as humans agree with each other. Our model for Respect obtains an adjusted R² of 0.258 on the perceptual ratings obtained in study 1, and a root-mean-square error (RMSE) of 0.840, compared with an RMSE of 0.842 for the average rater relative to other raters. Our model for Formality obtains an adjusted R² of 0.190, and an RMSE of 0.882 compared with 0.764 for the average rater (see SI Appendix, Model Comparison to Annotators for more details on how these values were calculated). These results indicate that, despite the sophisticated social and psychological cues participants are likely drawing upon in rating officers’ utterances, a constrained set of objectively measurable linguistic features can explain a meaningful portion of the variance in these ratings.

Fig. 2 lists the linguistic features that received significant weights in our model of Respect (arranged by their model coefficients). For example, apologizing, gratitude, and expressions of concern for citizen safety are all associated with respect. The bars on the right show the log-odds of the relative proportion of interactions in our dataset taken up by each feature, where negative numbers mean that a feature comprised a larger proportion of officers’ speech in interactions with black community members and positive numbers mean the same for interactions with white community members. Example utterances containing instances of the highest-weighted features for the Respect model are shown in Fig. 3. See SI Appendix, Study 2 for full regression outputs and more detailed discussion of particular linguistic findings.

• Download figure
• Open in new tab
• Download powerpoint

Fig. 2.

(Left) Respect weights assigned by final model to linguistic features and (Right) the corresponding log-odds of those features occurring in officer speech directed toward black versus white community members, calculated using Fisher’s exact test. ^†P < 0.1; ^∗P < 0.05; ^∗∗P < 0.01; ^∗∗∗P < 0.001.

• Download figure
• Open in new tab
• Download powerpoint

Fig. 3.

Sample sentences with automatically generated Respect scores. Features in blue have positive coefficients in the model and connote respect, such as offering reassurance (“no problem”) or mentioning community member well-being (“drive safe”). Features in red have negative coefficients in the model and connote disrespect, like informal titles (“my man”), or disfluencies (“that- that’s”).

Study 3: Racial Disparities in Respect.

Having demonstrated that people can reliably infer features of procedural justice from officer speech (study 1), and that these ratings can be reliably predicted from statistical models of linguistic features (study 2), we are now able to address our central question: Controlling for contextual factors of the interaction, is officers’ language more respectful when speaking to white as opposed to black community members?

We apply our models from study 2 to the entire corpus of transcribed interactions to generate predicted scores for Respect and Formality for each of the 36,738 utterances in our dataset. We then build linear mixed-effects models for Respect and Formality over these utterances. We include, as covariates in our primary model, community member race, age, and gender; officer race; whether a search was conducted; and the result of the stop (warning, citation, or arrest). We include random intercepts for interactions nested within officers.

Controlling for these contextual factors, utterances spoken by officers to white community members score higher in Respect [β = 0.05 (0.03, 0.08)]. Officer utterances were also higher in Respect when spoken to older [β = 0.07 (0.05, 0.09)] community members and when a citation was issued [β = 0.04 (0.02, 0.06)]; Respect was lower in stops where a search was conducted [β = −0.08 (−0.11, −0.05)]. Officer race did not contribute a significant effect. Furthermore, in an additional model on 965 stops for which geographic information was available, neither the crime rate nor density of businesses in the area of the stop were significant, although a higher crime rate was indicative of increased Formality [β = 0.03 (0.01, 0.05)].

One might consider the hypothesis that officers were less respectful when pulling over community members for more severe offenses. We tested this by running another model on a subset of 869 interactions for which we obtained ratings of offense severity on a four-point Likert scale from Oakland Police Department officers, including these ratings as a covariate in addition to those mentioned above. We found that the offense severity was not predictive of officer respect levels, and did not substantially change the results described above.

To consider whether this disparity persists in the most “everyday” interactions, we also reran our analyses on the subset of interactions that did not involve arrests or searches (N = 781), and found the results from our earlier models were fundamentally unchanged. Full regression tables for all models described above are given in SI Appendix, Study 3.

Another hypothesis is that the racial disparities might have been caused by officers being more formal to white community members, and more informal or colloquial to black community members. However, we found that race was not associated with the formality of officers’ utterances. Instead, utterances were higher in Formality in interactions with older [β = 0.05 (0.03, 0.07)] and female [β = 0.02 (0.00, 0.04)] community members.

Are the racial disparities in the respectfulness of officer speech we observe driven by a small number of officers? We calculated the officer-level difference between white and black stops for every officer (N = 90) in the dataset who had interactions with both blacks and whites (Fig. 4). We find a roughly normal distribution of these deltas for officers of all races. This contrasts with the case of stop-and-frisk, where individual outlier officers account for a substantial proportion of racial disparities (27); the disparities we observe here cannot be explained by a small number of extreme officers.

• Download figure
• Open in new tab
• Download powerpoint

Fig. 4.

Kernel density estimate of individual officer-level differences in Respect when talking to white as opposed to black community members, for the 90 officers in our dataset who have interactions with both blacks and whites. More positive numbers on the x axis represent a greater positive shift in Respect toward white community members.

Because our model is able to generate scores across all utterances in our dataset, we can also consider aspects of the trajectory of interactions beyond the mean level of respect (Fig. 5). Growth-curve analyses revealed that officers spoke with greater Respect [b = 0.35 (0.29, 0.40)] and reduced Formality [b = −0.57 (−0.62, −0.53)] as interactions progressed. However, these trajectories varied by community member race: Although stops of white and black drivers converged in the Formality expressed during the interaction [b = −0.09 (−0.13, −0.05)], the gap in Respect increased over time [b = 0.10 (0.05, 0.15)]. That is, officer Respect increased more quickly in interactions with white drivers [b = 0.45 (0.38, 0.54)] than in interactions with black drivers [b =  0.24 (0.19, 0.29)].

• Download figure
• Open in new tab
• Download powerpoint

Fig. 5.

Loess-smoothed estimates of the (Left) Respect and (Right) Formality of officers’ utterances relative to the point in an interaction at which they occur. Respect tends to start low and increase over an interaction, whereas the opposite is true for Formality. The race discrepancy in Respect is consistent throughout the interactions in our dataset.

Discussion.

Despite the formative role officer respect plays in establishing or eroding police legitimacy (7), it has been impossible to measure how police officers communicate with the public, let alone gauge racial disparities in officer respect. However, body-worn cameras capture such interactions every day. Computational linguistic techniques let us examine police–community contacts in a manner powerful enough to scale to any number of interactions, but sensitive enough to capture the interpersonal qualities that matter to the police and public alike.

In doing so, we first showed that people make consistent judgments about such interactions from officers’ language, and we identified two underlying, uncorrelated constructs perceived by participants: Respect and Formality. We then built computational linguistic models of these constructs, identifying crucial positive and negative politeness strategies in the police–community interactional context. Applying these models to an entire month of vehicle stops, we showed strong evidence for racial disparities in Respect, but not in Formality: Officers’ language is less respectful when speaking to black community members.

Indeed, we find that white community members are 57% more likely to hear an officer say one of the most respectful utterances in our dataset, whereas black community members are 61% more likely to hear an officer say one of the least respectful utterances in our dataset. (Here we define the top 10% of utterances to be most respectful and the bottom 10% to be least respectful.)

This work demonstrates the power of body camera footage as an important source of data, not just as evidence, addressing limitations with methodologies that rely on citizens’ recollection of past interactions (10) or direct researcher observation of police behavior (17⇓⇓–20). However, studying body camera footage presents numerous hurdles, including privacy concerns and the raw scale of the data. The computational linguistic models presented here offer a path toward addressing both these concerns, allowing for the analysis of transcribed datasets of any size, and generating reliable ratings of respect automatically. These models have the potential to allow for useful information about an interaction to be extracted while maintaining officer and community member privacy.

The racial disparities in officer respect are clear and consistent, yet the causes of these disparities are less clear. It is certainly possible that some of these disparities are prompted by the language and behavior of the community members themselves, particularly as historical tensions in Oakland and preexisting beliefs about the legitimacy of the police may induce fear, anger, or stereotype threat. However, community member speech cannot be the sole cause of these disparities. Study 1 found racial disparities in police language even when annotators judged that language in the context of the community member’s utterances. We observe racial disparities in officer respect even in police utterances from the initial 5% of an interaction, suggesting that officers speak differently to community members of different races even before the driver has had the opportunity to say much at all.

Regardless of cause, we have found that police officers’ interactions with blacks tend to be more fraught, not only in terms of disproportionate outcomes (as previous work has shown) but also interpersonally, even when no arrest is made and no use of force occurs. These disparities could have adverse downstream effects, as experiences of respect or disrespect in personal interactions with police officers play a central role in community members’ judgments of how procedurally fair the police are as an institution, as well as the community’s willingness to support or cooperate with the police (8, 9).

We now have a method for quantifying these troubled interactions. Although the circumstances of any particular stop can vary dramatically, our approach allows us to measure aggregate department-level trends, revealing disparities across hundreds of interactions. These disparities are part of a constellation of differences in officer language spoken toward black versus white community members; a simple classifier trained on only the words used by officers is able to correctly predict the race of the community member in over two thirds of the interactions (see SI Appendix, Linguistic Classification Accuracy of Race).

Future research could expand body camera analysis beyond text to include information from the audio such as speech intonation and emotional prosody, and video, such as the citizen’s facial expressions and body movement, offering even more insight into how interactions progress and can sometimes go awry. In addition, footage analysis could help us better understand what linguistic acts lead interactions to go well, which can inform police training and quantify its impacts over time.

The studies presented here open a path toward these future opportunities and represent an important area of research for the study of policing: Computational, large-scale analyses of language give us a way to examine and improve police–community interaction that we have never had before.