The effects of communicating uncertainty on public trust in facts and numbers

Significance Does openly communicating uncertainty around facts and numbers necessarily undermine audiences’ trust in the facts, or the communicators? Despite concerns among scientists, experts, and journalists, this has not been studied extensively. In four experiments and one field experiment on the BBC News website, words and numerical ranges were used to communicate uncertainty in news article-like texts. The texts included contested topics such as climate change and immigration statistics. While people’s prior beliefs about topics influenced their trust in the facts, they did not influence how people responded to the uncertainty being communicated. Communicating uncertainty numerically only exerted a minor effect on trust. Knowing this should allow academics and science communicators to be more transparent about the limits of human knowledge.


Overview of participant characteristics for and within each experiment
Note. N = sample size; % women = percentage women in sample; age M (SD) = mean age and standard deviation; % higher education = percentage of people who indicated they had attained tertiary education (a Bachelors, Masters, or doctoral degree, or equivalent); Numeracy scores ranged from 1 to 4, M (SD) = mean numeracy and standard deviation. Numeracy was not assessed in Experiment 5. To compare the education level in our samples to the national population in the UK: recent OECD data about education level in the UK shows that 45.7% of the 24-65-year-olds in the UK attainted tertiary education (Bachelors, Masters, PhD, etc.); 18.8% attained primary and middle school education, and 35.4% upper secondary education (GSCE & A-levels). Note. Participants were recruited via Prolific Academic. n = size of subsample; % women = percentage women in subsample; age M (SD) = mean age and standard deviation in subsample; % higher education = percentage of people in the subsample who indicated they had obtained a Bachelors, Masters, or doctoral degree, or equivalent; Numeracy M (SD) = mean numeracy and standard deviation in subsample. Note. Participants were recruited via Prolific Academic. n = size of subsample; % women = percentage women in subsample; age M (SD) = mean age and standard deviation in subsample; % higher education = percentage of people in the subsample who indicated they had obtained a Bachelors, Masters, or doctoral degree, or equivalent; Numeracy M (SD) = mean numeracy and standard deviation in subsample. Note. Participants were recruited via Prolific Academic. n = size of subsample; % women = percentage women in subsample; age M (SD) = mean age and standard deviation in subsample; % higher education = percentage of people in the subsample who indicated they had obtained a Bachelors, Masters, or doctoral degree, or equivalent; Numeracy M (SD) = mean numeracy and standard deviation in subsample. Note. Participants were recruited by Qualtrics Panels. n = size of subsample; % women = percentage women in subsample; age M (SD) = mean age and standard deviation in subsample; % higher education = percentage of people in the subsample who indicated they had obtained a Bachelors, Masters, or doctoral degree, or equivalent; Numeracy M (SD) = mean numeracy and standard deviation in subsample.
condition of the design of field Experiment 5.  Additional variables. After measuring these prior beliefs, we presented participants with one of 9 manipulation texts, and subsequently asked them to indicate their current affective state on a feeling thermometer, to recall the number they had just read and whether there was any uncertainty presented around it; after which we assessed our key dependent variables (as described in the Method in the main text). After this, we included a series of variables in our experiment for exploratory purposes that fall outside of the scope of this paper. Among these variables, we assessed participants' mood with 5 positive ( = .81, e.g. "inspired") and 5 negative ( = .87, e.g. "upset") items (4), scale from 1 = very slightly or not at all, to 5 = extremely), the results of which are reported below in the Additional Results section. We also included several additional variables for exploratory purposes in the survey that fall outside the scope of this paper and thus not further reported here.
Demographic variables. At the end of the survey, we asked participants about their age, gender, nationality, education level, vote in the Brexit referendum, and political orientation ("Here is a 7-point scale on which the political views that people might hold are arranged from very liberal to very conservative. Where would you place yourself on this scale?" from 1 = very liberal to 7 = very conservative).

Additional Results
In this first experiment, 1122 participants read a short text about one of three topics (tigers in India, climate science, or unemployment), that contained either no uncertainty  Figure S1 presents the results. As reported in the main text, participants perceived the number to be significantly more uncertain when uncertainty was communicated to them in a verbal format, as compared to numerical or no uncertainty.
Inspecting means per condition revealed that the interaction effect was driven by how people responded to uncertainty as a numerical range around the climate science estimate. Whereas participants perceived the unemployment and tiger numbers to be significantly more uncertain when uncertainty was communicated numerically as compared to control, and again when it was communicated verbally (see Figure S1  Trust in the number. We also asked people to indicate how reliable and trustworthy they thought the numbers were. Trust in the source. In addition, we asked people how trustworthy they thought "the writers of the report" were. A two-way ANOVA again showed a main effect of uncertainty communication format (F(2, 1113) = 11.66, p < .001; 2 = .02) and of topic (F(2, 1113) = 24.56, p < .001; 2 = .04), and no interaction. Panel C of Figure S1   These results suggest that while people's affective reactions were influenced by the topic about which they read, uncertainty communication did not influence mood or affective reactions; neither directly after the manipulation, nor later in the survey. In Experiments 2, 3, and 4, we continued to include the feeling thermometer directly after the manipulation to assess immediate people's affective response, but we did not assess mood in these studies.

Additional Methodological Information
Participants first answered questions about their belief about unemployment, satisfaction with the (economic) state of the country (r = .71), and Zeitgeist of societal discontent ( = .85; same measures as Experiment 1). Subsequently, they were presented with one of seven manipulation texts, after which we asked them to indicate how they felt on a feeling thermometer, we assessed comprehension checks, and our key dependent variables, with the same materials as in Experiment 1 (reported in the main text). We added one item to our set of key dependent variables that measured perceived reliability of government statistics ("To what extent do you think government statistics are reliable?" on a scale from 1 = not at all to 7 = very reliable). Just as in Experiment 1, we included additional variables for exploratory purposes in the survey that fall outside the scope of this paper and thus not further reported here. The survey again finished with the assessment of the same demographic variables.

Additional Results
Replicating Experiment 1. The control condition and "original magnitude" numerical and verbal uncertainty conditions of Experiment 2 enabled us to conduct a direct test of whether the results Experiment 1 would replicate. We found that for two of the three key dependent variables, the results of Experiment 2 were a direct replication of the results of           This puts the total number of people who are unemployed at 1.42 million.
The number of those in work increased and wage growth improved over the same period. However, weak incomes have been a problem for a decade. "It will take a long period of wages rising above the rate of inflation for people to feel significantly better off" one economics commentator is quoted as saying.

Additional Results
Affective response. Just as in Experiment 1 and 2, we assessed people's affective state with a feeling thermometer directly after the manipulation. A one-way ANOVA testing for differences between different formats of uncertainty communication showed no effect of format on people's reported affect, (F(7, 1191) = 0.80, p = .59). Again, different formats of uncertainty communication (including no uncertainty communication, the control condition) did not result in differences in how people reported the information made them feel.  Net migration is the difference between the number of people coming to live in the UK for at least 12 months and those emigrating. The 2017 overall net migration figure (both from the EU and non-EU countries) is also down, from record highs in 2015 and early 2016.
However, "The figures show that the government remains a long way off from meeting its objective to cut overall net migration, EU and non-EU, to the tens of thousands" one Home Affairs correspondent is quoted as saying." Numerical range with point estimate …101,000 (range between 68,000 and 132,000)… Numerical point estimate +/two standard errors …101,000 (+/-33,000)… Verbal explicit uncertainty statement …101,000 more people coming to the UK from the EU than leaving in 2017. The report states there is uncertainty around the exact figure -it could be higher or lower. […] Verbal uncertainty word … around 101,000…

Additional Results
Preregistered hypotheses and results. We preregistered our hypotheses for Experiment 4 on aspredicted.org (http://aspredicted.org/blind.php?x=d3xu67). This section will present each of these hypotheses in turn and discuss the level of support the experiment provides for each of the pre-registered hypotheses. These results were thus again mostly but not fully in line with our hypothesis: we did find the expected reduction in trust in numbers for the explicit verbal statement compared to control, and no effect for the numerical range; but the difference between trust in numbers for the "+/-standard errors" format and control condition was not significant in this study. We interpret these findings and the findings from Experiment 3 to indicate that words like "around" and "estimated" do not effectively communicate uncertainty, as we did not find differences in perceived uncertainty of the numbers. The difference between the control condition and the word "around" was not significant, but the trend was nonetheless interesting: using the word "around" descriptively made people feel

Mediation Analyses
Although this was not the main aim of the present research, following comments from one of the reviewers, we also explored the mechanism behind people's reactions to uncertainty communication. As a first step towards this aim, we conducted exploratory mediation analyses to examine whether the effect of being exposed to uncertainty communication on people's trust in the number was mediated by people's perceptions of uncertainty. We ran mediation models using the PROCESS macro for SPSS (5) with 10,000 bootstrap samples.
For the effect of communicating uncertainty (as a numeric range, numeric +/-, and explicit verbal statements) compared to not communicating uncertainty (control condition) on trust in the number, the analysis showed evidence for full mediation through perceived uncertainty of the number (see Figure S3) To further examine this mediation effect, we tested it specifically for two formats of uncertainty communication: as a numeric range (vs. control) and as an explicit verbal statement (vs. control). While the total effect of communicating a numeric range (vs. control) on trust in the number was not significant, the results presented in Figure S4  In sum, these results indicate that in the aggregate, there is a negative effect of uncertainty communication on trust in the number because people perceive more uncertainty following its communication.

Experiment 5: Field experiment Pilot study
We conducted a pilot study on the live BBC News website to gain experience with the technical and journalistic context of the field experiment, in collaboration with BBC News online. On the 10 th of September 2019, we used the ONS' release of the Labour Market Statistics bulletin as the context for the pilot study. The journalistic team for wrote a news article based on the ONS release with a focus on wage growth, and we worked with the journalistic team and Head of Statistics to select a target figure to communicate uncertainty around before the news article was published on the website. For this pilot study, that was earning growth (excluding bonuses) and was the first figure communicated in the article: "Earnings excluding bonuses grew at an estimated annual pace of 3.8% (between 3.3% and 4.3%) in the May to July period, down slightly from the previous reading." The pilot study had the same design as the field experiment: readers of the website were shown one of three versions of the article, either without additional uncertainty (control condition), with uncertainty communicated as a verbal cue ("estimated 3.8%") or with uncertainty communicated as a numerical range ("3.8% (between 3.3% and 4.3%)"). Participants were asked the same questions as reported for Experiment 5, adapted for this target number ("the earnings figure in the story").
Because this was a pilot study, we decided to start the experiment after the first release in the first update of the story, which meant that we missed much of the morning traffic on the website. In addition, technical issues meant that all users of the BBC News mobile application were shown only the control condition; the experimental conditions were only shown to users of the website in a computer browser. We ended data collection after 48 hours with a total of 589 participants: 366 in the control condition, 115 in the numeric condition, and 108 in the verbal condition. Total sample consists of 503 male and 70 female participants (2 other, 14 missing), with a mean age of 49.16 (SD = 15.41). Education level was distributed as follows: No school = 1.4%, School = 26.5%, Bachelor degree = 37%, Higher degree = 33.4%.

Results.
A series of ANOVAs showed no significant differences between uncertainty communication conditions for any of the outcomes.
Pilot experiment news article text (first paragraph):

Wage growth stays strong as unemployment falls
Wages have continued to grow at a strong pace and employment remains at record highs, official figures show.
Earnings excluding bonuses grew at an estimated annual pace of 3.8% (between 3.3% and 4.3%) in the May to July period, down slightly from the previous reading.
Including bonuses, wages rose at an annual pace of 4% -the highest rate since mid-2008.
The unemployment rate dipped to 3.8%, while the estimated employment rate remained at a record 76.1%.
Click here to take part in a short study about this article run by the University of Cambridge.

Additional Results
In addition to the measures described in the main text, participants answered a comprehension