Reconciling disagreement over climate–conflict results in Africa

Results

In a multipronged critique of Burke et al. (1), Buhaug (8) displays 12 alternative estimates (models 2–13 in Buhaug) presented across three tables that are incorrectly compared with Burke et al.’s benchmark model (model 1 in Buhaug). Of the four critical errors listed above, i invalidates Buhaug’s inferences based on Buhaug’s table 1 (denoted BT1), ii and iii invalidate Buhaug’s inferences based on Buhaug’s table 2 (denoted BT2), and i, ii, iii, and iv invalidate Buhaug’s inferences based on Buhaug’s table 3 (denoted BT3). We replicate and examine BT1–BT3 separately and respectively in Tables 1–3 of this article.

In BT1, Buhaug (8) removes country-fixed effects and country-specific trends from the statistical model used in Burke et al. (1), terms that were included in the original analysis to account for all time-invariant and linearly trending confounding variables. Buhaug infers from BT1 that the finding of Burke et al. is incorrect because the parameter estimates of the model change in magnitude and in significance; however, this interpretation does not logically follow from the results in BT1. Drawing inferences from a model without fixed effects and trends requires stronger assumptions about African countries than were originally assumed by Burke et al. Specifically, it assumes that the average rate of conflict is the same for all countries in sub-Saharan Africa and that the conflict rate do not exhibit a trend (common or country-specific) over time. Thus, Buhaug’s approach in BT1 implicitly requires the assumption that all countries in sub-Saharan Africa are comparable over space and time in the factors that influence conflict risk, which may include cultural patterns, geopolitics, natural resources, colonial history, international trade patterns, government policies, and geographic constraints. This assumption is not tested by Buhaug. A formal test of Buhaug’s assumption that countries have the same average risk of conflict and/or trends in conflict (i.e., country-fixed effects and/or country-specific trends all equal zero) is the calculation of a F-statistic that jointly tests whether average conflict risk and/or trends in conflict are statistically different from zero (13, pp. 150–157). In Table 1 we replicate the results in BT1 and conduct this test on the relevant terms in the benchmark model of Burke et al. We reject the null hypotheses that country-fixed effects are jointly zero (P < 0.0001), that country-specific trends are jointly zero (P < 0.0001), and that country-fixed effects and country-specific trends are jointly zero (P < 0.0001). (Because of limited degrees of freedom, we are mechanically unable to conduct joint F-tests with SEs clustered at the country level. However, estimates using country-clustered SEs are nearly identical in magnitude to heteroscedastic-robust SEs. By necessity, we use these latter values to conduct a F-test. The replication file is provided in Supporting Information.) These results strongly reject the assumption that models 2–4 in BT1 are correctly specified. The misspecification of these models is likely to alter Buhaug’s parameter estimates relative to the Burke et al. finding regardless of whether or not the Burke et al. result is correct, so the observation that these parameter estimates differ from Burke et al. does not provide grounds to reject the Burke et al. result.

In BT2, the definition of conflict used in Burke et al. (1), the incidence of war years, is compared with models that examine alternative definitions of conflict incidence and various measures of conflict outbreak. No justification is provided for why these alternative variables should respond to temperature similarly to the incidence of war years. However, if we assume that such a justification exists, then this comparison should be made properly. The coefficients presented in BT2 represent changes in the probability of observing a conflict event; however, the different types of conflict events occur with very different likelihoods (Table 2) and thus must be standardized before changes in these likelihoods can be compared. The most likely form of conflict is termed “incidence 25+” in Buhaug (8) and occurs with an unconditional probability of 0.254 in the sample (roughly once every 4 y), whereas the least likely is termed “outbreak +1000” and occurs with probability 0.012 (roughly once every 100 y). Comparing changes in probability for events that differ this much in their underlying likelihood is an apples-to-oranges comparison, because a probability change of 0.01 for incidence 25+ is a 4% change in the average risk of this event, whereas a 0.01 probability change for outbreak +1000 is an 83% change. We correct for the large differences in the underlying risk of these different forms of conflict by converting Buhaug’s results into units of percentage change in the likelihood of conflict per 1 °C change in temperature. We do this by dividing each dependent variable by its average likelihood of occurring before running the regressions in BT2, essentially converting outcomes into units of relative risk that permit a valid apples-to-apples comparison across different outcome measures (6). We replicate the results of BT2 in Table 2 following this standardization. Standardization makes it clear that the results in Buhaug exhibit very large uncertainties, with two 95% confidence intervals that span effects ranging from −100% to +100% per 1 °C. This high level of sampling variability might explain why none of these coefficients were statistically significant in Buhaug; however, this uncertainty must also be accounted for when comparing these results to Burke et al., as we do below.

In Table 2, we test whether the results in each of these models is different from the main result presented in Burke et al. (1) using seemingly unrelated regression (SUR), an approach that allows us to formally test whether or not two different regression models return statistically different results (14) (also 15, p. 153). Intuitively, this approach asks whether the “regression lines” describing the relationship between temperature and conflict in Burke et al.’s and Buhaug’s (8) analyses are statistically different from one another, while taking into account the fact that the studies are using related conflict outcomes where disturbances may be correlated. This test is necessary because it is extremely unlikely that any two studies using different data sets will recover identical results, due to sampling variability, even if the true underlying relationship is the same for both studies—so observing that two regression results are not identical is not sufficient evidence to conclude that the underlying relationship is different. Instead, we must ask whether the difference in the regression results is large enough that it is unlikely to be caused by sampling variability alone. This approach differs markedly from the analysis in Buhaug, where it was claimed that new results overturned the findings of Burke et al. without conducting any formal statistical comparisons between the two sets of models. We correct this error by testing whether the effect of current temperature, the effects of current and lagged temperature (jointly), and the effects of all weather variables (jointly) on alternative conflict measures from Buhaug differ significantly from the result reported in Burke et al. As shown in Table 2, of these 15 comparisons, only one (the effect of current temperature on incidence 1000+) is marginally significant at the 10% level. Thus, we fail to reject the hypothesis that Buhaug’s results are different from Burke et al.’s result because the magnitude of observed differences would be expected based on sampling variability alone. We conclude that once the variables in BT2 are standardized and intermodel comparisons are correctly implemented, the results in BT2 provide no support for the claim that the results of Buhaug are different from those of Burke et al.

In BT3, a logistic regression lacking country-fixed effects and country-specific trends is presented using an alternative measure of conflict. The effect of temperature is again not directly compared with the results of Burke et al. (1). Furthermore, the effect of temperature is displayed in raw coefficients from a nonlinear logit regression, which are difficult to interpret in terms of conflict risk and across conflict measures with different underlying likelihoods. To make the effect on this alternative conflict measure comparable and interpretable in terms of conflict risk, we replicate the main results from BT3 but report the estimated effects of temperature in terms of relative risk ratios in Table 3. This conversion makes it immediately clear that the results reported in BT3 are not statistically different from the benchmark result in Burke et al., because the Burke et al. result is contained within all four confidence intervals reported in BT3. Moreover, under all four models the upper bound of the 95% confidence interval for the effect of +1 °C dramatically exceeds the estimate reported in Burke et al., indicating that the relative risk for conflict may rise as much as 547 (model 12, the lowest upper bound) and 14.5 duodecillion (model 13, the highest upper bound). We think it is implausible that the true effect of temperature on conflict can be as large as these upper bounds suggest. Rather it seems more likely that the models in BT3 are not properly specified. Regardless, these estimates do not indicate that the relative risk ratio of 1.39 implied by Burke et al. is too large.