Heterogeneity in effect size estimates

To gauge the extent of heterogeneity in empirical social science research, we review heterogeneity estimates—reestimated using random-effects meta-analysis—of published crowd science projects. We outline the inclusion criteria and the estimation procedures in the Materials and Methods section; details about the individual studies are provided in SI Appendix, sections 1–3. The results are illustrated in Fig. 2, which also indicates benchmarks for low (I² = 25%, H = 1.15), medium (I² = 50%, H = 1.41), and high heterogeneity (I² = 75%, H = 2.00) as commonly used in meta-analysis to classify the magnitude of heterogeneity (48, 49). Estimates of the heterogeneity measures τ, I², and H (together with their corresponding 95% CIs) and the results of Cochran’s Q-test for each meta-analysis reviewed in our empirical analysis are tabulated in SI Appendix, Table S1.

Population heterogeneity can be measured by implementing the same research design and analysis in independent samples drawn from different populations and estimating the SD in true effect sizes across samples (τ) in a random-effects meta-analysis. This is what has been pioneered in the ManyLabs (ML) replication studies and various Registered Replication Reports (RRRs) in psychology, which are ideal for measuring population heterogeneity. Our analysis involves four ML studies (54–57) and nine RRRs (58–66). As some of the included studies report results for multiple effects, our sample of studies isolating population heterogeneity comprises 70 meta-analyses.

The estimated population heterogeneity varies substantially across the meta-analyses in our sample, with a large number of estimates (19/70 = 27.1%) indicating homogeneity (H = 1.00) and some estimates unveiling substantial heterogeneity of up to H = 3.91 (with 4/70 estimates exceeding the threshold value of H = 2.00, indicative of large heterogeneity). The median H across the 70 meta-analyses is 1.08; a large fraction of the estimates are in the small to moderate heterogeneity range. Cochran’s Q-test rejects the null hypothesis of homogeneity at the 5% level for 21 (30%) of the sampled meta-analyses and at the 0.5% level for 14 (20%) of the sampled meta-analyses (42). Some meta-analyses (46/70 in 4/13 papers) are based on effect sizes measured in terms of Cohen’s d; heterogeneity can be reasonably compared across studies in absolute terms (τ) for this subsample. The estimated τ varies between 0.00 and 0.69 for these estimates, with a median of 0.06. Note that the distributions of H and τ estimates of population heterogeneity are subject to some upward bias due to following the convention to truncate τ² estimates at zero (i.e., to prevent the identification of excess homogeneity) (46, 47). For genuinely homogeneous effect sizes, randomness would lead to both negative and positive estimates of τ². This upward bias can be substantial whenever the fraction of meta-analyses with zero estimated heterogeneity is large—as is the case for our sample of studies isolating population heterogeneity.

Design heterogeneity can be measured by randomly allocating experimental participants sampled from the same population to different research designs while holding the analysis constant and estimating the SD in true effect sizes across research designs (τ) in a random-effects meta-analysis. We identified two studies that implemented such a research design, reporting the results of 11 meta-analytic estimates for six empirical claims: Landy et al. (67) tested five hypotheses on moral judgments, negotiations, and implicit cognition in 12 to 13 experimental designs each (once in a “main study” and once in a replication). Huber et al. (68) examined the effect of competition on moral behavior across 45 crowd-sourced experimental protocols. The estimates of H for the 11 meta-analyses reported in the sampled studies (Fig. 2) suggest that the extent of design heterogeneity is substantial. The estimates of H vary between 1.92 and 10.44, with a median of 3.36, and Cochran’s Q-test rejects the null hypothesis of homogeneity (P < 0.005) for each of the 11 meta-analyses. In our data, design heterogeneity is substantially larger than population heterogeneity. Design heterogeneity of this magnitude adds substantial uncertainty to estimates based on an individual design and implies low generalizability across study designs. All estimates of design heterogeneity are in Cohen’s d units: The estimated τ varies between 0.14 and 0.78, with a median of 0.23.

An effective means to estimate analytical heterogeneity involves randomly allocating independent analysts to test the same hypothesis on mutually exclusive random subsamples of a dataset and estimate the SD in true effect sizes across analysts (τ) in a random-effects meta-analysis. To the best of our knowledge, no studies have employed this method yet. Studies closest to this ideal are those relying on the multianalyst approach, where different analysts independently test the same hypothesis on the same data. Our review involves three papers that fulfill our inclusion criteria (see Materials and Methods for details), examining the variability in effect sizes due to analytical flexibility for five hypotheses (69–71).

As the analysts in multianalyst studies are required to estimate the effect in question using the same data, the analysts’ individual estimates are not independent. Despite the violation of the model assumptions, we use random-effects meta-analyses to estimate τ and H as an approximation of the analytical heterogeneity for the sampled multianalyst studies. A recent multianalyst study in biology (72) also used this method to estimate the heterogeneity of results across analysts. The estimates reported in Fig. 2 should be interpreted cautiously since relying on the random-effects meta-analytic model will underestimate heterogeneity for correlated observations (as the within-study variation will be lower in the case of dependent observations). The estimated analytical heterogeneity is large in our data, with H estimates ranging from 1.72 to 12.69, with a median of 4.08. Cochran’s Q-test is statistically significant (P < 0.005) for each of the five meta-analyses. In the Materials and Methods section, we provide a robustness test on the estimates of analytical heterogeneity, showing that the high estimate of 12.69 for Silberzahn et al. (70) is driven by two outliers in terms of sampling variance; removing these outliers reduces the estimated heterogeneity factor H to 2.72. In the sampled studies, analytical heterogeneity is substantial, in the same ballpark as the estimates for design heterogeneity, and substantially larger than our estimate of population heterogeneity. None of the estimates of analytical heterogeneity are in standardized effect size units, and the analytical heterogeneity estimates cannot be reasonably compared in absolute terms across the studies.

Our empirical estimations come with several limitations and should be interpreted with due care. First, it is worth emphasizing that the estimated heterogeneity factors H carry significant uncertainty for all three types of heterogeneity: Heterogeneity estimates can be highly sensitive to outliers in effect sizes and sample variances of the individual studies included in the meta-analyses [as illustrated above for the study by Silberzahn et al. (70)]. Therefore, the individual estimates of H in Fig. 1 should be interpreted with great caution. Second, all studies included in our review are from psychology, except for two studies (68, 69) from economics. It may well be possible that heterogeneity varies across fields in the social sciences. Third, multianalyst studies have been criticized for overestimating the analytical variation, e.g., due to ambiguity about the studied research question (73, 74). Our estimates of heterogeneity can thus be considered to be subject to heterogeneity themselves, implying that the conclusions drawn about the extent of generalizability of empirical claims are not necessarily generalizable beyond the set of studies included in our systematic review. Fourth, the heterogeneity estimates are derived from a small sample of studies, especially those for design and analytical heterogeneity, and the number of studies differs markedly across the three types of heterogeneity. Fifth, comparisons across types of heterogeneity may be considered unjust given that the range of observed variation might be a function of the limited set of studies used for estimating each type of heterogeneity; population heterogeneity might, for instance, be small just because the sampled populations in the included studies are relatively similar.

Notwithstanding, it is important to note that even the smallest estimates obtained from our review of the literature point toward considerable heterogeneity due to the variability in designs (H = 1.92) and analyses (H = 1.72). It is also worth emphasizing that a typical empirical study will involve heterogeneity due to all three sources investigated above, which implies that the overall level of heterogeneity may be even higher than the estimates reported in Fig. 2.

The estimated levels of design and analytical heterogeneity in our data indicate that the generalizability of individual studies can be low. An important first step to overcome the adverse effects of unwarranted generalizations is to increase awareness of the importance of heterogeneity to pave the way toward more nuanced assessments and interpretations of empirical results (75). Considerations of heterogeneity and generalizability should also find their way into editorial guidelines to inform both authors and reviewers about their importance. Methodological standardization, sharper theoretical predictions, and clearer alignment of theoretical conceptualizations and empirical instrumentalizations can also narrow down the set of plausible research designs and analytical choices, thereby reducing heterogeneity (37–39).

Another possibility would be to directly incorporate the added uncertainty due to heterogeneity into inference. Directly incorporating heterogeneity would imply multiplying the estimated SE by a sensible estimate of H. A conservative interpretation of the estimates based on the studies in our sample suggests that accounting for heterogeneity would approximately double sample SE and CI. However, it seems hasty to recommend a specific multiplier due to the limited number of included studies, the uncertainty surrounding our H estimates, and the limited information about whether and how H varies across methods and fields. Further evidence is needed. A promising method of obtaining further evidence of analytical heterogeneity and its magnitude (H) is the multianalyst approach.

Multianalyst studies offer a methodology to map and gauge analytical heterogeneity (76–78) and were used as a basis for estimating H due to analytical heterogeneity in our empirical review. While the multianalyst approach brings with it the potential to cope with analytical heterogeneity by pooling results based on various justifiable analysis paths, we are not aware of any multianalyst study proposing or conducting joint inference. For the multianalyst approach to become an appropriate methodology that allows going beyond demonstrating the variability of effect sizes and conclusions, it should be refined and steered in the direction of pooled hypothesis tests. The challenge in devising joint inference procedures across the analysis paths implemented by the analysts is that the estimates are not independent, causing difficulties in estimating both the uncertainty of the mean effect size and heterogeneity across analysts. Both simple pooled tests and more sophisticated tests should be explored. A simple pooled hypothesis test would rely on the average t-/z-value of all analyses. While the mean t-/z-statistic will not incorporate analytical heterogeneity, it can be a straightforward and sensible means to pooled inference whenever the number of analysts is sufficiently large (as the impact of heterogeneity becomes negligible due to the fact that the analytical variation gets divided by the square root of the number of analysts). A more sophisticated test would rely on bootstrapping (79) both the analysis paths and the data sample to estimate the SEM effect size to construct a z-test statistic for a pooled hypothesis test. Such a bootstrap test would incorporate both the sampling variance and analytical heterogeneity while taking into account the correlation in effect size estimates across analysts.

As emphasized in the discussion of our estimates of analytical heterogeneity, the question of how to handle “outlier analysis paths” plays a crucial role in estimating meta-effects and heterogeneity in multianalyst settings. It is essential to have methods and procedures in place to ensure quality control of the analysis paths chosen by analysts and to determine whether or not these paths qualify as legitimate to address the hypothesis in question (80). We recommend that multianalyst studies report estimates of the extent of analytical heterogeneity in terms of H (or I²) obtained via a random-effects meta-analysis and in terms of the ratio between the SD of effect size estimates across analysts and the mean SE as introduced by Huntington-Klein et al. (63). The latter measure, which can be converted into a proxy measure of H (see the Materials and Methods section), is used in a robustness test reported below. A viable alternative approach to estimating analytical heterogeneity is multiverse-style analysis.

A practical alternative to the multianalyst approach is multiverse-style analyses (81–83), in which a multitude of justifiable analytical choices are factorially combined to map the analytical space. Multiverse-style analyses can be conceived of as modeling the theoretical variability of estimates, whereas multianalyst studies gauge analytic variability “in the wild.” A multianalyst study can be thought of as a weighted multiverse analysis with weights determined based on the frequency with which analysts implement (combinations of) particular analytical choices. While multiverse-style analyses are on the rise in psychology and related fields (84), we are not aware of any studies reporting indicators of analytical heterogeneity such as H. We recommend routinely reporting heterogeneity estimates in the context of multiverse-style analyses. The same indicators of heterogeneity as proposed for multianalyst studies apply.

Multiverse-style analyses typically plot the distribution of effect sizes or P-values across analysis paths and provide descriptive statistics as the focal outcomes, but there is also a limited set of literature on how to aggregate individual results into pooled hypothesis tests. In introducing the multiverse approach to psychology, Steegen et al. (82) mention the possibility of using the average P-value across analysis paths as a pooled hypothesis test. However, averaging p-values is inexpedient whenever effect size estimates point in different directions, as is often the case in multianalyst and multiverse-style studies. Using the average t-/z-value instead corresponds to the simple test mentioned above. Simonsohn et al. (83) propose approaching the joint inference problem using permutation and bootstrapping techniques and put forward a number of pooled tests based on median effect sizes and the fraction of significant results. While the median estimate is more robust to outliers, the mean effect size could be argued to be a more relevant pooled measure if all implemented analysis paths are equally legitimate. There is scope for more work on appropriate and practical joint inference methods, with multianalyst studies and multiverse-style analyses facing similar challenges.

Multiverse-style analyses offer a viable tool for estimating and incorporating analytical heterogeneity into statistical inference. Yet, similar to the multianalyst approach, the expediency of multiverse-style methods hinges on the identification of the analytical space (84); reliable mechanisms for doing so are needed. Moving toward preregistered multiverse analysis would also be an important step to avoid analytic forks and choice points being chosen selectively.

The standard tool for incorporating heterogeneity into inference is applying random-effects meta-analysis, pooling effect size estimates across different (independent) studies. Typically, standard meta-analyses synthesize estimates based on varying populations, study designs, and analyses such that various sources of heterogeneity are modeled jointly. Pooling many studies in a random-effects meta-analysis has the important advantage that the uncertainty due to heterogeneity is substantially reduced, and with a sufficiently large number of studies, precise estimates of meta-analytic effect sizes can be obtained. However, as currently practiced, standard meta-analyses have important limitations. The estimated results are affected by publication bias and p-hacking (16–19, 85), the results are sensitive to the decision of which studies to include/exclude, and the scope for intentionally or unintentionally p-hacking meta-analytic results is large. With the rise of preregistration, detailed preanalysis plans, and Registered Reports, the validity of standard meta-analysis may increase over time. In particular, meta-analyses that only include results from Registered Reports have the potential (conditional on proper inclusion criteria) to provide unbiased meta-analytic effect sizes, which incorporate the uncertainty due to heterogeneity from various sources. Below, we discuss the opportunities and challenges involved in transitioning toward preregistered prospective meta-analysis to further enhance the validity of meta-analytic estimates.

An alternative to conducting traditional meta-analyses based on the existing literature would be heading toward fewer but much larger empirical studies in which reasonable research designs, analysis paths, and relevant populations are systematically varied as part of an encompassing research design. We think of such studies as prospective meta-analyses to distinguish them from traditional ones. When analyzing such studies using random-effects meta-analytic models, heterogeneity is incorporated into the SE of the meta-analytic effect size. The systematic variation of populations, designs, and analytical choices can be used to identify sources of heterogeneity, yielding important insights to build more complete and nuanced theories (39). Importantly, such studies should be preregistered to limit researcher degrees of freedom and can involve elements of crowd science to reach the necessary scale (86, 87).

The ManyLabs studies (54–57) in psychology can be thought of as examples of pioneering work in this direction, although primarily concerned with studying population heterogeneity. Another example is Huber et al. (61), systematically varying experimental designs across random samples from the same population. Prospective meta-analyses could also be augmented by the multianalyst approach to facilitate the incorporation of analytic heterogeneity. If independent research teams, for instance, decide on both their research design and a preferred analysis pipeline, and the design and analysis proposals of all teams are implemented in independent random subsamples, the meta-analytic mean will account for both analytical and design heterogeneity. Likewise, analytical heterogeneity could be integrated by implementing multiverse-style analyses for the various research designs considered in a prospective meta-analysis by estimating the meta-analytic effect across analysis paths for each design and subsequently aggregating the estimates in a random-effects model. The meta-analytic estimate would also incorporate population heterogeneity if the various design and analysis proposals were implemented in different populations.

We advocate moving toward conducting more research in the form of preregistered prospective meta-analyses. Conceptually, our plea for more large-scale studies systematically integrating heterogeneity is similar in spirit to a recent proposal by List (86)—referred to as “option C,” as a mnemonic for an approach beyond A/B testing—, recommending the use of field experiments to generate evidence for policymaking before implementing policies on a larger scale. List (88) argues for complementing the typical initial small efficacy test under ideal conditions with a test under natural conditions, which directly produces the evidence needed to evaluate the policy after scaling. Integrating option C thinking into research practice implies larger and more costly data collections in exchange for more efficient and comprehensive knowledge generation and avoids scaling up initially promising projects that will not thrive on a grand scale. Shifting evidence generation toward relying more on preregistered prospective meta-analyses would involve similar trade-offs: fewer but more costly studies in exchange for expedited and more comprehensive evidence generation—a trade-off that seems worthwhile. However, steering common research practices in the direction of prospective meta-analysis calls for adapting the “rules” of the academic enterprise: Funding bodies would need to prioritize large-scale data collections, the academic reward system (including, e.g., hiring decisions) would need to adapt to acknowledge crowd science contributions, and the infrastructure for conducting team science would need to be strengthened.

We reestimated the random-effects meta-analytic models for each included study based on the original data. In SI Appendix, Table S1, we provide detailed results for each included meta-analysis, comprising the Q-test, whether effect sizes were measured in Cohen’s d units, the between-study variation (τ and its 95% CI), the within-study variation (σ), the ratio between the between- and within-study variation (τ/σ; which we refer to as the heterogeneity ratio HR), I² and its 95% CI, and H and its 95% CI. If not indicated otherwise in SI Appendix, sections 1–3, we were able to precisely (computationally) reproduce the results reported in the papers. As such, our study provides—as a “side product”—evidence on the computational reproducibility (104–108) of large-scale meta-scientific results.

To keep things simple and easily replicable, we created copies of the relevant input data for the meta-analyses (i.e., the effect size estimate and the corresponding SE for each study included in the meta-analysis) for each paper based on the original data (all of which are publicly available under a CC-by license). These copies of the original data constitute our raw data. All data and analysis scripts used to generate the results reported in the main text and SI Appendix are available at our project’s OSF repository: osf.io/yegsx.

Random effects meta-analyses were estimated using the metafor package (v-4.4.0) (109) in R (v-4.3.2) (110). Estimates for the CI around the heterogeneity measures τ², I², and H² were based on the Q-profile method (111) implemented using the confint() function shipped with the metafor package. For all papers reporting the results of meta-analyses (i.e., all papers on population or design heterogeneity), we used the same estimator for τ² as used in the original paper. Most of these papers relied on the restricted maximum likelihood estimator (112); but one study used the DerSimonian–Laird (113) estimator, and one used the Hartung–Knapp (114) estimator.

For multianalyst studies, heterogeneity estimates were based on the restricted maximum likelihood estimator. Note that estimating a random-effects model on multianalyst-style data is unconventional, as discussed in the main text. Hence it does not come as a surprise that none of the multianalyst studies included in our review reported the results of a random-effects meta-analysis; however, a recent multianalyst study in biology (72) used a meta-analytic random-effects model to estimate the heterogeneity of results across analysts. The estimated heterogeneity measures τ², I², and H² for multianalyst studies can be interpreted as lower bound estimates as they are derived based on the within-study variance that would be observed if the effect size estimates reported by multiple analysts were independent observations; if the sampling variances of the multiple analysts are correlated (which is the case for multianalyst studies, since analysts based their estimates on the same dataset), the actual within-study variance is lower, and the between-study variance is higher.

Huntington-Klein et al. (69) introduced the ratio between the SD of effect size estimates across analysts and the mean SE as a measure of the analytical heterogeneity in multianalyst studies. This measure can be interpreted as a proxy for the ratio of the between-study variation and the within-study variation (HR) and can be converted to a proxy measure of H by taking the square root of 1 plus the squared ratio; to distinguish the two measures from HR and H obtained from the estimates of random-effects meta-analyses, we denote them as HR_P and H_P. In SI Appendix, Table S2, we report both HR_P and H_P for the multianalyst studies included in our review. HR_P varies between 1.48 and 3.98 for the multianalyst studies, with a median of 3.07; H_P varies between 1.79 and 4.11 for the multianalyst studies, with a median of 3.23. Note that while the between-study variation (τ) estimated in the random-effects meta-analysis is a lower bound of the SD in effect sizes across analysts, the proxy H_P may exceed H as estimated using a random-effects meta-analysis as the estimated within-study variation (σ) in the random-effects meta-analysis can differ from the average SE of estimates generated by the analysts. The proxy H_P is quite similar to H based on the random-effects meta-analysis for four of the five multianalyst estimates but differs substantially for Silberzahn et al. (70). This is due to two outliers in terms of low SE strongly affecting the estimated within-study variance in the random-effects meta-analysis (as the individual effects are weighted by the inverse of their variance). This is an indication that the estimate of H for Silberzahn et al. (70) should be interpreted very cautiously; but also the proxy H_P indicates substantial analytical heterogeneity. Removing the two outliers for Silberzahn et al. (70) (implying k = 27 effect size estimates) results in the following heterogeneity estimates in a random-effect meta-analysis: Q(26) = 130.3, P < 0.001; τ = 0.107, I² = 86.5%, H = 2.717, HR = 2.527; the proxy measures based on ref. 69 remain qualitatively unchanged (H_P = 1.721, HR_P = 1.401).

Consider a generic two-tailed z-test with power π to detect an effect θ at a type-I error rate α. The effect size θ (measured in z-score units in the generic test) corresponds to the noncentrality parameter δ = |z_α/2| + |z_β|, where z_p denotes the pth quantile of the inverse cumulative standard normal distribution, and β = 1 − π denotes the false negative rate. Assuming that the true effects to be estimated are homogeneous, θ_i ~ N(μ₀, σ²) under the null hypothesis H₀ (with μ₀ indicating the test value and σ² denoting the test's sampling variance); under the alternative hypothesis H_A, θ_i ~ N(δ, σ²).

Now suppose there is variation in the true effect size above and beyond the uncertainty that is accounted for by the test's sampling variance (σ_i²). Put differently, effect size estimates are subject to an additional source of uncertainty—heterogeneity—such that the overall variance of study i’s estimate θ_i is given by ν_i² = σ² + τ². The heterogeneity estimate τ² indicates the variance of the genuine effect, such that θ_i ~ N(μ₀, ν_i²) under H₀ and θ_i ~ N(δ, ν_i²) under H_A.

Instead of quantifying the extent of heterogeneity in absolute terms (i.e., in terms of τ² or τ, respectively), it is expedient to denote it relative to the test’s sampling variance (σ_i²). Following the notational conventions applicable to random-effects meta-analysis (45), we define a heterogeneity factor H as

which is equivalent to the square root of H², a commonly used measure of heterogeneity reported in meta-analyses. H² can be interpreted as a “variance inflation factor” due to heterogeneity (46, 47), i.e., as the factor a test’s sampling error σ_i needs to be multiplied by to incorporate the uncertainty due to heterogeneity into statistical inference. We prefer H over H² as thinking about heterogeneity in terms of SD units appears more convenient than thinking about it in terms of variance units. H² is defined as the relative excess of the Q-statistic over its degrees of freedom, i.e., H² = Q/(k − 1). Following conventions, we presume H = max(1, H) but acknowledge that this prevents identifying excessive homogeneity—i.e., less variability than would be expected due to chance (46).

We distinguish between nominal and effective error rates below, where the effective error rates are the observed error rates after accounting for heterogeneity. The effective false positive rate α’ in a two-tailed z-test in the presence of heterogeneity (expressed in terms of the heterogeneity factor H) is given by

where Φ(·) indicates the cumulative standard normal distribution, and z_α/2 denotes the α/2-percentile of the inverse cumulative standard normal density function Φ⁻¹(·) (i.e., the critical value of a two-tailed z-test at a nominal significance threshold α). It follows that α’ > α for any H > 1. Correspondingly, the effective false negative rate β′ is given by

The PSP of the hypothesis being true is defined as the ratio of true positive results to the total number of positive classifications, which implies that the PSP is a function of the prior probability ϕ for the alternative hypothesis being genuinely true, i.e.,

Note that PSP is equivalent to the inverse of the false discovery rate, i.e., our results could just as well be expressed in terms of the rate of false positive classifications.

Since heterogeneity inflates the effective type-I error rate (for any nominal α-level in a two-tailed test) and the effective type-II error rate (for any nominal β > 0.5), it follows that the effective poststudy probability, PSP_G, is given by

Since the cumulative normal density Φ(·) is convex in the domain (−∞, 0], it follows that PSP_G < PSP for any H > 1. Fig. 1 illustrates PSP_G as a function of H for various levels of the prior ϕ. PSP_G can be conceived as an indicator of the generalizability of an individual study, measuring the probability of the meta-hypothesis being true, conditional on an individual study reporting a statistically significant result.

F.H., M.J., R.B., A.D., J.H., and M.K. designed research; F.H. and M.J. performed research; F.H. analyzed data; F.H., M.J., R.B., A.D., J.H., and M.K. reviewed and edited the paper; and F.H., M.J., and M.K. wrote the paper.

The authors declare no competing interest.