Standard analyses fail to show that US studies overestimate effect sizes in softer research

Fanelli and Ioannidis (1) have recently hypothesized that scientific biases are worsened by the relatively high publication pressures in the United States and by the use of “softer” methodologies in much of the behavioral sciences. The authors analyzed nearly 1,200 studies from 82 meta-analyses and found more extreme effect sizes in studies from the United States, and when using soft behavioral (BE) versus less-soft biobehavioral (BB) and nonbehavioral (NB) methods. Their results are based on nonstandard analyses, withas the dependent variable, where is the effect size (log of the odds ratio) of study i in meta-analysis j, and is the summary effect size of meta-analysis j. After obtaining the data from Fanelli, we performed more standard metaregression analyses on to verify their conclusion that effect sizes and publication bias differ between methods and the United States (US) vs. other countries. For our analyses, we used the R package metafor (2).

First, we ran 82 mixed-effects meta-analyses:We multiplied by −1 if the primary researchers expected a negative effect, if the primary study was conducted in the United States, and 0 otherwise. is the study’s SE, where a positive signifies publication bias [tantamount to Egger’s test (3)]. Next, we ran two mixed-effects metameta-regressions on the 82 , both with and without the method (NB, BB, or BE) as a moderator. The goal was to examine whether the regression weights from the 82 meta-analyses differed between methods, and whether they deviated from zero when averaged over the three methods.

In the metameta-regression, method had no effect on (. The overall effect of in the intercept-only model was also not significant (, meaning that publication bias was not different for the United States and other countries.

Because there was no overall interaction, we reran the 82 meta-analyses without this interaction, and then again analyzed both and with metameta-regressions. Fig. 1 shows the distributions of and . There was no effect of method on (, and no overall effect of the United States (. Hence, contrary to Fanelli and Ioannidis (1), using standard analyses we found no evidence of higher effect sizes in the United States for any of the three methods. There was also no effect of method on (, but the overall positive effect of SE ( signifies publication bias across all methods.

• Download figure
• Open in new tab
• Download powerpoint

Fig. 1.

Histograms of the effect of United States and SE on effect size.

To conclude, we failed to find that US studies overestimate effect sizes in softer research. It is rather surprising that Fanelli and Ioannidis did find an effect of US, because the distribution of is almost centered on zero (Fig. 1, Left). We found no effect of United States and no effects of “softness” of methods using standard analyses. However, we found overall publication bias for all methods. Hence, the conclusions of Fanelli and Ioannidis (1) are not robust to the method of analysis.