Fanelli and Ioannidis (1) have recently hypothesized that scientific biases are worsened by the relatively high publication pressures in the United States and by the use of “softer” methodologies in much of the behavioral sciences. The authors analyzed nearly 1,200 studies from 82 meta-analyses and found more extreme effect sizes in studies from the United States, and when using soft behavioral (BE) versus less-soft biobehavioral (BB) and nonbehavioral (NB) methods. Their results are based on nonstandard analyses, with
as the dependent variable, where 
is the effect size (log of the odds ratio) of study i in meta-analysis j, and
is the summary effect size of meta-analysis j. After obtaining the data from Fanelli, we performed more standard metaregression analyses on 
to verify their conclusion that effect sizes and publication bias differ between methods and the United States (US) vs. other countries. For our analyses, we used the R package metafor (2).
First, we ran 82 mixed-effects meta-analyses:
We multiplied 
by −1 if the primary researchers expected a negative effect, 
if the primary study was conducted in the United States, and 0 otherwise. 
is the study’s SE, where a positive 
signifies publication bias [tantamount to Egger’s test (3)]. Next, we ran two mixed-effects metameta-regressions on the 82 
, both with and without the method (NB, BB, or BE) as a moderator. The goal was to examine whether the regression weights from the 82 meta-analyses differed between methods, and whether they deviated from zero when averaged over the three methods.
In the metameta-regression, method had no effect on 
(
. The overall effect of 
in the intercept-only model was also not significant (
, meaning that publication bias was not different for the United States and other countries.
Because there was no overall 
interaction, we reran the 82 meta-analyses without this interaction, and then again analyzed both 
and 
with metameta-regressions. Fig. 1 shows the distributions of 
and 
. There was no effect of method on 
(
, and no overall effect of the United States (
. Hence, contrary to Fanelli and Ioannidis (1), using standard analyses we found no evidence of higher effect sizes in the United States for any of the three methods. There was also no effect of method on 
(
, but the overall positive effect of SE (
signifies publication bias across all methods.
Histograms of the effect of United States and SE on effect size.
To conclude, we failed to find that US studies overestimate effect sizes in softer research. It is rather surprising that Fanelli and Ioannidis did find an effect of US, because the distribution of 
is almost centered on zero (Fig. 1, Left). We found no effect of United States and no effects of “softness” of methods using standard analyses. However, we found overall publication bias for all methods. Hence, the conclusions of Fanelli and Ioannidis (1) are not robust to the method of analysis.
The preparation of this article was supported by Grants 016-125-385 and 406-13-050 from the Netherlands Organization for Scientific Research.
Author contributions: M.A.L.M.v.A. and J.M.W. designed research; M.B.N. and R.C.M.v.A. performed research; M.B.N. and R.C.M.v.A. analyzed data; and M.B.N., M.A.L.M.v.A., and J.M.W. wrote the paper.
The authors declare no conflict of interest.
