New Research In
Physical Sciences
Social Sciences
Featured Portals
Articles by Topic
Biological Sciences
Featured Portals
Articles by Topic
- Agricultural Sciences
- Anthropology
- Applied Biological Sciences
- Biochemistry
- Biophysics and Computational Biology
- Cell Biology
- Developmental Biology
- Ecology
- Environmental Sciences
- Evolution
- Genetics
- Immunology and Inflammation
- Medical Sciences
- Microbiology
- Neuroscience
- Pharmacology
- Physiology
- Plant Biology
- Population Biology
- Psychological and Cognitive Sciences
- Sustainability Science
- Systems Biology
Standard analyses fail to show that US studies overestimate effect sizes in softer research

Fanelli and Ioannidis (1) have recently hypothesized that scientific biases are worsened by the relatively high publication pressures in the United States and by the use of “softer” methodologies in much of the behavioral sciences. The authors analyzed nearly 1,200 studies from 82 meta-analyses and found more extreme effect sizes in studies from the United States, and when using soft behavioral (BE) versus less-soft biobehavioral (BB) and nonbehavioral (NB) methods. Their results are based on nonstandard analyses, withas the dependent variable, where
is the effect size (log of the odds ratio) of study i in meta-analysis j, and
is the summary effect size of meta-analysis j. After obtaining the data from Fanelli, we performed more standard metaregression analyses on
to verify their conclusion that effect sizes and publication bias differ between methods and the United States (US) vs. other countries. For our analyses, we used the R package metafor (2).
First, we ran 82 mixed-effects meta-analyses:We multiplied
by −1 if the primary researchers expected a negative effect,
if the primary study was conducted in the United States, and 0 otherwise.
is the study’s SE, where a positive
signifies publication bias [tantamount to Egger’s test (3)]. Next, we ran two mixed-effects metameta-regressions on the 82
, both with and without the method (NB, BB, or BE) as a moderator. The goal was to examine whether the regression weights from the 82 meta-analyses differed between methods, and whether they deviated from zero when averaged over the three methods.
In the metameta-regression, method had no effect on (
. The overall effect of
in the intercept-only model was also not significant (
, meaning that publication bias was not different for the United States and other countries.
Because there was no overall interaction, we reran the 82 meta-analyses without this interaction, and then again analyzed both
and
with metameta-regressions. Fig. 1 shows the distributions of
and
. There was no effect of method on
(
, and no overall effect of the United States (
. Hence, contrary to Fanelli and Ioannidis (1), using standard analyses we found no evidence of higher effect sizes in the United States for any of the three methods. There was also no effect of method on
(
, but the overall positive effect of SE (
signifies publication bias across all methods.
Histograms of the effect of United States and SE on effect size.
To conclude, we failed to find that US studies overestimate effect sizes in softer research. It is rather surprising that Fanelli and Ioannidis did find an effect of US, because the distribution of is almost centered on zero (Fig. 1, Left). We found no effect of United States and no effects of “softness” of methods using standard analyses. However, we found overall publication bias for all methods. Hence, the conclusions of Fanelli and Ioannidis (1) are not robust to the method of analysis.
Acknowledgments
The preparation of this article was supported by Grants 016-125-385 and 406-13-050 from the Netherlands Organization for Scientific Research.
Footnotes
- ↵1To whom correspondence should be addressed. E-mail: m.b.nuijten{at}uvt.nl.
Author contributions: M.A.L.M.v.A. and J.M.W. designed research; M.B.N. and R.C.M.v.A. performed research; M.B.N. and R.C.M.v.A. analyzed data; and M.B.N., M.A.L.M.v.A., and J.M.W. wrote the paper.
The authors declare no conflict of interest.
References
- ↵
- Fanelli D,
- Ioannidis JPA
- ↵
- Viechtbauer W
- ↵
- Egger M,
- Davey Smith G,
- Schneider M,
- Minder C
Citation Manager Formats
Sign up for Article Alerts
You May Also be Interested in
More Articles of This Classification
Related Content
Cited by...
- No citing articles found.