Skip to main content
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian
  • Log in
  • My Cart

Main menu

  • Home
  • Articles
    • Current
    • Latest Articles
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • Archive
  • Front Matter
  • News
    • For the Press
    • Highlights from Latest Articles
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Purpose and Scope
    • Editorial and Journal Policies
    • Submission Procedures
    • For Reviewers
    • Author FAQ
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian

User menu

  • Log in
  • My Cart

Search

  • Advanced search
Home
Home

Advanced Search

  • Home
  • Articles
    • Current
    • Latest Articles
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • Archive
  • Front Matter
  • News
    • For the Press
    • Highlights from Latest Articles
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Purpose and Scope
    • Editorial and Journal Policies
    • Submission Procedures
    • For Reviewers
    • Author FAQ

New Research In

Physical Sciences

Featured Portals

  • Physics
  • Chemistry
  • Sustainability Science

Articles by Topic

  • Applied Mathematics
  • Applied Physical Sciences
  • Astronomy
  • Computer Sciences
  • Earth, Atmospheric, and Planetary Sciences
  • Engineering
  • Environmental Sciences
  • Mathematics
  • Statistics

Social Sciences

Featured Portals

  • Anthropology
  • Sustainability Science

Articles by Topic

  • Economic Sciences
  • Environmental Sciences
  • Political Sciences
  • Psychological and Cognitive Sciences
  • Social Sciences

Biological Sciences

Featured Portals

  • Sustainability Science

Articles by Topic

  • Agricultural Sciences
  • Anthropology
  • Applied Biological Sciences
  • Biochemistry
  • Biophysics and Computational Biology
  • Cell Biology
  • Developmental Biology
  • Ecology
  • Environmental Sciences
  • Evolution
  • Genetics
  • Immunology and Inflammation
  • Medical Sciences
  • Microbiology
  • Neuroscience
  • Pharmacology
  • Physiology
  • Plant Biology
  • Population Biology
  • Psychological and Cognitive Sciences
  • Sustainability Science
  • Systems Biology

Standard analyses fail to show that US studies overestimate effect sizes in softer research

Michèle B. Nuijten, Marcel A. L. M. van Assen, Robbie C. M. van Aert, and Jelte M. Wicherts
PNAS February 18, 2014 111 (7) E712-E713; https://doi.org/10.1073/pnas.1322149111
Michèle B. Nuijten
Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, 5000 LE Tilburg, The Netherlands
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: m.b.nuijten@uvt.nl
Marcel A. L. M. van Assen
Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, 5000 LE Tilburg, The Netherlands
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Robbie C. M. van Aert
Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, 5000 LE Tilburg, The Netherlands
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jelte M. Wicherts
Department of Methodology and Statistics, Tilburg School of Social and Behavioral Sciences, Tilburg University, 5000 LE Tilburg, The Netherlands
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site

This article has letters. Please see:

  • US studies may overestimate effect sizes in softer research
  • Reply to Nuijten et al.: Reanalyses actually confirm that US studies overestimate effects in softer research
  • Article
  • Figures & SI
  • Info & Metrics
  • PDF
Loading

Fanelli and Ioannidis (1) have recently hypothesized that scientific biases are worsened by the relatively high publication pressures in the United States and by the use of “softer” methodologies in much of the behavioral sciences. The authors analyzed nearly 1,200 studies from 82 meta-analyses and found more extreme effect sizes in studies from the United States, and when using soft behavioral (BE) versus less-soft biobehavioral (BB) and nonbehavioral (NB) methods. Their results are based on nonstandard analyses, withEmbedded Imageas the dependent variable, where Graphic is the effect size (log of the odds ratio) of study i in meta-analysis j, andGraphic is the summary effect size of meta-analysis j. After obtaining the data from Fanelli, we performed more standard metaregression analyses on Graphic to verify their conclusion that effect sizes and publication bias differ between methods and the United States (US) vs. other countries. For our analyses, we used the R package metafor (2).

First, we ran 82 mixed-effects meta-analyses:Embedded ImageWe multiplied Graphic by −1 if the primary researchers expected a negative effect, Graphic if the primary study was conducted in the United States, and 0 otherwise. Graphic is the study’s SE, where a positive Graphic signifies publication bias [tantamount to Egger’s test (3)]. Next, we ran two mixed-effects metameta-regressions on the 82 Graphic, both with and without the method (NB, BB, or BE) as a moderator. The goal was to examine whether the regression weights from the 82 meta-analyses differed between methods, and whether they deviated from zero when averaged over the three methods.

In the metameta-regression, method had no effect on Graphic (Graphic. The overall effect of Graphic in the intercept-only model was also not significant (Graphic, meaning that publication bias was not different for the United States and other countries.

Because there was no overall Graphic interaction, we reran the 82 meta-analyses without this interaction, and then again analyzed both Graphic and Graphic with metameta-regressions. Fig. 1 shows the distributions of Graphic and Graphic. There was no effect of method on Graphic (Graphic, and no overall effect of the United States (Graphic. Hence, contrary to Fanelli and Ioannidis (1), using standard analyses we found no evidence of higher effect sizes in the United States for any of the three methods. There was also no effect of method on Graphic (Graphic, but the overall positive effect of SE (Graphic signifies publication bias across all methods.

Fig. 1.
  • Download figure
  • Open in new tab
  • Download powerpoint
Fig. 1.

Histograms of the effect of United States and SE on effect size.

To conclude, we failed to find that US studies overestimate effect sizes in softer research. It is rather surprising that Fanelli and Ioannidis did find an effect of US, because the distribution of Graphic is almost centered on zero (Fig. 1, Left). We found no effect of United States and no effects of “softness” of methods using standard analyses. However, we found overall publication bias for all methods. Hence, the conclusions of Fanelli and Ioannidis (1) are not robust to the method of analysis.

Acknowledgments

The preparation of this article was supported by Grants 016-125-385 and 406-13-050 from the Netherlands Organization for Scientific Research.

Footnotes

  • ↵1To whom correspondence should be addressed. E-mail: m.b.nuijten{at}uvt.nl.
  • Author contributions: M.A.L.M.v.A. and J.M.W. designed research; M.B.N. and R.C.M.v.A. performed research; M.B.N. and R.C.M.v.A. analyzed data; and M.B.N., M.A.L.M.v.A., and J.M.W. wrote the paper.

  • The authors declare no conflict of interest.

References

  1. ↵
    1. Fanelli D,
    2. Ioannidis JPA
    (2013) US studies may overestimate effect sizes in softer research. Proc Natl Acad Sci USA 110(37):15031–15036.
    OpenUrlAbstract/FREE Full Text
  2. ↵
    1. Viechtbauer W
    (2010) The Metafor Package: A Meta-Analysis Package for R (Version 1.3-0).
  3. ↵
    1. Egger M,
    2. Davey Smith G,
    3. Schneider M,
    4. Minder C
    (1997) Bias in meta-analysis detected by a simple, graphical test. BMJ 315(7109):629–634.
    OpenUrlAbstract/FREE Full Text
View Abstract
PreviousNext
Back to top
Article Alerts
Email Article

Thank you for your interest in spreading the word on PNAS.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Standard analyses fail to show that US studies overestimate effect sizes in softer research
(Your Name) has sent you a message from PNAS
(Your Name) thought you would like to see the PNAS web site.
Citation Tools
Reanalysis of Fanelli & Ioannidis (2013)
Michèle B. Nuijten, Marcel A. L. M. van Assen, Robbie C. M. van Aert, Jelte M. Wicherts
Proceedings of the National Academy of Sciences Feb 2014, 111 (7) E712-E713; DOI: 10.1073/pnas.1322149111

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Request Permissions
Share
Reanalysis of Fanelli & Ioannidis (2013)
Michèle B. Nuijten, Marcel A. L. M. van Assen, Robbie C. M. van Aert, Jelte M. Wicherts
Proceedings of the National Academy of Sciences Feb 2014, 111 (7) E712-E713; DOI: 10.1073/pnas.1322149111
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Mendeley logo Mendeley
Proceedings of the National Academy of Sciences: 116 (8)
Current Issue

Submit

Sign up for Article Alerts

Jump to section

  • Article
    • Acknowledgments
    • Footnotes
    • References
  • Figures & SI
  • Info & Metrics
  • PDF

You May Also be Interested in

News Feature: Cities serve as testbeds for evolutionary change
Urban living can pressure flora and fauna to adapt in intriguing ways. Biologists are starting to take advantage of this convenient laboratory of evolution.
Image credit: Kristin Winchell (Washington University in St. Louis, St. Louis).
Several aspects of the proposal, which aims to expand open access, require serious discussion and, in some cases, a rethink.
Opinion: “Plan S” falls short for society publishers—and for the researchers they serve
Several aspects of the proposal, which aims to expand open access, require serious discussion and, in some cases, a rethink.
Image credit: Dave Cutler (artist).
Featured Profile
PNAS Profile of NAS member and biochemist Hao Wu
 Nonmonogamous strawberry poison frog (Oophaga pumilio).  Image courtesy of Yusan Yang (University of Pittsburgh, Pittsburgh).
Putative signature of monogamy
A study suggests a putative gene-expression hallmark common to monogamous male vertebrates of some species, namely cichlid fishes, dendrobatid frogs, passeroid songbirds, common voles, and deer mice, and identifies 24 candidate genes potentially associated with monogamy.
Image courtesy of Yusan Yang (University of Pittsburgh, Pittsburgh).
Active lifestyles. Image courtesy of Pixabay/MabelAmber.
Meaningful life tied to healthy aging
Physical and social well-being in old age are linked to self-assessments of life worth, and a spectrum of behavioral, economic, health, and social variables may influence whether aging individuals believe they are leading meaningful lives.
Image courtesy of Pixabay/MabelAmber.

More Articles of This Classification

  • Improving integration in societal consequences to climate change
  • Reply to Jaffe et al.: Paleoscience precision in an archeological or historical context
  • Need for early, minimally invasive cancer diagnosis
Show more

Related Content

  • US studies may overestimate effects
  • Reanalyses confirm the US effect
  • Scopus
  • PubMed
  • Google Scholar

Cited by...

  • No citing articles found.
  • Scopus (4)
  • Google Scholar

Similar Articles

Site Logo
Powered by HighWire
  • Submit Manuscript
  • Twitter
  • Facebook
  • RSS Feeds
  • Email Alerts

Articles

  • Current Issue
  • Latest Articles
  • Archive

PNAS Portals

  • Classics
  • Front Matter
  • Teaching Resources
  • Anthropology
  • Chemistry
  • Physics
  • Sustainability Science

Information

  • Authors
  • Editorial Board
  • Reviewers
  • Press
  • Site Map

Feedback    Privacy/Legal

Copyright © 2019 National Academy of Sciences. Online ISSN 1091-6490