Skip to main content
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian
  • Log in
  • My Cart

Main menu

  • Home
  • Articles
    • Current
    • Latest Articles
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • Archive
  • Front Matter
  • News
    • For the Press
    • Highlights from Latest Articles
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Purpose and Scope
    • Editorial and Journal Policies
    • Submission Procedures
    • For Reviewers
    • Author FAQ
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian

User menu

  • Log in
  • My Cart

Search

  • Advanced search
Home
Home

Advanced Search

  • Home
  • Articles
    • Current
    • Latest Articles
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • Archive
  • Front Matter
  • News
    • For the Press
    • Highlights from Latest Articles
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Purpose and Scope
    • Editorial and Journal Policies
    • Submission Procedures
    • For Reviewers
    • Author FAQ

New Research In

Physical Sciences

Featured Portals

  • Physics
  • Chemistry
  • Sustainability Science

Articles by Topic

  • Applied Mathematics
  • Applied Physical Sciences
  • Astronomy
  • Computer Sciences
  • Earth, Atmospheric, and Planetary Sciences
  • Engineering
  • Environmental Sciences
  • Mathematics
  • Statistics

Social Sciences

Featured Portals

  • Anthropology
  • Sustainability Science

Articles by Topic

  • Economic Sciences
  • Environmental Sciences
  • Political Sciences
  • Psychological and Cognitive Sciences
  • Social Sciences

Biological Sciences

Featured Portals

  • Sustainability Science

Articles by Topic

  • Agricultural Sciences
  • Anthropology
  • Applied Biological Sciences
  • Biochemistry
  • Biophysics and Computational Biology
  • Cell Biology
  • Developmental Biology
  • Ecology
  • Environmental Sciences
  • Evolution
  • Genetics
  • Immunology and Inflammation
  • Medical Sciences
  • Microbiology
  • Neuroscience
  • Pharmacology
  • Physiology
  • Plant Biology
  • Population Biology
  • Psychological and Cognitive Sciences
  • Sustainability Science
  • Systems Biology

Reply to Inbar: Contextual sensitivity helps explain the reproducibility gap between social and cognitive psychology

Jay J. Van Bavel, Peter Mende-Siedlecki, William J. Brady, and Diego A. Reinero
PNAS August 23, 2016 113 (34) E4935-E4936; published ahead of print August 10, 2016 https://doi.org/10.1073/pnas.1609700113
Jay J. Van Bavel
aDepartment of Psychology, New York University, New York, NY 10003
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: jay.vanbavel@nyu.edu
Peter Mende-Siedlecki
aDepartment of Psychology, New York University, New York, NY 10003
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
William J. Brady
aDepartment of Psychology, New York University, New York, NY 10003
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Diego A. Reinero
aDepartment of Psychology, New York University, New York, NY 10003
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site

See related content:

  • Contextual sensitivity in scientific reproducibility
    - Jun 07, 2016

This article has a reply. Please see:

  • Association between contextual dependence and replicability in psychology may be spurious
  • Article
  • Figures & SI
  • Info & Metrics
  • PDF
Loading

We find that contextual sensitivity correlates with the reproducibility of 100 psychology studies from the Reproducibility Project (1). This relationship remains after adjusting for several methodological factors believed to account for reproducibility (e.g., sample size, effect size) and does not differ across subdisciplines (social vs. cognitive psychology). We also report a strong relationship between subdiscipline and context (1): social psychology studies are more likely to examine contextually sensitive topics than cognitive psychology studies. We conclude that context might play a role in reproducibility across (rather than within) subdisciplines.

Inbar (2) notes that the reproducibility rate of social psychology (28%) is lower than cognitive psychology (53%). He also reports that the association between context and reproducibility becomes nonsignificant when adjusting for subdiscipline. On this basis, Inbar argues that the relationship between context and reproducibility is spurious. This claim is conceptually misguided and overlooks a more important issue: contextual sensitivity helps explain the reproducibility gap between social and cognitive psychology.

The authors of the Reproducibility Project: Psychology (RP:P) argued that the lower reproducibility rate of social psychology is a result of weaker statistical power and effect sizes (3). But subdiscipline predicts reproducibility (P = 0.044) even after adjusting for methodological factors (e.g., sample size, effect size; see model 2 in ref. 1). However, subdiscipline no longer predicts reproducibility once context is added to the model (P = 0.453; see https://osf.io/cgur9/ for details) (Table 1). Although he does not report it, Inbar’s (2) own model reveals that the association between subdiscipline and reproducibility becomes nonsignificant when adjusting for context. In sum, context appears to explain the reproducibility gap better than these methodological factors.

View this table:
  • View inline
  • View popup
Table 1.

Regression coefficients predicting reproducibility and variance explained in two steps of a hierarchical regression

The fact that context and subdiscipline become nonsignificant when entered simultaneously in a regression demonstrates multicollinearity and makes it inappropriate to categorize one as a confound (4). Before claiming a spurious relationship, one needs to rule out a mediating relationship. It seems more likely that subdiscipline affects reproducibility through contextual sensitivity, rather than the other way around. Contextual sensitivity cannot cause a change in subdiscipline. Thus, Inbar’s (2) implied conceptual model appears to be backward.

There are other problems with prioritizing subdiscipline. First, no coding scheme for subdiscipline was reported in the RP:P (3). It is therefore impossible to know if the RP:P coders used context as a heuristic for categorizing studies as social vs. cognitive. Second, the binary coding scheme imposes a false dichotomy between social and cognitive psychology that cannot account for fields like social cognition (5). These coding limitations might explain why several studies appear to have been miscoded in the RP:P [as we noted in our paper (1)].

Our analysis suggests that different rates of reproducibility between social and cognitive psychology partially stem from differences in contextual sensitivity. This will come as little surprise to social psychologists: The notion that human psychology is shaped by the social context has been the central premise of the field for nearly a century (6). We expect that same principle applies across the social sciences.

Footnotes

  • ↵1To whom correspondence should be addressed. Email: jay.vanbavel{at}nyu.edu.
  • Author contributions: J.J.V.B., P.M.-S., W.J.B., and D.A.R. designed research, performed research, analyzed data, and wrote the paper.

  • The authors declare no conflict of interest.

References

  1. ↵
    1. Van Bavel JJ,
    2. Mende-Siedlecki P,
    3. Brady WJ,
    4. Reinero DA
    (2016) Contextual sensitivity in scientific reproducibility. Proc Natl Acad Sci USA 113(23):6454–6459.
    .
    OpenUrlAbstract/FREE Full Text
  2. ↵
    1. Inbar Y
    (2016) Association between contextual dependence and replicability in psychology may be spurious. Proc Natl Acad Sci USA 113:E4933–E4934.
    .
    OpenUrlFREE Full Text
  3. ↵
    1. Open Science Collaboration
    (2015) PSYCHOLOGY. Estimating the reproducibility of psychological science. Science 349(6251):aac4716.
    .
    OpenUrlAbstract/FREE Full Text
  4. ↵
    1. Cohen J,
    2. Cohen P,
    3. West SG,
    4. Aiken LS
    (2013) Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences (Routledge, New York).
    .
  5. ↵
    1. Fiske ST,
    2. Taylor SE
    (2013) Social Cognition: From Brains to Culture (Sage, London) 2nd Ed.
    .
  6. ↵
    1. Lewin K
    (1936) Principles of Topological Psychology (McGraw-Hill, New York) trans Heider F and Heider G.
    .
View Abstract
PreviousNext
Back to top
Article Alerts
Email Article

Thank you for your interest in spreading the word on PNAS.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Reply to Inbar: Contextual sensitivity helps explain the reproducibility gap between social and cognitive psychology
(Your Name) has sent you a message from PNAS
(Your Name) thought you would like to see the PNAS web site.
Citation Tools
Context and the reproducibility gap in psychology
Jay J. Van Bavel, Peter Mende-Siedlecki, William J. Brady, Diego A. Reinero
Proceedings of the National Academy of Sciences Aug 2016, 113 (34) E4935-E4936; DOI: 10.1073/pnas.1609700113

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Request Permissions
Share
Context and the reproducibility gap in psychology
Jay J. Van Bavel, Peter Mende-Siedlecki, William J. Brady, Diego A. Reinero
Proceedings of the National Academy of Sciences Aug 2016, 113 (34) E4935-E4936; DOI: 10.1073/pnas.1609700113
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Mendeley logo Mendeley
Proceedings of the National Academy of Sciences: 116 (7)
Current Issue

Submit

Sign up for Article Alerts

Jump to section

  • Article
    • Footnotes
    • References
  • Figures & SI
  • Info & Metrics
  • PDF

You May Also be Interested in

Several aspects of the proposal, which aims to expand open access, require serious discussion and, in some cases, a rethink.
Opinion: “Plan S” falls short for society publishers—and for the researchers they serve
Several aspects of the proposal, which aims to expand open access, require serious discussion and, in some cases, a rethink.
Image credit: Dave Cutler (artist).
Several large or long-lived animals seem strangely resistant to developing cancer. Elucidating the reasons why could lead to promising cancer-fighting strategies in humans.
Core Concept: Solving Peto’s Paradox to better understand cancer
Several large or long-lived animals seem strangely resistant to developing cancer. Elucidating the reasons why could lead to promising cancer-fighting strategies in humans.
Image credit: Shutterstock.com/ronnybas frimages.
Featured Profile
PNAS Profile of NAS member and biochemist Hao Wu
 Nonmonogamous strawberry poison frog (Oophaga pumilio).  Image courtesy of Yusan Yang (University of Pittsburgh, Pittsburgh).
Putative signature of monogamy
A study suggests a putative gene-expression hallmark common to monogamous male vertebrates of some species, namely cichlid fishes, dendrobatid frogs, passeroid songbirds, common voles, and deer mice, and identifies 24 candidate genes potentially associated with monogamy.
Image courtesy of Yusan Yang (University of Pittsburgh, Pittsburgh).
Active lifestyles. Image courtesy of Pixabay/MabelAmber.
Meaningful life tied to healthy aging
Physical and social well-being in old age are linked to self-assessments of life worth, and a spectrum of behavioral, economic, health, and social variables may influence whether aging individuals believe they are leading meaningful lives.
Image courtesy of Pixabay/MabelAmber.

More Articles of This Classification

  • SNPs deciding the rapid growth of cyanobacteria are alterable
  • Reply to Zhou and Li: Plasticity of the genomic haplotype of Synechococcus elongatus leads to rapid strain adaptation under laboratory conditions
  • Genetic variant rs17185536 regulates SIM1 gene expression in human brain hypothalamus
Show more

Related Content

  • Contextual dependence and replicability
  • Context sensitivity in scientific reproducibility
  • Scopus
  • PubMed
  • Google Scholar

Cited by...

  • No citing articles found.
  • Scopus (11)
  • Google Scholar

Similar Articles

Site Logo
Powered by HighWire
  • Submit Manuscript
  • Twitter
  • Facebook
  • RSS Feeds
  • Email Alerts

Articles

  • Current Issue
  • Latest Articles
  • Archive

PNAS Portals

  • Classics
  • Front Matter
  • Teaching Resources
  • Anthropology
  • Chemistry
  • Physics
  • Sustainability Science

Information

  • Authors
  • Editorial Board
  • Reviewers
  • Press
  • Site Map

Feedback    Privacy/Legal

Copyright © 2019 National Academy of Sciences. Online ISSN 1091-6490