New Research In
Physical Sciences
Social Sciences
Featured Portals
Articles by Topic
Biological Sciences
Featured Portals
Articles by Topic
- Agricultural Sciences
- Anthropology
- Applied Biological Sciences
- Biochemistry
- Biophysics and Computational Biology
- Cell Biology
- Developmental Biology
- Ecology
- Environmental Sciences
- Evolution
- Genetics
- Immunology and Inflammation
- Medical Sciences
- Microbiology
- Neuroscience
- Pharmacology
- Physiology
- Plant Biology
- Population Biology
- Psychological and Cognitive Sciences
- Sustainability Science
- Systems Biology
Reply to Inbar: Contextual sensitivity helps explain the reproducibility gap between social and cognitive psychology
See related content:
- Contextual sensitivity in scientific reproducibility- Jun 07, 2016
This article has a reply. Please see:

We find that contextual sensitivity correlates with the reproducibility of 100 psychology studies from the Reproducibility Project (1). This relationship remains after adjusting for several methodological factors believed to account for reproducibility (e.g., sample size, effect size) and does not differ across subdisciplines (social vs. cognitive psychology). We also report a strong relationship between subdiscipline and context (1): social psychology studies are more likely to examine contextually sensitive topics than cognitive psychology studies. We conclude that context might play a role in reproducibility across (rather than within) subdisciplines.
Inbar (2) notes that the reproducibility rate of social psychology (28%) is lower than cognitive psychology (53%). He also reports that the association between context and reproducibility becomes nonsignificant when adjusting for subdiscipline. On this basis, Inbar argues that the relationship between context and reproducibility is spurious. This claim is conceptually misguided and overlooks a more important issue: contextual sensitivity helps explain the reproducibility gap between social and cognitive psychology.
The authors of the Reproducibility Project: Psychology (RP:P) argued that the lower reproducibility rate of social psychology is a result of weaker statistical power and effect sizes (3). But subdiscipline predicts reproducibility (P = 0.044) even after adjusting for methodological factors (e.g., sample size, effect size; see model 2 in ref. 1). However, subdiscipline no longer predicts reproducibility once context is added to the model (P = 0.453; see https://osf.io/cgur9/ for details) (Table 1). Although he does not report it, Inbar’s (2) own model reveals that the association between subdiscipline and reproducibility becomes nonsignificant when adjusting for context. In sum, context appears to explain the reproducibility gap better than these methodological factors.
Regression coefficients predicting reproducibility and variance explained in two steps of a hierarchical regression
The fact that context and subdiscipline become nonsignificant when entered simultaneously in a regression demonstrates multicollinearity and makes it inappropriate to categorize one as a confound (4). Before claiming a spurious relationship, one needs to rule out a mediating relationship. It seems more likely that subdiscipline affects reproducibility through contextual sensitivity, rather than the other way around. Contextual sensitivity cannot cause a change in subdiscipline. Thus, Inbar’s (2) implied conceptual model appears to be backward.
There are other problems with prioritizing subdiscipline. First, no coding scheme for subdiscipline was reported in the RP:P (3). It is therefore impossible to know if the RP:P coders used context as a heuristic for categorizing studies as social vs. cognitive. Second, the binary coding scheme imposes a false dichotomy between social and cognitive psychology that cannot account for fields like social cognition (5). These coding limitations might explain why several studies appear to have been miscoded in the RP:P [as we noted in our paper (1)].
Our analysis suggests that different rates of reproducibility between social and cognitive psychology partially stem from differences in contextual sensitivity. This will come as little surprise to social psychologists: The notion that human psychology is shaped by the social context has been the central premise of the field for nearly a century (6). We expect that same principle applies across the social sciences.
Footnotes
- ↵1To whom correspondence should be addressed. Email: jay.vanbavel{at}nyu.edu.
Author contributions: J.J.V.B., P.M.-S., W.J.B., and D.A.R. designed research, performed research, analyzed data, and wrote the paper.
The authors declare no conflict of interest.
References
- ↵.
- Van Bavel JJ,
- Mende-Siedlecki P,
- Brady WJ,
- Reinero DA
- ↵.
- Inbar Y
- ↵.
- Open Science Collaboration
- ↵.
- Cohen J,
- Cohen P,
- West SG,
- Aiken LS
- ↵.
- Fiske ST,
- Taylor SE
- ↵.
- Lewin K
Citation Manager Formats
Sign up for Article Alerts
Jump to section
You May Also be Interested in
More Articles of This Classification
Related Content
Cited by...
- No citing articles found.