Thaler and Sunstein’s “nudge” (1) has spawned a revolution in behavioral science research. Despite its popularity, the “nudge approach” has been criticized for having a “limited evidence base” (e.g., ref. 2). Mertens et al. (3) seek to address that limitation with a timely and comprehensive metaanalysis. Mertens et al.’s headline finding is that “choice architecture [nudging] is an effective and widely applicable behavior change tool” (p. 8). We propose their finding of “moderate publication bias” (p. 1) is the real headline; when this publication bias is appropriately corrected for, no evidence for the effectiveness of nudges remains (Fig. 1).
Fig. 1.
RoBMAPSMA model-averaged posterior mean effect size estimates with 95% credible intervals and Bayes factors for the absence of the effect for the combined sample or split by either the domain or intervention category (ignoring the clustering of SEs). BF01 quantifies evidence for the null hypothesis. BF01 larger than one corresponds to evidence in favor of the null hypothesis, and BF01 lower than one corresponds to evidence in favor of the alternative hypothesis (evidence for the alternative hypothesis can be obtained by reciprocating the Bayes factor; BF10 = 1/BF01). As a rule of thumb, Bayes factors between 3 and 10 indicate moderate evidence, and Bayes factors larger than 10 indicate strong evidence.
Mertens et al. (3) find significant publication bias, through Egger regression. Their sensitivity analysis (4) indicates that the true effect size could be as low as d = 0.08 (if publication bias is severe). Mertens et al. argue that severe publication bias is only partially supported by the funnel plot and proceed largely without taking publication bias into account in their subsequent analyses. However, the reported Egger coefficient (b = 2.10) is “severe” (5).
A newly proposed bias correction technique, robust Bayesian metaanalysis (RoBMA) (6), avoids an all-or-none debate over whether or not publication bias is “severe.” RoBMA simultaneously applies 1) selection models that estimate relative publication probabilities (7) and 2) models of the relationship between effect sizes and SEs [i.e., Precision Effect Test and Precision Effect Estimate with Standard Error (6, 8, 9)]. Multimodel inference is then guided mostly by those models that predict the observed data best (6, 9, 10). RoBMA makes multimodel inferences about the presence or absence of an effect, heterogeneity, and publication bias (6, 9).
Table 1 compares the unadjusted results to the publication bias–adjusted results.* Since publication bias–corrected three-level selection models are computationally intractable, we analyzed the data in two ways: 1) ignoring the three-level structure (column 2) and 2) using only the most precise estimate from studies with multiple results (column 3). Strikingly, there is an absence of evidence for an overall effect and evidence against an effect in the “information” and “assistance” intervention categories, whereas the evidence is undecided for “structure” interventions. When using only the most precise estimates, we further find evidence against an effect in most of the domains, apart from “other,” “food,” and “prosocial” (the evidence is indecisive) and weak evidence for the overall effect. However, all intervention categories and domains apart from “finance” show evidence for heterogeneity, which implies that some nudges might be effective, even when there is evidence against the mean effect. Finally, we find strong evidence for publication bias across all subdomains (BFpb > 10), apart from food, when using only the most precise estimates (BFpb = 2.49).
Table 1.
Comparison of unadjusted and adjusted effect size estimates for all studies and for subsets of studies based on different categories or domains
 Random effectsRoBMAPSMARoBMAPSMA (precise)
Combined0.43 [0.38, 0.48]0.04 [0.00, 0.14]0.11 [0.00, 0.24]
t(333) = 16.51BF01 = 0.95BF01 = 0.31
Intervention category
 Information0.25 [0.19, 0.30]0.00 [0.00, 0.00]0.00 [0.00, 0.07]
t(88) = 8.79BF01 = 33.84BF01 = 10.57
 Structure0.58 [0.50, 0.66]0.12 [0.00, 0.43]0.23 [0.00, 0.49]
t(186) = 13.93BF01 = 1.12BF01 = 0.33
 Assistance0.22 [0.15, 0.29]0.01 [0.00, 0.07]0.01 [0.00, 0.12]
t(65) = 6.42BF01 = 9.05BF01 = 8.00
Domain   
 Health0.31 [0.22, 0.39]0.01 [0.00, 0.10]0.02 [0.00, 0.19]
t(64) = 7.03BF01 = 8.98BF01 = 3.53
 Food0.66 [0.52, 0.81]0.02 [−0.09, 0.32]0.27 [0.00, 0.64]
t(81) = 9.01BF01 = 5.16BF01 = 0.55
 Environment0.48 [0.37, 0.58]0.01 [−0.18, 0.25]0.00 [−0.44, 0.34]
t(56) = 9.16BF01 = 4.41BF01 = 3.05
 Finance0.23 [0.15, 0.31]0.00 [0.00, 0.00]0.00 [0.00, 0.00]
t(34) = 6.08BF01 = 41.23BF01 = 30.95
 Prosocial0.32 [0.22, 0.42]0.00 [0.00, 0.05]0.05 [0.00, 0.27]
t(38) = 6.36BF01 = 11.93BF01 = 1.89
 Other0.40 [0.29, 0.50]0.08 [0.00, 0.33]0.04 [−0.22, 0.40]
t(55) = 7.66BF01 = 1.38BF01 = 2.45
First column: Random effects metaanalysis estimates with 95% CI based on clustered SEs, all P values < 0.001. Second and third columns: RoBMAPSMA model-averaged posterior mean effect size estimates with 95% credible intervals and Bayes factor for the presence of the effect ignoring the clustering of SEs or using the most precise estimates (precise). Results differ slightly from the moderator analysis presented in the article because we analyzed each subfield separately to allow 1) testing for the presence of the effect in each category/domain in the Bayesian framework, and 2) publication bias to operate differently in different subdomains.
We conclude that the “nudge” literature analyzed in ref. 3 is characterized by severe publication bias. Contrary to Mertens et al. (3), our Bayesian analysis indicates that, after correcting for this bias, no evidence remains that nudges are effective as tools for behaviour change.

Data Availability

Data and analysis script are available in ref. 11.

Acknowledgments

We thank Mertens et al. for sharing well-documented data and code.

References

1
R. H. Thaler, C. R. Sunstein, Nudge: Improving Decisions about Health, Wealth, and Happiness (Yale University Press, 2008).
2
Y. Lin, M. Osman, R. Ashcroft, Nudge: Concept, effectiveness, and ethics. Basic Appl. Soc. Psych. 39, 293–306 (2017).
3
S. Mertens, M. Herberz, U. J. J. Hahnel, T. Brosch, The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains. Proc. Natl. Acad. Sci. U.S.A. 119, e2107346118 (2022).
4
J. L. Vevea, C. M. Woods, Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychol. Methods 10, 428–443 (2005).
5
C. Doucouliagos, T. D. Stanley, Theory competition and selectivity: Are all economic facts greatly exaggerated? J. Econ. Surv. 27, 316–339 (2013).
6
M. Maier, F. Bartoš, E. J. Wagenmakers, Robust Bayesian meta-analysis: Addressing publication bias with model-averaging. Psychol. Methods, https://doi.org/10.1037/met0000405 (2022).
7
J. L. Vevea, L. V. Hedges, A general linear model for estimating effect size in the presence of publication bias. Psychometrika 60, 419–435 (1995).
8
T. D. Stanley, H. Doucouliagos, Meta-regression approximations to reduce publication selection bias. Res. Synth. Methods 5, 60–78 (2014).
9
F. Bartoš, M. Maier, E.-J. Wagenmakers, H. Doucouliagos, T. D. Stanley, No need to choose: Model-averaging across complementary publication bias adjustment methods. Evidence Synthesis Methods, in press.
10
J. A. Hoeting, D. Madigan, A. E. Raftery, C. T. Volinsky, Bayesian model averaging: A tutorial. Stat. Sci. 14, 382–417 (1999).
11
S. Mertens, M. Herberz, U. J. J. Hahnel, T. Brosch, mhhb_nma_data_corrected.csv. Open Science Framework. https://osf.io/ubt9a/. Accessed 3 May 2022.
12
M. Maier et al., Code and data for analyses in “No evidence for nudging after adjusting for publication bias.” Open Science Framework. https://osf.io/svz6e/. Deposited 6 January 2022.

Information & Authors

Information

Published in

Go to Proceedings of the National Academy of Sciences
Go to Proceedings of the National Academy of Sciences
Proceedings of the National Academy of Sciences
Vol. 119 | No. 31
August 2, 2022
PubMed: 35858389

Classifications

Data Availability

Data and analysis script are available in ref. 11.

Submission history

Published online: July 19, 2022
Published in issue: August 2, 2022

Acknowledgments

We thank Mertens et al. for sharing well-documented data and code.

Notes

*Our analysis is based on the corrected dataset in ref. 12.
We also reanalyzed the data by including only models of selection for statistical significance, confirming our results.

Authors

Affiliations

Department of Experimental Psychology, University College London, London WC1H 0AP, United Kingdom
Department of Psychological Methods, University of Amsterdam, Amsterdam, 1018 WS, The Netherlands
T. D. Stanley
Deakin Laboratory for the Meta-Analysis of Research, Deakin University, Burwood, VIC 3125, Australia
Department of Economics, Deakin University, Burwood, VIC 3125, Australia
Department of Experimental Psychology, University College London, London WC1H 0AP, United Kingdom
Adam J. L. Harris
Department of Experimental Psychology, University College London, London WC1H 0AP, United Kingdom
Department of Psychological Methods, University of Amsterdam, Amsterdam, 1018 WS, The Netherlands

Notes

2
To whom correspondence may be addressed. Email: [email protected].
Author contributions: M.M. and F.B. designed research; M.M. and F.B. performed research; M.M. and F.B. analyzed data; and M.M., F.B., T.D.S., D.R.S., A.J.L.H., and E.-J.W. wrote the paper.
1
M.M. and F.B. contributed equally to this work.

Competing Interests

The authors declare no competing interest.

Metrics & Citations

Metrics

Note: The article usage is presented with a three- to four-day delay and will update daily once available. Due to ths delay, usage data will not appear immediately following publication. Citation information is sourced from Crossref Cited-by service.


Citation statements




Altmetrics

Citations

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.

Cited by

    Loading...

    View Options

    View options

    PDF format

    Download this article as a PDF file

    DOWNLOAD PDF

    Media

    Figures

    Tables

    Other

    Share

    Share

    Share article link

    Share on social media