Whether labeled a replication effort or an attempt to gauge robustness [a distinction discussed in our paper (1)], our study finds that swapping nouns for verbs in a treatment script does not produce the enormous 11–14 percentage-point turnout increase reported by Bryan et al. (2), but instead produces a precisely estimated zero-treatment effect. Because the effects in Bryan et al. are many times larger than the 0–2 percentage-point effects common in general election voter mobilization experiments, their article (2) has attracted attention as a powerful demonstration of an important broader claim: Extremely minor psychologically inspired interventions can have outsized behavioral effects.

Bryan et al. (3) explain our null results by asserting that the treatment they developed should have no effect unless certain conditions hold; we learn from Bryan et al.’s letter (3) that the election must be high-profile or competitive. However, this is an ex-post argument, absent from their 2011 article (2). We provide four reasons our null findings are unlikely to stem from political-context differences.

First, turnout effects in mobilization experiments are generally smaller, not larger, in high-profile elections because campaign communication environments are saturated and baseline participation rates are high. To see the difficulty in producing a double-digit turnout boost in such contexts, note that in Bryan et al.’s (2) California 2008 study only 18.2% of the control group (verb) did not vote. Under reasonable assumptions, the reported 13.7 percentage-point treatment effect suggests the subtle noun treatment caused a remarkable 75 percentage-point (that is, 13.7 divided by 18.2) turnout increase among the 18.2% who would not have voted absent treatment.

Second, although the Bryan et al. letter (3) faults our use of expected competitiveness, when our sample’s 31 districts are partitioned by realized closeness, there is still no evidence of meaningful treatment effects (table S2 in ref. 4). For the subset in which the closest primary contest in a district was decided by less than 5%, the turnout difference between noun and verb groups is just 0.2 points (SE = 2.7, n = 829).

Third, Bryan et al.’s letter (3) presents a new survey in which subjects rank the importance of voting in elections like Bryan et al.’s (2) New Jersey 2009 contest or an uncompetitive House primary. Despite extreme question wording (characterizing all primaries as uncompetitive), the survey shows the psychological attachment to voting is relatively invariant across these contexts, and respondents frequently indicate voting in primaries is important. Crucially, because such a large fraction of subjects say primary election voting is important, even if the treatment is effective only among this subset of voters (a very strong assumption), if the true effects are as large as those reported in Bryan et al. (2) (table S5 in ref. 4) we should still observe substantial aggregate effects in House primaries.

Fourth, false positives are common in small-scale experiments [in Bryan et al. (2), n = 214 for New Jersey, n = 88 for California] (5).

Although we find Bryan et al.’s (3) explanation unconvincing, this exchange is well-timed. The original findings have (to our knowledge) never been successfully replicated, and this November provides ample opportunity to test noun vs. verb in the political environment Bryan et al. (3) suggest is ideal for producing 11–14 percentage-point effects.