New Research In
Physical Sciences
Social Sciences
Featured Portals
Articles by Topic
Biological Sciences
Featured Portals
Articles by Topic
- Agricultural Sciences
- Anthropology
- Applied Biological Sciences
- Biochemistry
- Biophysics and Computational Biology
- Cell Biology
- Developmental Biology
- Ecology
- Environmental Sciences
- Evolution
- Genetics
- Immunology and Inflammation
- Medical Sciences
- Microbiology
- Neuroscience
- Pharmacology
- Physiology
- Plant Biology
- Population Biology
- Psychological and Cognitive Sciences
- Sustainability Science
- Systems Biology
Reevaluating “cluster failure” in fMRI using nonparametric control of the false discovery rate
See related content:
- fMRI clustering and false-positive rates- Apr 25, 2017

In a substantial contribution to the fMRI field, Eklund et al. (1) use nonparametric methods to demonstrate that random field theory (RFT)-based familywise error (FWE) correction for cluster inference does not control errors appropriately, and this discrepancy is more pronounced for lenient cluster-defining thresholds (CDT). Moreover, they point to violations of RFT assumptions as the culprit for this discrepancy.
Given these results, how should we interpret existing fMRI literature that used RFT-based, FWE-corrected P values (
Here, we undertake an initial attempt at such guidance. We heed Eklund et al.’s (1) warning and prefer nonparametric null distributions to RFT. However, we focus on the false discovery rate (FDR) (2), which is a more natural target for multiple testing control [as recognized by Nichols and coworkers in previous work (3)]: A researcher is naturally more concerned with the proportion of reported clusters that are false positives (FDR) than whether any are false positives (FWE). Thus, a reader considering a table of clusters significant under RFT–FWE might ask which of these results would have survived had the study instead used a nonparametric FDR-based method.
We address this question using the same task fMRI data (4, 5) analyzed by Eklund et al. (1) (available from openfMRI, ref. 6).
For each contrast, we generate 5,000 realizations of the data through sign flipping (code, data, and extended methods: https://github.com/mangstad/FDR_permutations). To obtain a null distribution of cluster extents (for an arbitrary cluster) we combine normalized frequencies of extents at each realization. This distribution is used to assign uncorrected P values to each observed cluster. We next submit the vector of uncorrected P values for each contrast to Benjamini and Hochberg’s (2) FDR procedure with
We compare
Based on our results (Fig. 1), we suggest nearly all clusters identified as significant when using CDT = 0.001 and RFT–FWE correction are trustworthy by the nonparametric FDR benchmark. For clusters identified as significant with CDT = 0.01 and RFT–FWE correction, the guidance depends on the corrected P value: Clusters with
Assessing RFT-based FWE using an FDR benchmark. We submitted the same task data analyzed by Eklund et al. (1, 5, 6) to nonparametric clusterwise FDR analysis. For
These findings have promising implications for past fMRI studies using RFT-based cluster-level inference that used CDT = 0.001, estimated to be upward of 8,500 reports (8, 9). Although the story is mixed for CDT = 0.01 (used in ∼3,500 studies) (8, 9), our findings suggest that not all such previously reported clusters are unreliable. We identify 0.00001 as a potential cutoff for trustworthiness.
Our results provide more granular guidance on the relationship between
Acknowledgments
We thank Anders Eklund and Thomas Nichols for providing us with processed data and for very helpful comments on earlier versions of this letter.
Footnotes
↵1D.K., M.A., and C.S.S. contributed equally to this work.
- ↵2To whom correspondence should be addressed. Email: kesslerd{at}umich.edu.
Author contributions: D.K., M.A., and C.S.S. designed research, performed research, analyzed data, and wrote the paper.
The authors declare no conflict of interest.
References
- ↵.
- Eklund A,
- Nichols TE,
- Knutsson H
- ↵.
- Benjamini Y,
- Hochberg Y
- ↵
- ↵
- ↵.
- Tom SM,
- Fox CR,
- Trepel C,
- Poldrack RA
- ↵
- ↵
- ↵.
- Nichols TE
- ↵
Citation Manager Formats
Sign up for Article Alerts
You May Also be Interested in
More Articles of This Classification
Related Content
Cited by...
- Discrete Modules and Mesoscale Functional Circuits for Thermal Nociception within Primate S1 Cortex
- A Computational Account of Optimizing Social Predictions Reveals That Adolescents Are Conservative Learners in Social Contexts
- Controversy in statistical analysis of functional magnetic resonance imaging data
- Reply to Brown and Behrmann, Cox, et al., and Kessler et al.: Data and code sharing is the way forward for fMRI