Skip to main content
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian
  • Log in
  • My Cart

Main menu

  • Home
  • Articles
    • Current
    • Latest Articles
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • Archive
  • Front Matter
  • News
    • For the Press
    • Highlights from Latest Articles
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Purpose and Scope
    • Editorial and Journal Policies
    • Submission Procedures
    • For Reviewers
    • Author FAQ
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian

User menu

  • Log in
  • My Cart

Search

  • Advanced search
Home
Home

Advanced Search

  • Home
  • Articles
    • Current
    • Latest Articles
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • Archive
  • Front Matter
  • News
    • For the Press
    • Highlights from Latest Articles
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Purpose and Scope
    • Editorial and Journal Policies
    • Submission Procedures
    • For Reviewers
    • Author FAQ

New Research In

Physical Sciences

Featured Portals

  • Physics
  • Chemistry
  • Sustainability Science

Articles by Topic

  • Applied Mathematics
  • Applied Physical Sciences
  • Astronomy
  • Computer Sciences
  • Earth, Atmospheric, and Planetary Sciences
  • Engineering
  • Environmental Sciences
  • Mathematics
  • Statistics

Social Sciences

Featured Portals

  • Anthropology
  • Sustainability Science

Articles by Topic

  • Economic Sciences
  • Environmental Sciences
  • Political Sciences
  • Psychological and Cognitive Sciences
  • Social Sciences

Biological Sciences

Featured Portals

  • Sustainability Science

Articles by Topic

  • Agricultural Sciences
  • Anthropology
  • Applied Biological Sciences
  • Biochemistry
  • Biophysics and Computational Biology
  • Cell Biology
  • Developmental Biology
  • Ecology
  • Environmental Sciences
  • Evolution
  • Genetics
  • Immunology and Inflammation
  • Medical Sciences
  • Microbiology
  • Neuroscience
  • Pharmacology
  • Physiology
  • Plant Biology
  • Population Biology
  • Psychological and Cognitive Sciences
  • Sustainability Science
  • Systems Biology

Reevaluating “cluster failure” in fMRI using nonparametric control of the false discovery rate

Daniel Kessler, Mike Angstadt, and Chandra S. Sripada
PNAS April 25, 2017 114 (17) E3372-E3373; published ahead of print April 18, 2017 https://doi.org/10.1073/pnas.1614502114
Daniel Kessler
aDepartment of Psychiatry, University of Michigan, Ann Arbor, MI 48109
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Daniel Kessler
  • For correspondence: kesslerd@umich.edu
Mike Angstadt
aDepartment of Psychiatry, University of Michigan, Ann Arbor, MI 48109
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Mike Angstadt
Chandra S. Sripada
aDepartment of Psychiatry, University of Michigan, Ann Arbor, MI 48109
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site

See related content:

  • fMRI clustering and false-positive rates
    - Apr 25, 2017
  • Controversy in statistical analysis of functional magnetic resonance imaging data
    - Apr 25, 2017

This article has letters. Please see:

  • Reply to Brown and Behrmann, Cox, et al., and Kessler et al.: Data and code sharing is the way forward for fMRI
  • Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates
  • Article
  • Figures & SI
  • Info & Metrics
  • PDF
Loading

In a substantial contribution to the fMRI field, Eklund et al. (1) use nonparametric methods to demonstrate that random field theory (RFT)-based familywise error (FWE) correction for cluster inference does not control errors appropriately, and this discrepancy is more pronounced for lenient cluster-defining thresholds (CDT). Moreover, they point to violations of RFT assumptions as the culprit for this discrepancy.

Given these results, how should we interpret existing fMRI literature that used RFT-based, FWE-corrected P values (pRFT-FWE)? To suggest caution is reasonable but incomplete; we require concrete, quantitative guidelines to enable appropriate calibration of skepticism.

Here, we undertake an initial attempt at such guidance. We heed Eklund et al.’s (1) warning and prefer nonparametric null distributions to RFT. However, we focus on the false discovery rate (FDR) (2), which is a more natural target for multiple testing control [as recognized by Nichols and coworkers in previous work (3)]: A researcher is naturally more concerned with the proportion of reported clusters that are false positives (FDR) than whether any are false positives (FWE). Thus, a reader considering a table of clusters significant under RFT–FWE might ask which of these results would have survived had the study instead used a nonparametric FDR-based method.

We address this question using the same task fMRI data (4, 5) analyzed by Eklund et al. (1) (available from openfMRI, ref. 6).

For each contrast, we generate 5,000 realizations of the data through sign flipping (code, data, and extended methods: https://github.com/mangstad/FDR_permutations). To obtain a null distribution of cluster extents (for an arbitrary cluster) we combine normalized frequencies of extents at each realization. This distribution is used to assign uncorrected P values to each observed cluster. We next submit the vector of uncorrected P values for each contrast to Benjamini and Hochberg’s (2) FDR procedure with αFDR=.05 (cf. ref. 7 for a parametric implementation of clusterwise FDR).

We compare pRFT-FWE values to qFDR values and note whether they survive FDR correction under αFDR=.05. We generate separate plots for this analysis conducted at CDT = {0.001, 0.01}.

Based on our results (Fig. 1), we suggest nearly all clusters identified as significant when using CDT = 0.001 and RFT–FWE correction are trustworthy by the nonparametric FDR benchmark. For clusters identified as significant with CDT = 0.01 and RFT–FWE correction, the guidance depends on the corrected P value: Clusters with pRFT-FWE<.00001 seem consistently trustworthy by the nonparametric FDR benchmark, whereas clusters above this value are not reliably trustworthy.

Fig. 1.
  • Download figure
  • Open in new tab
  • Download powerpoint
Fig. 1.

Assessing RFT-based FWE using an FDR benchmark. We submitted the same task data analyzed by Eklund et al. (1, 5, 6) to nonparametric clusterwise FDR analysis. For CDT=.001 (Top), RFT-based FWE approximates effective FDR control with αFDR=.05. For CDT=.01 (Bottom), only clusters with pRFT-FWE≤.00001 reliably survived correction at αFDR=.05.

These findings have promising implications for past fMRI studies using RFT-based cluster-level inference that used CDT = 0.001, estimated to be upward of 8,500 reports (8, 9). Although the story is mixed for CDT = 0.01 (used in ∼3,500 studies) (8, 9), our findings suggest that not all such previously reported clusters are unreliable. We identify 0.00001 as a potential cutoff for trustworthiness.

Our results provide more granular guidance on the relationship between pRFT-FWE and trustworthiness of results. A more comprehensive examination of fMRI task datasets that used RFT-based FWE can further refine this guidance.

Acknowledgments

We thank Anders Eklund and Thomas Nichols for providing us with processed data and for very helpful comments on earlier versions of this letter.

Footnotes

  • ↵1D.K., M.A., and C.S.S. contributed equally to this work.

  • ↵2To whom correspondence should be addressed. Email: kesslerd{at}umich.edu.
  • Author contributions: D.K., M.A., and C.S.S. designed research, performed research, analyzed data, and wrote the paper.

  • The authors declare no conflict of interest.

References

  1. ↵
    1. Eklund A,
    2. Nichols TE,
    3. Knutsson H
    (2016) Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. Proc Natl Acad Sci USA 113:7900–7905. Erratum in Proc Natl Acad Sci USA 113:E4929.
    .
  2. ↵
    1. Benjamini Y,
    2. Hochberg Y
    (1995) Controlling the false discovery rate: A practical and powerful approach to multiple testing. J R Stat Soc B 57:289–300.
    .
    OpenUrl
  3. ↵
    1. Genovese CR,
    2. Lazar NA,
    3. Nichols T
    (2002) Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 15:870–878.
    .
    OpenUrlCrossRefPubMed
  4. ↵
    1. Duncan KJ,
    2. Pattamadilok C,
    3. Knierim I,
    4. Devlin JT
    (2009) Consistency and variability in functional localisers. Neuroimage 46:1018–1026.
    .
    OpenUrlCrossRefPubMed
  5. ↵
    1. Tom SM,
    2. Fox CR,
    3. Trepel C,
    4. Poldrack RA
    (2007) The neural basis of loss aversion in decision-making under risk. Science 315:515–518.
    .
    OpenUrlAbstract/FREE Full Text
  6. ↵
    1. Poldrack RA, et al.
    (2013) Toward open sharing of task-based fMRI data: The OpenfMRI project. Front Neuroinform 7:12.
    .
    OpenUrlPubMed
  7. ↵
    1. Chumbley JR,
    2. Friston KJ
    (2009) False discovery rate revisited: FDR and topological inference using Gaussian random fields. Neuroimage 44:62–70.
    .
    OpenUrlCrossRefPubMed
  8. ↵
    1. Nichols TE
    (2016) Bibliometrics of cluster inference. Available at blogs.warwick.ac.uk/nichols/entry/bibliometrics_of_cluster/.
    .
  9. ↵
    1. Woo CW,
    2. Krishnan A,
    3. Wager TD
    (2014) Cluster-extent based thresholding in fMRI analyses: Pitfalls and recommendations. Neuroimage 91:412–419.
    .
    OpenUrlCrossRefPubMed
View Abstract
PreviousNext
Back to top
Article Alerts
Email Article

Thank you for your interest in spreading the word on PNAS.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Reevaluating “cluster failure” in fMRI using nonparametric control of the false discovery rate
(Your Name) has sent you a message from PNAS
(Your Name) thought you would like to see the PNAS web site.
Citation Tools
“Cluster failure” and the false discovery rate
Daniel Kessler, Mike Angstadt, Chandra S. Sripada
Proceedings of the National Academy of Sciences Apr 2017, 114 (17) E3372-E3373; DOI: 10.1073/pnas.1614502114

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Request Permissions
Share
“Cluster failure” and the false discovery rate
Daniel Kessler, Mike Angstadt, Chandra S. Sripada
Proceedings of the National Academy of Sciences Apr 2017, 114 (17) E3372-E3373; DOI: 10.1073/pnas.1614502114
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Mendeley logo Mendeley
Proceedings of the National Academy of Sciences: 116 (7)
Current Issue

Submit

Sign up for Article Alerts

Jump to section

  • Article
    • Acknowledgments
    • Footnotes
    • References
  • Figures & SI
  • Info & Metrics
  • PDF

You May Also be Interested in

Several aspects of the proposal, which aims to expand open access, require serious discussion and, in some cases, a rethink.
Opinion: “Plan S” falls short for society publishers—and for the researchers they serve
Several aspects of the proposal, which aims to expand open access, require serious discussion and, in some cases, a rethink.
Image credit: Dave Cutler (artist).
Several large or long-lived animals seem strangely resistant to developing cancer. Elucidating the reasons why could lead to promising cancer-fighting strategies in humans.
Core Concept: Solving Peto’s Paradox to better understand cancer
Several large or long-lived animals seem strangely resistant to developing cancer. Elucidating the reasons why could lead to promising cancer-fighting strategies in humans.
Image credit: Shutterstock.com/ronnybas frimages.
Featured Profile
PNAS Profile of NAS member and biochemist Hao Wu
 Nonmonogamous strawberry poison frog (Oophaga pumilio).  Image courtesy of Yusan Yang (University of Pittsburgh, Pittsburgh).
Putative signature of monogamy
A study suggests a putative gene-expression hallmark common to monogamous male vertebrates of some species, namely cichlid fishes, dendrobatid frogs, passeroid songbirds, common voles, and deer mice, and identifies 24 candidate genes potentially associated with monogamy.
Image courtesy of Yusan Yang (University of Pittsburgh, Pittsburgh).
Active lifestyles. Image courtesy of Pixabay/MabelAmber.
Meaningful life tied to healthy aging
Physical and social well-being in old age are linked to self-assessments of life worth, and a spectrum of behavioral, economic, health, and social variables may influence whether aging individuals believe they are leading meaningful lives.
Image courtesy of Pixabay/MabelAmber.

More Articles of This Classification

  • SNPs deciding the rapid growth of cyanobacteria are alterable
  • Reply to Zhou and Li: Plasticity of the genomic haplotype of Synechococcus elongatus leads to rapid strain adaptation under laboratory conditions
  • Genetic variant rs17185536 regulates SIM1 gene expression in human brain hypothalamus
Show more

Related Content

  • fMRI clustering and false-positive rates
  • Data and code sharing is the way forward for fMRI
  • Controversy in statistical analysis of fMRI data
  • Cluster failure: Inflated false positives for fMRI
  • Scopus
  • PubMed
  • Google Scholar

Cited by...

  • Discrete Modules and Mesoscale Functional Circuits for Thermal Nociception within Primate S1 Cortex
  • A Computational Account of Optimizing Social Predictions Reveals That Adolescents Are Conservative Learners in Social Contexts
  • Controversy in statistical analysis of functional magnetic resonance imaging data
  • Reply to Brown and Behrmann, Cox, et al., and Kessler et al.: Data and code sharing is the way forward for fMRI
  • Scopus (10)
  • Google Scholar

Similar Articles

Site Logo
Powered by HighWire
  • Submit Manuscript
  • Twitter
  • Facebook
  • RSS Feeds
  • Email Alerts

Articles

  • Current Issue
  • Latest Articles
  • Archive

PNAS Portals

  • Classics
  • Front Matter
  • Teaching Resources
  • Anthropology
  • Chemistry
  • Physics
  • Sustainability Science

Information

  • Authors
  • Editorial Board
  • Reviewers
  • Press
  • Site Map

Feedback    Privacy/Legal

Copyright © 2019 National Academy of Sciences. Online ISSN 1091-6490