Skip to main content
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian
  • Log in
  • My Cart

Main menu

  • Home
  • Articles
    • Current
    • Latest Articles
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • Archive
  • Front Matter
  • News
    • For the Press
    • Highlights from Latest Articles
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Purpose and Scope
    • Editorial and Journal Policies
    • Submission Procedures
    • For Reviewers
    • Author FAQ
  • Submit
  • About
    • Editorial Board
    • PNAS Staff
    • FAQ
    • Rights and Permissions
    • Site Map
  • Contact
  • Journal Club
  • Subscribe
    • Subscription Rates
    • Subscriptions FAQ
    • Open Access
    • Recommend PNAS to Your Librarian

User menu

  • Log in
  • My Cart

Search

  • Advanced search
Home
Home

Advanced Search

  • Home
  • Articles
    • Current
    • Latest Articles
    • Special Features
    • Colloquia
    • Collected Articles
    • PNAS Classics
    • Archive
  • Front Matter
  • News
    • For the Press
    • Highlights from Latest Articles
    • PNAS in the News
  • Podcasts
  • Authors
    • Information for Authors
    • Purpose and Scope
    • Editorial and Journal Policies
    • Submission Procedures
    • For Reviewers
    • Author FAQ

New Research In

Physical Sciences

Featured Portals

  • Physics
  • Chemistry
  • Sustainability Science

Articles by Topic

  • Applied Mathematics
  • Applied Physical Sciences
  • Astronomy
  • Computer Sciences
  • Earth, Atmospheric, and Planetary Sciences
  • Engineering
  • Environmental Sciences
  • Mathematics
  • Statistics

Social Sciences

Featured Portals

  • Anthropology
  • Sustainability Science

Articles by Topic

  • Economic Sciences
  • Environmental Sciences
  • Political Sciences
  • Psychological and Cognitive Sciences
  • Social Sciences

Biological Sciences

Featured Portals

  • Sustainability Science

Articles by Topic

  • Agricultural Sciences
  • Anthropology
  • Applied Biological Sciences
  • Biochemistry
  • Biophysics and Computational Biology
  • Cell Biology
  • Developmental Biology
  • Ecology
  • Environmental Sciences
  • Evolution
  • Genetics
  • Immunology and Inflammation
  • Medical Sciences
  • Microbiology
  • Neuroscience
  • Pharmacology
  • Physiology
  • Plant Biology
  • Population Biology
  • Psychological and Cognitive Sciences
  • Sustainability Science
  • Systems Biology

fMRI clustering and false-positive rates

Robert W. Cox, Gang Chen, Daniel R. Glen, Richard C. Reynolds, and Paul A. Taylor
PNAS April 25, 2017 114 (17) E3370-E3371; published ahead of print April 18, 2017 https://doi.org/10.1073/pnas.1614961114
Robert W. Cox
aScientific and Statistical Computing Core, National Institute of Mental Health, US Department of Health and Human Services, National Institutes of Health, Bethesda, MD 20892
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: robertcox@mail.nih.gov
Gang Chen
aScientific and Statistical Computing Core, National Institute of Mental Health, US Department of Health and Human Services, National Institutes of Health, Bethesda, MD 20892
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Daniel R. Glen
aScientific and Statistical Computing Core, National Institute of Mental Health, US Department of Health and Human Services, National Institutes of Health, Bethesda, MD 20892
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Richard C. Reynolds
aScientific and Statistical Computing Core, National Institute of Mental Health, US Department of Health and Human Services, National Institutes of Health, Bethesda, MD 20892
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Paul A. Taylor
aScientific and Statistical Computing Core, National Institute of Mental Health, US Department of Health and Human Services, National Institutes of Health, Bethesda, MD 20892
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site

See related content:

  • Controversy in statistical analysis of functional magnetic resonance imaging data
    - Apr 25, 2017
  • Reevaluating “cluster failure” in fMRI using nonparametric control of the false discovery rate
    - Apr 25, 2017

This article has letters. Please see:

  • Reply to Brown and Behrmann, Cox, et al., and Kessler et al.: Data and code sharing is the way forward for fMRI
  • Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates
  • Article
  • Figures & SI
  • Info & Metrics
  • PDF
Loading

Recently, Eklund et al. (1) analyzed clustering methods in standard fMRI packages: AFNI (which we maintain), FSL, and SPM. They claim that (i) false-positive rates (FPRs) in traditional approaches are greatly inflated, questioning the validity of “countless published fMRI studies”; (ii) nonparametric methods produce valid, but slightly conservative, FPRs; (iii) a common flawed assumption is that the spatial autocorrelation function (ACF) of fMRI noise is Gaussian-shaped; and (iv) a 15-y-old bug in AFNI’s 3dClustSim significantly contributed to producing “particularly high” FPRs compared with other software. We repeated simulations from ref. 1 [Beijing_Zang data (2), cf. ref. 3) and comment on each point briefly.

AFNI and 3dClustSim

Fig. 1 A–D compares results of the “buggy” and “fixed” 3dClustSim. For each simulation, the typical difference was small: ΔFPR≲3−5% at per-voxel P=0.01 and ≲1−2% for P=0.001. The bug had only a minor impact.

Fig. 1.
  • Download figure
  • Open in new tab
  • Download powerpoint
Fig. 1.

FPRs for various software scenarios, with 1,000 two-sample one-sided t-tests (as in ref. 1; cf. ref. 3 for more details) using 20 subjects’ data in each sample. For “buggy” (A and B) and “fixed” (C and D), cluster-size thresholds were selected using the Gaussian shape model with the FWHM being the median of the 40 individual subjects’ values: “buggy” via 3dClustSim before the bug fix, “fixed” via 3dClustSim after the bug fix. For “mixed ACF” (E and F), the cluster-size threshold was selected using a non-Gaussian ACF model allowing for heavy tails (3). For “nonparam” (G and H), 3dttest++ was used to perform spatial model-free, nonparametric permutation testing (3); paired, two-sided, and tests with covariates gave similar results. Two different per-voxel P-value thresholds are shown. The black line shows the nominal 5% FPR (out of 1,000 trials), and the gray band shows its binomial 95% confidence interval, 3.65–6.35%. As in ref. 1, different smoothing values were tested (4–10 mm), and four test designs were used: B1 = 10-s block; B2 = 30-s block; E1 = regular event-related; E2 = randomized event-related.

Figures 1 and 2 of ref. 1 actually show similar FPRs for AFNI, FSL-OLS, and SPM: Most tests were in a range of 20−40% FPR at P=0.01 and 5−15% FPR at P=0.001 (nor did their famous 70% FPR come from AFNI). The data given in the Results section of ref. 1 simply do not support the statement in the Discussion section that AFNI had “particularly high” FPRs.

Smoothness

To test the effect of assuming a Gaussian ACF in fMRI noise, an empirical “mixed ACF” allowing for longer tails was computed from residuals (3). All FPRs (Fig. 1 E and F) decreased. Block designs remained >5%, likely reflecting dependence of the noise’s spatial smoothness on temporal frequency. Heavy tails in spatial smoothness indeed have significant consequences for clustering.

Nonparametric Approach

A spatial model-free, nonparametric randomization approach was added to AFNI’s group-level GLM program, 3dttest++ (3). All FPRs (Fig. 1 G and H) were within the nominal confidence interval. Although this approach shows promise (as in ref. 1), it may not be feasible to generalize nonparametric permutations to complicated covariate structures and models (e.g., complex ANOVA, analysis of covariance, or linear mixed effects) (4, 5).

Inflated FPRs

Several cases showed significant FPR inflation across existing fMRI software within the testing framework of ref. 1. However, deviations from nominal FPR were not uniformly large and depended strongly on several factors. Fig. 1 and figure 1 of ref. 1 show quite good cluster results for stricter per-voxel P values (which ref. 6 found to be predominantly used in fMRI analyses) and for event-related stimuli (emphasizing the importance of good experimental design): FPR inflation was often ≲10% (Beijing) or ≲5% (Cambridge), affecting only clusters with marginally significant volume.

We strongly disagree with Eklund et al.’s (1) summary statement: “Alarmingly, the parametric methods can give a very high degree of false positives (up to 70%, compared with the nominal 5%).” For comparison, their own nonparametric method’s results actually showed up to 40% FPR. When characterizing results, medians or percentile ranges are generally more informative summary statistics than maxima. Looking backward, the typical ranges show much smaller FPR inflation than what had been highlighted, and looking forward they provide useful suggestions for experimental design and analyses (lower voxelwise P, event-related paradigms, etc.). By concentrating on the highest observed FPRs, the conclusions of Eklund et al. (1) are unnecessarily alarmist.

Acknowledgments

This work was supported by the National Institute of Mental Health and National Institute of Neurological Disorders and Stroke Intramural Research Programs ZICMH002888 of the NIH, US Department of Health and Human Services. This work used the computational resources of the NIH High-Performance Computing Biowulf cluster (https://hpc.nih.gov/).

Footnotes

  • ↵1To whom correspondence should be addressed. Email: robertcox{at}mail.nih.gov.
  • Author contributions: R.W.C. designed research; R.W.C. performed research; R.W.C. contributed new reagents/analytic tools; R.W.C. and R.C.R. analyzed data; and R.W.C., G.C., D.R.G., R.C.R., and P.A.T. wrote the paper.

  • The authors declare no conflict of interest.

References

  1. ↵
    1. Eklund A,
    2. Nichols TE,
    3. Knutsson H
    (2016) Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. Proc Natl Acad Sci USA 113:7900–7905. Erratum in Proc Natl Acad Sci USA 113:E4929.
    .
  2. ↵
    1. Biswal BB, et al.
    (2010) Toward discovery science of human brain function. Proc Natl Acad Sci USA 107:4734–4739.
    .
    OpenUrlAbstract/FREE Full Text
  3. ↵
    1. Cox RW,
    2. Reynolds RC,
    3. Taylor PA
    (2016) AFNI and clustering: False positive rates redux. bioRxiv 065862, doi:10.1101/065862.
    .
    OpenUrlAbstract/FREE Full Text
  4. ↵
    1. Chen G,
    2. Adleman NE,
    3. Saad ZS,
    4. Leibenluft E,
    5. Cox RW
    (2014) Applications of multivariate modeling to neuroimaging group analysis: A comprehensive alternative to univariate general linear model. Neuroimage 99:571–588.
    .
    OpenUrlCrossRefPubMed
  5. ↵
    1. Chen G,
    2. Saad ZS,
    3. Britton JC,
    4. Pine DS,
    5. Cox RW
    (2013) Linear mixed-effects modeling approach to FMRI group analysis. Neuroimage 73:176–190.
    .
    OpenUrlCrossRefPubMed
  6. ↵
    1. Carp J
    (2012) The secret lives of experiments: Methods reporting in the fMRI literature. Neuroimage 63:289–300.
    .
    OpenUrlCrossRefPubMed
View Abstract
PreviousNext
Back to top
Article Alerts
Email Article

Thank you for your interest in spreading the word on PNAS.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
fMRI clustering and false-positive rates
(Your Name) has sent you a message from PNAS
(Your Name) thought you would like to see the PNAS web site.
Citation Tools
fMRI clustering and false-positive rates
Robert W. Cox, Gang Chen, Daniel R. Glen, Richard C. Reynolds, Paul A. Taylor
Proceedings of the National Academy of Sciences Apr 2017, 114 (17) E3370-E3371; DOI: 10.1073/pnas.1614961114

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Request Permissions
Share
fMRI clustering and false-positive rates
Robert W. Cox, Gang Chen, Daniel R. Glen, Richard C. Reynolds, Paul A. Taylor
Proceedings of the National Academy of Sciences Apr 2017, 114 (17) E3370-E3371; DOI: 10.1073/pnas.1614961114
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Mendeley logo Mendeley
Proceedings of the National Academy of Sciences: 116 (8)
Current Issue

Submit

Sign up for Article Alerts

Jump to section

  • Article
    • AFNI and 3dClustSim
    • Smoothness
    • Nonparametric Approach
    • Inflated FPRs
    • Acknowledgments
    • Footnotes
    • References
  • Figures & SI
  • Info & Metrics
  • PDF

You May Also be Interested in

News Feature: Cities serve as testbeds for evolutionary change
Urban living can pressure flora and fauna to adapt in intriguing ways. Biologists are starting to take advantage of this convenient laboratory of evolution.
Image credit: Kristin Winchell (Washington University in St. Louis, St. Louis).
Several aspects of the proposal, which aims to expand open access, require serious discussion and, in some cases, a rethink.
Opinion: “Plan S” falls short for society publishers—and for the researchers they serve
Several aspects of the proposal, which aims to expand open access, require serious discussion and, in some cases, a rethink.
Image credit: Dave Cutler (artist).
Featured Profile
PNAS Profile of NAS member and biochemist Hao Wu
 Nonmonogamous strawberry poison frog (Oophaga pumilio).  Image courtesy of Yusan Yang (University of Pittsburgh, Pittsburgh).
Putative signature of monogamy
A study suggests a putative gene-expression hallmark common to monogamous male vertebrates of some species, namely cichlid fishes, dendrobatid frogs, passeroid songbirds, common voles, and deer mice, and identifies 24 candidate genes potentially associated with monogamy.
Image courtesy of Yusan Yang (University of Pittsburgh, Pittsburgh).
Active lifestyles. Image courtesy of Pixabay/MabelAmber.
Meaningful life tied to healthy aging
Physical and social well-being in old age are linked to self-assessments of life worth, and a spectrum of behavioral, economic, health, and social variables may influence whether aging individuals believe they are leading meaningful lives.
Image courtesy of Pixabay/MabelAmber.

More Articles of This Classification

  • Need for early, minimally invasive cancer diagnosis
  • Reply to Schellenberg: Is there more to auditory plasticity than meets the ear?
  • Reply to Rajendran and Schnupp: Frequency tagging is sensitive to the temporal structure of signals
Show more

Related Content

  • Data and code sharing is the way forward for fMRI
  • Controversy in statistical analysis of fMRI data
  • “Cluster failure” and the false discovery rate
  • Cluster failure: Inflated false positives for fMRI
  • Scopus
  • PubMed
  • Google Scholar

Cited by...

  • Changes in Endogenous Dopamine Induced by Methylphenidate Predict Functional Connectivity in Nonhuman Primates
  • The Integration of Functional Brain Activity from Adolescence to Adulthood
  • Discrete Modules and Mesoscale Functional Circuits for Thermal Nociception within Primate S1 Cortex
  • Early childhood stress exposure, reward pathways, and adult decision making
  • Hierarchically Organized Medial Frontal Cortex-Basal Ganglia Loops Selectively Control Task- and Response-Selection
  • Reply to Brown and Behrmann, Cox, et al., and Kessler et al.: Data and code sharing is the way forward for fMRI
  • Controversy in statistical analysis of functional magnetic resonance imaging data
  • Scopus (38)
  • Google Scholar

Similar Articles

Site Logo
Powered by HighWire
  • Submit Manuscript
  • Twitter
  • Facebook
  • RSS Feeds
  • Email Alerts

Articles

  • Current Issue
  • Latest Articles
  • Archive

PNAS Portals

  • Classics
  • Front Matter
  • Teaching Resources
  • Anthropology
  • Chemistry
  • Physics
  • Sustainability Science

Information

  • Authors
  • Editorial Board
  • Reviewers
  • Press
  • Site Map

Feedback    Privacy/Legal

Copyright © 2019 National Academy of Sciences. Online ISSN 1091-6490