Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates
Edited by Emery N. Brown, Massachusetts General Hospital, Boston, MA, and approved May 17, 2016 (received for review February 12, 2016)
Commentary
June 29, 2016
Letter
April 18, 2017
Significance
Functional MRI (fMRI) is 25 years old, yet surprisingly its most common statistical methods have not been validated using real data. Here, we used resting-state fMRI data from 499 healthy controls to conduct 3 million task group analyses. Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of a number of fMRI studies and may have a large impact on the interpretation of weakly significant neuroimaging results.
Abstract
The most widely used task functional magnetic resonance imaging (fMRI) analyses use parametric statistical methods that depend on a variety of assumptions. In this work, we use real resting-state data and a total of 3 million random task group analyses to compute empirical familywise error rates for the fMRI software packages SPM, FSL, and AFNI, as well as a nonparametric permutation method. For a nominal familywise error rate of 5%, the parametric statistical methods are shown to be conservative for voxelwise inference and invalid for clusterwise inference. Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape. By comparison, the nonparametric permutation test is found to produce nominal results for voxelwise as well as clusterwise inference. These findings speak to the need of validating the statistical methods being used in the field of neuroimaging.
Acknowledgments
We thank Robert Cox, Stephen Smith, Mark Woolrich, Karl Friston, and Guillaume Flandin, who gave us valuable feedback on this work. This study would not be possible without the recent data-sharing initiatives in the neuroimaging field. We therefore thank the Neuroimaging Informatics Tools and Resources Clearinghouse and all of the researchers who have contributed with resting-state data to the 1,000 Functional Connectomes Project. Data were also provided by the Human Connectome Project, WU-Minn Consortium (principal investigators: David Van Essen and Kamil Ugurbil; Grant 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research, and by the McDonnell Center for Systems Neuroscience at Washington University. We also thank Russ Poldrack and his colleagues for starting the OpenfMRI Project (supported by National Science Foundation Grant OCI-1131441) and all of the researchers who have shared their task-based data. The Nvidia Corporation, which donated the Tesla K40 graphics card used to run all the permutation tests, is also acknowledged. This research was supported by the Neuroeconomic Research Initiative at Linköping University, by Swedish Research Council Grant 2013-5229 (“Statistical Analysis of fMRI Data”), the Information Technology for European Advancement 3 Project BENEFIT (better effectiveness and efficiency by measuring and modelling of interventional therapy), and the Wellcome Trust.
Supporting Information
Appendix (PDF)
Supporting Information
- Download
- 3.76 MB
References
1
S Ogawa, et al., Intrinsic signal changes accompanying sensory stimulation: Functional brain mapping with magnetic resonance imaging. Proc Natl Acad Sci USA 89, 5951–5955 (1992).
2
NK Logothetis, What we can do and what we cannot do with fMRI. Nature 453, 869–878 (2008).
3
M Welvaert, Y Rosseel, A review of fMRI simulation studies. PLoS One 9, e101953 (2014).
4
BB Biswal, et al., Toward discovery science of human brain function. Proc Natl Acad Sci USA 107, 4734–4739 (2010).
5
DC Van Essen, et al., The WU-Minn Human Connectome Project: An overview. Neuroimage; WU-Minn HCP Consortium 80, 62–79 (2013).
6
RA Poldrack, KJ Gorgolewski, Making big data open: Data sharing in neuroimaging. Nat Neurosci 17, 1510–1517 (2014).
7
RA Poldrack, et al., Toward open sharing of task-based fMRI data: The OpenfMRI project. Front Neuroinform 7, 12 (2013).
8
SG Mueller, et al., The Alzheimer’s disease neuroimaging initiative. Neuroimaging Clin N Am 15, 869–877, xi–xii (2005).
9
Jr CR Jack, et al., The Alzheimer’s Disease Neuroimaging Initiative (ADNI): MRI methods. J Magn Reson Imaging 27, 685–691 (2008).
10
JB Poline, et al., Data sharing in neuroimaging research. Front Neuroinform 6, 9 (2012).
11
C Scarpazza, G Sartori, MS De Simone, A Mechelli, When the single matters more than the group: Very high false positive rates in single case voxel based morphometry. Neuroimage 70, 175–188 (2013).
12
J Ashburner, KJ Friston, Voxel-based morphometry—the methods. Neuroimage 11, 805–821 (2000).
13
M Silver, G Montana, TE Nichols, False positives in neuroimaging genetics using voxel-based morphometry data. Neuroimage; Alzheimer’s Disease Neuroimaging Initiative 54, 992–1000 (2011).
14
A Eklund, M Andersson, C Josephson, M Johannesson, H Knutsson, Does parametric fMRI analysis with SPM yield valid results? An empirical study of 1484 rest datasets. Neuroimage 61, 565–578 (2012).
15
K Friston, J Ashburner, S Kiebel, T Nichols, W Penny Statistical Parametric Mapping: The Analysis of Functional Brain Images (Elsevier/Academic, London, 2007).
16
J Ashburner, SPM: A history. Neuroimage 62, 791–800 (2012).
17
M Jenkinson, CF Beckmann, TE Behrens, MW Woolrich, SM Smith, FSL. Neuroimage 62, 782–790 (2012).
18
RW Cox, AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res 29, 162–173 (1996).
19
KJ Friston, KJ Worsley, RSJ Frackowiak, JC Mazziotta, AC Evans, Assessing the significance of focal activations using their spatial extent. Hum Brain Mapp 1, 210–220 (1994).
20
SD Forman, et al., Improved assessment of significant activation in functional magnetic resonance imaging (fMRI): Use of a cluster-size threshold. Magn Reson Med 33, 636–647 (1995).
21
CW Woo, A Krishnan, TD Wager, Cluster-extent based thresholding in fMRI analyses: Pitfalls and recommendations. Neuroimage 91, 412–419 (2014).
22
TE Nichols, AP Holmes, Nonparametric permutation tests for functional neuroimaging: A primer with examples. Hum Brain Mapp 15, 1–25 (2002).
23
AM Winkler, GR Ridgway, MA Webster, SM Smith, TE Nichols, Permutation inference for the general linear model. Neuroimage 92, 381–397 (2014).
24
MJ Brammer, et al., Generic brain activation mapping in functional magnetic resonance imaging: A nonparametric approach. Magn Reson Imaging 15, 763–770 (1997).
25
ET Bullmore, et al., Global, voxel, and cluster tests, by theory and permutation, for a difference between two groups of structural MR images of the brain. IEEE Transactions on Medical Imaging 18, 32–42 (1999).
26
J Carp, The secret lives of experiments: Methods reporting in the fMRI literature. Neuroimage 63, 289–300 (2012).
27
A Eklund, P Dufort, M Villani, S Laconte, BROCCOLI: Software for fast fMRI analysis on many-core CPUs and GPUs. Front Neuroinform 8, 24 (2014).
28
MD Lieberman, WA Cunningham, Type I and type II error concerns in fMRI research: Re-balancing the scale. Soc Cogn Affect Neurosci 4, 423–428 (2009).
29
MW Woolrich, TE Behrens, CF Beckmann, M Jenkinson, SM Smith, Multilevel linear modelling for FMRI group analysis using Bayesian inference. Neuroimage 21, 1732–1747 (2004).
30
S Hayasaka, TE Nichols, Validating cluster size inference: Random field and permutation methods. Neuroimage 20, 2343–2356 (2003).
31
N Kriegeskorte, et al., Artifactual time-course correlations in echo-planar fMRI with implications for studies of brain function. Int J Imaging Syst Technol 18, 345–349 (2008).
32
SJ Kiebel, JB Poline, KJ Friston, AP Holmes, KJ Worsley, Robust smoothness estimation in statistical parametric maps using standardized residuals from the general linear model. Neuroimage 10, 756–766 (1999).
33
S Hayasaka, KL Phan, I Liberzon, KJ Worsley, TE Nichols, Nonstationary cluster-size inference with random field and permutation methods. Neuroimage 22, 676–687 (2004).
34
G Salimi-Khorshidi, SM Smith, TE Nichols, Adjusting the effect of nonstationarity in cluster-based and TFCE inference. Neuroimage 54, 2006–2019 (2011).
35
SM Tom, CR Fox, C Trepel, RA Poldrack, The neural basis of loss aversion in decision-making under risk. Science 315, 515–518 (2007).
36
KJ Duncan, C Pattamadilok, I Knierim, JT Devlin, Consistency and variability in functional localisers. Neuroimage 46, 1018–1026 (2009).
37
JP Ioannidis, Why most published research findings are false. PLoS Med 2, e124 (2005).
38
M Pavlicová, NA Cressie, TJ Santner, Testing for activation in data from FMRI experiments. J Data Sci 4, 275–289 (2006).
39
C Scarpazza, S Tognin, S Frisciata, G Sartori, A Mechelli, False positive rates in voxel-based morphometry studies of the human brain: Should we be worried? Neurosci Biobehav Rev 52, 49–55 (2015).
40
A Meyer-Lindenberg, et al., False positives in imaging genetics. Neuroimage 40, 655–661 (2008).
41
T Nichols, S Hayasaka, Controlling the familywise error rate in functional neuroimaging: A comparative review. Stat Methods Med Res 12, 419–446 (2003).
42
DA Fair, et al., A method for using blocked and event-related fMRI data to study “resting state” functional connectivity. Neuroimage 35, 396–405 (2007).
43
A Eklund, P Dufort, D Forsberg, SM LaConte, Medical image processing on the GPU—past, present and future. Med Image Anal 17, 1073–1094 (2013).
44
KJ Gorgolewski, et al., NeuroVault.org: A repository for sharing unthresholded statistical maps, parcellations, and atlases of the human brain. Neuroimage 124, 1242–1244 (2016).
Information & Authors
Information
Published in
Cover image: Pictured are a white rat and black mouse with characteristic wavy phenotypes in the tail affected by disruption of the transcription factor Mohawk (Mkx). Hidetsugu Suzuki et al. used CRISPR/Cas9 technology to study how Mkx influences tendon development and homeostasis in rats. The results suggest that Mkx expression influences tendon cell differentiation and maintenance by boosting tendon-related genes and suppressing cartilage-related genes, findings that could illuminate tendon physiology and pathology. See the article by Suzuki et al. on pages 7840–7845. Image courtesy of Hidetsugu Suzuki and Tomohiro Kayama.
Classifications
Copyright
Freely available online through the PNAS open access option.
Submission history
Published online: June 28, 2016
Published in issue: July 12, 2016
Keywords
Acknowledgments
We thank Robert Cox, Stephen Smith, Mark Woolrich, Karl Friston, and Guillaume Flandin, who gave us valuable feedback on this work. This study would not be possible without the recent data-sharing initiatives in the neuroimaging field. We therefore thank the Neuroimaging Informatics Tools and Resources Clearinghouse and all of the researchers who have contributed with resting-state data to the 1,000 Functional Connectomes Project. Data were also provided by the Human Connectome Project, WU-Minn Consortium (principal investigators: David Van Essen and Kamil Ugurbil; Grant 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research, and by the McDonnell Center for Systems Neuroscience at Washington University. We also thank Russ Poldrack and his colleagues for starting the OpenfMRI Project (supported by National Science Foundation Grant OCI-1131441) and all of the researchers who have shared their task-based data. The Nvidia Corporation, which donated the Tesla K40 graphics card used to run all the permutation tests, is also acknowledged. This research was supported by the Neuroeconomic Research Initiative at Linköping University, by Swedish Research Council Grant 2013-5229 (“Statistical Analysis of fMRI Data”), the Information Technology for European Advancement 3 Project BENEFIT (better effectiveness and efficiency by measuring and modelling of interventional therapy), and the Wellcome Trust.
Notes
This article is a PNAS Direct Submission.
See Commentary on page 7699.
Authors
Competing Interests
The authors declare no conflict of interest.
Metrics & Citations
Metrics
Citation statements
Altmetrics
Citations
Cite this article
113 (28) 7900-7905,
Export the article citation data by selecting a format from the list below and clicking Export.
Cited by
Loading...
View Options
View options
PDF format
Download this article as a PDF file
DOWNLOAD PDFLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Personal login Institutional LoginRecommend to a librarian
Recommend PNAS to a LibrarianPurchase options
Purchase this article to access the full text.