The Quantitative Evaluation of Functional Neuroimaging Experiments: Mutual Information Learning Curves

doi:10.1006/nimg.2001.1033

NeuroImage

Volume 15, Issue 4, April 2002, Pages 772-786

https://doi.org/10.1006/nimg.2001.1033 Get rights and content

Abstract

Learning curves are presented as an unbiased means for evaluating the performance of models for neuroimaging data analysis. The learning curve measures the predictive performance in terms of the generalization or prediction error as a function of the number of independent examples (e.g., subjects) used to determine the parameters in the model. Cross-validation resampling is used to obtain unbiased estimates of a generic multivariate Gaussian classifier, for training set sizes from 2 to 16 subjects. We apply the framework to four different activation experiments, in this case [¹⁵O]water data sets, although the framework is equally valid for multisubject fMRI studies. We demonstrate how the prediction error can be expressed as the mutual information between the scan and the scan label, measured in units of bits. The mutual information learning curve can be used to evaluate the impact of different methodological choices, e.g., classification label schemes, preprocessing choices. Another application for the learning curve is to examine the model performance using bias/variance considerations enabling the researcher to determine if the model performance is limited by statistical bias or variance. We furthermore present the sensitivity map as a general method for extracting activation maps from statistical models within the probabilistic framework and illustrate relationships between mutual information and pattern reproducibility as derived in the NPAIRS framework described in a companion paper.

References (36)

S.A. Frutiger et al.
PET studies of perceptuomotor learning in a mirror-reversal paradigm
NeuroImage
(1998)
S.A. Frutiger et al.
Multivariate predictive relationship between kinematic and functional ativation patterns in a PET study of visiomotor learning
NeuroImage
(2000)
L. Hansen et al.
Generalizable patterns in neuroimaging: How many principal components?
NeuroImage
(1999)
S.A. Muley et al.
Effects of changes in experimental design on PET studies of isometric force
NeuroImage
(2001)
P. Skudlarski et al.
ROC analysis of statistical methods used in functional MRI: Individual subjects
NeuroImage
(1999)
S.C. Strother et al.
The quantitative evaluation of functional neuroimaging experiments: The NPAIRS data analysis framework
NeuroImage
(2002)
S.C. Strother et al.
Multidimensional state spaces for fMRI and PET activation studies
NeuroImage
(1996)
K.J. Worsley et al.
Characterizing the response of PET and fMRI data using multivariate linear models
NeuroImage
(1997)
C. Bishop
Bayesian pca
C.M. Bishop
Neural Networks for Pattern Recognition
(1995)

C.J.C. Burges

A tutorial on support vector machines for pattern recognition

Data Mining Knowl. Discovery

(1998)

V. Cherkassky et al.

Learning from Data: Concepts, Theory and Methods

(1998)

T.M. Cover et al.

Elements of Information Theory

(1991)

R.O. Duda et al.

Pattern Classification

(2001)

K. Friston et al.

Spatial registration and normalization of images

Hum. Brain Mapping

(1995)

K.J. Friston et al.

Statistical parametric maps in functional neuroimaging: A general linear approach

Hum. Brain Mapping

(1995)

K. Friston et al.

A multivariate analysis of pet activation studies

Hum. Brain Mapping

(1996)

Cited by (59)

Sensitivity maps of the Hilbert–Schmidt independence criterion
2018, Applied Soft Computing Journal
Kernel dependence measures yield accurate estimates of nonlinear relations between random variables, and they are also endorsed with solid theoretical properties and convergence rates. Besides, the empirical estimates are easy to compute in closed form just involving linear algebra operations. However, they are hampered by two important problems: the high computational cost involved, as two kernel matrices of the sample size have to be computed and stored, and the interpretability of the measure, which remains hidden behind the implicit feature map. We here address these two issues. We introduce the sensitivity maps (SMs) for the Hilbert–Schmidt independence criterion (HSIC). Sensitivity maps allow us to explicitly analyze and visualize the relative relevance of both examples and features on the dependence measure. We also present the randomized HSIC (RHSIC) and its corresponding sensitivity maps to cope with large scale problems. We build upon the framework of random features and the Bochner's theorem to approximate the involved kernels in the canonical HSIC. The power of the RHSIC measure scales favourably with the number of samples, and it approximates HSIC and the sensitivity maps efficiently. Convergence bounds of both the measure and the sensitivity map are also provided. Our proposals are illustrated in many synthetic illustrative examples, and challenging real problems of dependence estimation, feature selection, and causal inference from empirical data. The methods allow estimating and visualizing HSIC in large scale data regimes, while still yielding closed-form analytical solutions.
Predicting aphasia type from brain damage measured with structural MRI
2015, Cortex
Chronic aphasia is a common consequence of a left-hemisphere stroke. Since the early insights by Broca and Wernicke, studying the relationship between the loci of cortical damage and patterns of language impairment has been one of the concerns of aphasiology. We utilized multivariate classification in a cross-validation framework to predict the type of chronic aphasia from the spatial pattern of brain damage. Our sample consisted of 98 patients with five types of aphasia (Broca's, Wernicke's, global, conduction, and anomic), classified based on scores on the Western Aphasia Battery (WAB). Binary lesion maps were obtained from structural MRI scans (obtained at least 6 months poststroke, and within 2 days of behavioural assessment); after spatial normalization, the lesions were parcellated into a disjoint set of brain areas. The proportion of damage to the brain areas was used to classify patients' aphasia type. To create this parcellation, we relied on five brain atlases; our classifier (support vector machine – SVM) could differentiate between different kinds of aphasia using any of the five parcellations. In our sample, the best classification accuracy was obtained when using a novel parcellation that combined two previously published brain atlases, with the first atlas providing the segmentation of grey matter, and the second atlas used to segment the white matter. For each aphasia type, we computed the relative importance of different brain areas for distinguishing it from other aphasia types; our findings were consistent with previously published reports of lesion locations implicated in different types of aphasia. Overall, our results revealed that automated multivariate classification could distinguish between aphasia types based on damage to atlas-defined brain areas.
Pattern classification of fMRI data: Applications for analysis of spatially distributed cortical networks
2014, NeuroImage
Citation Excerpt :
Selecting the best method for data analysis is still an open question. A popular analysis approach is that of pattern classification, where brain patterns are examined to predict the behavioral task being performed when a given brain volume was acquired (see, for example, Haxby et al., 2001; Kamitani and Tong, 2005; Kjems et al., 2002; Mitchell et al., 2004; Morch et al., 1997; Strother et al., 2002; and review papers by Haynes and Rees, 2006; Norman et al., 2006; Pereira et al., 2009). This is frequently posed as a binary classification problem, where two classes of brain volumes correspond to two behavioral task conditions, and the classifier models attempt to predict the task conditions under which individual brain volumes were acquired.
The field of fMRI data analysis is rapidly growing in sophistication, particularly in the domain of multivariate pattern classification. However, the interaction between the properties of the analytical model and the parameters of the BOLD signal (e.g. signal magnitude, temporal variance and functional connectivity) is still an open problem. We addressed this problem by evaluating a set of pattern classification algorithms on simulated and experimental block-design fMRI data. The set of classifiers consisted of linear and quadratic discriminants, linear support vector machine, and linear and nonlinear Gaussian naive Bayes classifiers. For linear discriminant, we used two methods of regularization: principal component analysis, and ridge regularization. The classifiers were used (1) to classify the volumes according to the behavioral task that was performed by the subject, and (2) to construct spatial maps that indicated the relative contribution of each voxel to classification. Our evaluation metrics were: (1) accuracy of out-of-sample classification and (2) reproducibility of spatial maps. In simulated data sets, we performed an additional evaluation of spatial maps with ROC analysis. We varied the magnitude, temporal variance and connectivity of simulated fMRI signal and identified the optimal classifier for each simulated environment. Overall, the best performers were linear and quadratic discriminants (operating on principal components of the data matrix) and, in some rare situations, a nonlinear Gaussian naïve Bayes classifier. The results from the simulated data were supported by within-subject analysis of experimental fMRI data, collected in a study of aging. This is the first study that systematically characterizes interactions between analysis model and signal parameters (such as magnitude, variance and correlation) on the performance of pattern classifiers for fMRI.
Evaluation of spatio-temporal decomposition techniques for group analysis of fMRI resting state data sets
2014, NeuroImage
The existing functional connectivity assessment techniques rely on different mathematical and neuro-physiological models. They may consequently provide different sets of spatial connectivity maps and associated temporal responses within their significant spatiotemporal sets of components. Note that the word component is used to generically refer to spatio-temporal pairs of maps and associated time courses. Such differences may confound the application of functional connectivity measurements in neuroscientific and clinical applications. Using several performance metrics we evaluated six fMRI resting-state connectivity measurement techniques including three fully exploratory techniques: 1) Melodic-Independent Component Analysis (ICA), 2) agnostic Canonical Variates Analysis (aCVA), and 3) generalized Canonical Correlation Analysis (gCCA); and three seed-based techniques: 1) seed gCCA (sgCCA) and 2, 3) seed Partial Least Squares (sPLS) with a posterior cingulate seed and two different time-series normalizations. We separately assessed the temporal and spatial domains for: 1) technique stability as a function of sample size using RV coefficients, and 2) subspace component similarity between pairs of techniques using CCA. Overall gCCA was the only technique that displayed high temporal and spatial stabilities, together with high spatial and temporal subspace similarities with multiple other techniques. ICA, aCVA and sgCCA tended to be the most stable spatially and produced similar spatial subspaces. All techniques produced relatively unstable and dissimilar temporal subspaces, except sPLS that produced relatively high temporal and lower spatial subspace stabilities, but with unique power-spectral Hurst coefficients ≪ 1. Our results indicate that spatial maps from resting state data sets are much less dependent on the analysis technique used than are the associated time series. Such temporal variability is coupled with individual spatial component maps, which may be quite dissimilar across techniques even with similar spatial subspaces. Therefore, we suggest that consensus estimation approaches, i.e. a 2nd-level gCCA, would have great utility to produce and aid interpretation of stable results from BOLD fMRI resting state data analysis.
Neuroanatomical correlates of Klinefelter syndrome studied in relation to the neuropsychological profile
2014, NeuroImage: Clinical
Brain imaging in Klinefelter syndrome (47, XXY) (KS), a genetic disorder characterized by the presence of an extra X chromosome, may contribute to understanding the relationship between gene expression, brain structure, and subsequent cognitive disabilities and psychiatric disorders.
We conducted the largest to date voxel-based morphometry study of 65 KS subjects and 65 controls matched for age and education and correlated these data to neuropsychological test scores. The KS patients had significantly smaller total brain volume (TBV), total gray matter volume (GMV) and total white matter volume (WMV) compared to controls, whereas no volumetric difference in cerebral spinal fluid (CSF) was found. There were no differences in TBV, GMV, WMV or CSF between testosterone treated KS (T-KS) and untreated KS (U-KS) patients. Compared to controls, KS patients had significantly decreased GMV bilaterally in insula, putamen, caudate, hippocampus, amygdala, temporal pole and frontal inferior orbita. Additionally, the right parahippocampal region and cerebellar volumes were reduced in KS patients. KS patients had significantly larger volumes in right postcentral gyrus, precuneus and parietal regions. Multivariate classification analysis discriminated KS patients from controls with 96.9% (p < 0.001) accuracy. Regression analyses, however, revealed no significant association between GMV differences and cognitive and psychological factors within the KS patients and controls or the groups combined. These results show that although gene dosage effect of having and extra X-chromosome may lead to large scale alterations of brain morphometry and extended cognitive disabilities no simple correspondence links these measures.
Predicting intrinsic brain activity
2013, NeuroImage
Multivariate supervised learning methods exhibit a remarkable ability to decode externally driven sensory, behavioral, and cognitive states from functional neuroimaging data. Although they are typically applied to task-based analyses, supervised learning methods are equally applicable to intrinsic effective and functional connectivity analyses. The obtained models of connectivity incorporate the multivariate interactions between all brain regions simultaneously, which will result in a more accurate representation of the connectome than the ones available with standard bivariate methods. Additionally the models can be applied to decode or predict the time series of intrinsic brain activity of a region from an independent dataset. The obtained prediction accuracy provides a measure of the integration between a brain region and other regions in its network, as well as a method for evaluating acquisition and preprocessing pipelines for resting state fMRI data. This article describes a method for learning multivariate models of connectivity. The method is applied in the non-parametric prediction accuracy, influence, and reproducibility–resampling (NPAIRS) framework, to study the regional variation of prediction accuracy and reproducibility (Strother et al., 2002). The resulting spatial distribution of these metrics is consistent with the functional hierarchy proposed by Mesulam (1998). Additionally we illustrate the utility of the multivariate regression connectivity modeling method for optimizing experimental parameters and assessing the quality of functional neuroimaging data.

View all citing articles on Scopus

¹: To whom correspondence should be addressed. E-mail: [email protected].

View full text

Regular ArticleThe Quantitative Evaluation of Functional Neuroimaging Experiments: Mutual Information Learning Curves

Abstract

NeuroImage

NeuroImage

NeuroImage

NeuroImage

NeuroImage

NeuroImage

NeuroImage

NeuroImage

Bayesian pca

Neural Networks for Pattern Recognition

A tutorial on support vector machines for pattern recognition

Data Mining Knowl. Discovery

Learning from Data: Concepts, Theory and Methods

Elements of Information Theory

Pattern Classification

Spatial registration and normalization of images

Hum. Brain Mapping

Statistical parametric maps in functional neuroimaging: A general linear approach

Hum. Brain Mapping

A multivariate analysis of pet activation studies

Hum. Brain Mapping

Regular Article
The Quantitative Evaluation of Functional Neuroimaging Experiments: Mutual Information Learning Curves