Test–retest reliability of fMRI-based graph theoretical properties during working memory, emotion processing, and resting state
Introduction
The examination of the human brain functional connectome with functional magnetic resonance imaging (fMRI) and graph theory methods is an attractive research strategy in contemporary neuroscience. Here, whole-brain topological properties are calculated from pairwise connectivity matrices derived from the temporal correlations of blood-oxygen level dependent (BOLD) time series of spatially distant neural regions (Bullmore and Bassett, 2011, Bullmore and Sporns, 2009, Wig et al., 2011). By characterizing the connectivity structure in its entirety, this approach provides valuable insights into the configuration and efficiency of brain functional networks that are unattainable by conventional activation and connectivity analyses. Prior graph theory work suggests that the functional architecture of the human brain is modulated by age (Achard and Bullmore, 2007, Meunier et al., 2009, Wang et al., 2010), sex (Tian et al., 2011), intelligence (van den Heuvel et al., 2009), genetic predisposition (Fornito et al., 2011b), and brain disorders (Liu et al., 2008, Lynall et al., 2010, Sanz-Arigita et al., 2010). These data indicate the value, practicality, and broad applicability of this method in neuroimaging research, including potential future applications in drug development studies.
Many prior studies used fMRI and graph theory methods to study the brain functional connectome at rest, one of many possible neural functional states in humans. Less attention has been directed to the examination of brain topologies constructed from fMRI data collected during active task performance, which is important because certain connectivity features during a task may not be evident at rest (Bilek et al., 2013, Pezawas et al., 2005). Accordingly, a rising interest in this topic is evident in the recent literature. For instance, brain functional topologies have been examined during working memory (Ginestet and Simmons, 2011, He et al., 2012, Wang et al., 2010), visual stimulation (Moussa et al., 2011), motor learning (Bassett et al., 2011b, Heitger et al., 2012), auditory stimulation (Guo et al., 2011, Ma et al., 2012) and emotion processing (Kinnison et al., 2012). However, the combination of active fMRI tasks and graph-based analyses in neuropsychiatric research has just started to gain impetus (Fornito et al., 2011a, He et al., 2012, Heitger et al., 2012), and very few studies have used the potential of this approach to gather imaging biomarkers for therapy research (Giessing and Thiel, 2012).
At least three empirical questions remain to be addressed before the utility of graph theory methods for the analysis of active fMRI tasks can be fully appreciated. The first question relates to the test–retest reliability of topological estimates, which inevitably constrains the validity of within-subject designs such as those routinely used in drug development. Although prior studies indicated a fair to good robustness of resting-state brain graph measures (Braun et al., 2012, Deuker et al., 2009, Jin et al., 2011, Wang et al., 2011), little is known about the test–retest reliability of graph theoretical properties derived from active fMRI tasks. In our prior study, we showed relevant differences in the reliability of activation estimates derived from paradigms targeting cognitive, emotional and motivational processes (Plichta et al., 2012). Similar differences in robustness may apply for brain graphs constructed from active fMRI tasks that challenge different brain functional domains. To date, only one magnetoencephalography (MEG) study has compared the reliabilities of graph measurements derived from an active working memory task with those from a resting state experiment (Deuker et al., 2009). While this study suggested higher reliabilities during active task performance relative to rest, it remains unclear whether this translates to fMRI, and if so, whether there are differences in the reliabilities of graph theoretical properties derived from active tasks challenging different higher order functions.
The second question relates to the adequate preprocessing method for the fMRI time series of active tasks prior to graph construction. A fundamental step in graph analyses is the computation of temporal correlations (or other measures of shared variance) between pre-defined neural nodes. However, compared to resting state, fMRI time series from active tasks are impacted by additional sources of signal variability linked to the temporal structure of the experiment. These task-dependent signal fluctuations are, in a sense, the point of the activation experiment, but may complicate the interpretation of the resulting functional connectivity estimates (Gavrilescu et al., 2008, Horwitz, 2003, Jones et al., 2010). Specifically, it has been argued that in blocked fMRI designs, the systematic “ramping up and down” of BOLD signals in response to the on- and offsets of block conditions may lead to an artificial inflation of the derived connectivity measures (Gavrilescu et al., 2008, Jones et al., 2010). Consequently, the removal of block-to-block signal fluctuations has been proposed as a basic correction step (Gavrilescu et al., 2008, Meyer-Lindenberg, 2009, Muller et al., 2011). Accordingly, two procedures have been widely employed: linear regression of block-dependent activations or deactivations (Esslinger et al., 2009, Esslinger et al., 2011, Villalobos et al., 2005), and the splitting and concatenating of time segments that correspond to a given condition of interest (Honey et al., 2002, Just et al., 2004, Mostofsky et al., 2009). Both methods result in a substantial modification of the original time series and may affect the connectivity matrix of interacting neural nodes, and thus the properties of the derived brain graphs. To date, no final agreement has been reached regarding the optimal processing strategy for active fMRI tasks, and little is known about the effects of these procedures on the reliability of the outcome properties in graph theoretical analyses (Gavrilescu et al., 2008, Jones et al., 2010).
The third question relates to the choice of brain atlas used for node definition. While the impact of different parcellation strategies on the topological organization of brain networks is widely acknowledged (Fornito et al., 2010, Power et al., 2011, Wang et al., 2009, Zalesky et al., 2010), the question of which parcellation scheme to use for structural and/or functional neuroimaging data is an open one. Specifically, the effect of different node definitions on the test–retest reliability of graph theoretic investigations of active fMRI experiments is unknown.
Here, we aim to address these questions by studying the test–retest reliability of brain graphs calculated from three well-established fMRI experiments: an n-back working memory task, an emotional face-matching task, and a resting-state experiment. Consistent with published procedures, we calculated intra-class correlation coefficients (ICC) as index of interest (Bennett and Miller, 2010, Shrout and Fleiss, 1979) and compared the reliabilities of the derived network estimates between experiments. To answer the second question, we used five different strategies for the processing of the task-related time series (e.g., use of the original time series, removal of task effects by regression, removal of task effects by splitting and concatenation of epochs) and compared the reliabilities of the derived brain graphs between these procedures. Moreover, we calculated and compared the reliabilities of network estimates derived from two established methods for node definition: a widely-used automated anatomical labeling (AAL) template (Tzourio-Mazoyer et al., 2002) and a larger set of putative functional nodes proposed by a prior study combining meta-analytic data from active fMRI experiments and functional connectivity mapping (Power et al., 2011).
Section snippets
Subjects
Twenty-six healthy volunteers (mean age: 24.4 ± 2.8 years, 15 females, 2 smokers) participated in this study. All participants were recruited from communities in and around the city of Mannheim, Germany, and provided written informed consent for a protocol approved by the institutional review board of the University of Heidelberg. Exclusion criteria included a lifetime history of significant general medical, psychiatric, or neurological illness, prior psychotropic pharmacological treatment, head
Results
Details on the acquired physiological and behavioral measures are presented in Supplementary Table S3. No significant between-session differences in the participants' sleeping hours, cigarette consumption or coffee intake were observed (all P's > 0.28). Analyses of the performance data of the active fMRI tasks demonstrated stable results across sessions (all P's > 0.21). All calculated graphs displayed small-world network properties (σ = γ/λ > 1, range 1.16–2.67), as indicated by the presence of large
Discussion
In this study, we aimed to quantify the test–retest reliability of network properties derived from graph theory methods to inform and optimize the choice of data processing strategies, parcellation schemes and outcome diagnostics for use in fMRI research with active tasks and within-subject designs, in particular future pharmacological challenge studies. Overall, our results support three main conclusions. First, for the active fMRI experiments, the choice of processing strategy impacts the
Recommendations for future studies
From our data, several suggestions can be derived for future fMRI studies using graph theory methods and within-subject designs. First, if the main focus of the research is not on a specific cognitive domain, resting-state experiments are preferable over active tasks. Second, most global connectivity and global network properties are suitable outcome diagnostics in resting-state experiments. The same notion applies for some nodal diagnostics such as connectivity strength and node degree,
Limitations
The results reported in this work and the conclusions regarding best practices are dependent upon several methodological choices made early in the processing stream. Firstly, the statistical measure of association is the topic of some disagreements in recent work (David et al., 2004, Dawson et al., 2013, Gates and Molenaar, 2012, Smith et al., 2011). Since some prior studies suggest that the temporal correlation of time series derived from neuroimaging data is a sensitive measure (Smith et al.,
Acknowledgments
H.C. is a Ph.D. scholarship awardee of the Chinese Scholarship Council. A.M.-L. gratefully acknowledges the grant support by the NEWMEDS Innovative Medicines Initiative Joint Undertaking (IMI) under grant agreement No. 115008. H.T. gratefully acknowledges grant support by the German Federal Ministry of Education and Research (BMBF 01GQ1102). We thank Dagmar Gass, Daniela Mier and Carina Saur for the research assistance and Urs Braun for the valuable comments on our data.
Conflict of interest
The
References (99)
- et al.
Conserved and variable architecture of human white matter connectivity
Neuroimage
(2011) - et al.
Altered resting state complexity in schizophrenia
Neuroimage
(2012) - et al.
Test–retest reliability of resting-state connectivity network characteristics using fMRI and graph theoretical measures
Neuroimage
(2012) - et al.
Response and habituation of the human amygdala during visual processing of facial expression
Neuron
(1996) - et al.
Identifying the brain's most globally connected regions
Neuroimage
(2010) - et al.
Evaluation of different measures of functional connectivity using a neural mass model
Neuroimage
(2004) - et al.
Evaluation and calibration of functional network modeling methods based on known anatomical connections
Neuroimage
(2013) - et al.
When to use agreement versus reliability measures
J. Clin. Epidemiol.
(2006) - et al.
Reproducibility of graph metrics of human brain functional networks
Neuroimage
(2009) - et al.
Cognitive state and connectivity effects of the genome-wide significant psychosis variant in ZNF804A
Neuroimage
(2011)
Brain habituation during repeated exposure to fearful and neutral faces: a functional MRI study
Brain Res. Bull.
General and specific functional connectivity disturbances in first-episode schizophrenia during cognitive control performance
Biol. Psychiatry
Group search algorithm recovers effective connectivity maps for individuals in homogeneous and heterogeneous samples
Neuroimage
Statistical parametric network analysis of functional connectivity dynamics during a working memory task
Neuroimage
Abnormal neural activities in first-episode, treatment-naive, short-illness-duration, and treatment-response patients with major depressive disorder: a resting-state fMRI study
J. Affect. Disord.
Motor learning-induced changes in functional brain connectivity as revealed by means of graph-theoretical network analysis
Neuroimage
Selective enhancement of functional connectivity in the left prefrontal cortex during sentence processing
Neuroimage
Effects of verbal working memory load on corticocortical connectivity modeled by path analysis of functional magnetic resonance imaging data
Neuroimage
The elusive concept of brain connectivity
Neuroimage
Stability of amygdala BOLD response to fearful faces over multiple scan sessions
Neuroimage
Sources of group differences in functional connectivity: an investigation applied to autism spectrum disorder
Neuroimage
Connectivity pattern changes in default-mode network with deep non-REM and REM sleep
Neurosci. Res.
Test–retest study of fMRI signal change evoked by electroacupuncture stimulation
Neuroimage
Modulations of functional connectivity in the healthy and schizophrenia groups during task and rest
Neuroimage
Age-related changes in modular organization of human brain functional networks
Neuroimage
Cortical functional connectivity decodes subconscious, task-irrelevant threat-related emotion processing
Neuroimage
Test–retest reliability of evoked BOLD signals from a cognitive–emotive fMRI test battery
Neuroimage
Functional network organization of the human brain
Neuron
Complex network measures of brain connectivity: uses and interpretations
Neuroimage
Weight-conserving characterization of complex functional brain networks
Neuroimage
Negative edges and soft thresholding in complex network analysis of resting state functional connectivity data
Neuroimage
Network modelling methods for FMRI
Neuroimage
Hemisphere- and gender-related differences in small-world brain networks: a resting-state functional MRI study
Neuroimage
Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain
Neuroimage
Reduced functional connectivity between V1 and inferior frontal cortex associated with visuomotor performance in autism
Neuroimage
Age-related changes in topological patterns of large-scale brain functional networks during memory encoding and recognition
Neuroimage
A comprehensive assessment of regional variation in the impact of head micromovements on functional connectomics
Neuroimage
Whole-brain anatomical networks: does the choice of nodes matter?
Neuroimage
On the use of correlation as a measure of network connectivity
Neuroimage
The oscillating brain: complex and reliable
Neuroimage
Efficiency and cost of economical brain functional networks
PLoS Comput. Biol.
A resilient, low-frequency, small-world human brain functional network with highly connected association cortical hubs
J. Neurosci.
Small-world brain networks
Neuroscientist
Cognitive fitness of cost-efficient brain functional networks
Proc. Natl. Acad. Sci. U. S. A.
Dynamic reconfiguration of human brain networks during learning
Proc. Natl. Acad. Sci. U. S. A.
Robust detection of dynamic community structure in networks
Chaos
How reliable are the results from functional magnetic resonance imaging?
Ann. N. Y. Acad. Sci.
Application of high-frequency repetitive transcranial magnetic stimulation to the DLPFC alters human prefrontal–hippocampal functional interaction
J. Neurosci.
Cortical hubs revealed by intrinsic functional connectivity: mapping, assessment of stability, and relation to Alzheimer's disease
J. Neurosci.
Cited by (183)
Childhood trauma is linked to abnormal static-dynamic brain topology in adolescents with major depressive disorder
2023, International Journal of Clinical and Health PsychologyCognitive control and dishonesty
2022, Trends in Cognitive Sciences