Elsevier

NeuroImage

Volume 84, 1 January 2014, Pages 888-900
NeuroImage

Test–retest reliability of fMRI-based graph theoretical properties during working memory, emotion processing, and resting state

https://doi.org/10.1016/j.neuroimage.2013.09.013Get rights and content

Highlights

  • We quantified the test-retest reliabilities of graph metrics during active tasks.

  • We measured the impact of processing methods and atlases on graph reliabilities.

  • Choice of processing methods for active tasks impacts robustness of graph metrics.

  • Functional brain atlas presented higher reliability than structural atlas.

  • Reliability of graph estimates varies between cognitive states and properties.

Abstract

The investigation of the brain connectome with functional magnetic resonance imaging (fMRI) and graph theory analyses has recently gained much popularity, but little is known about the robustness of these properties, in particular those derived from active fMRI tasks. Here, we studied the test–retest reliability of brain graphs calculated from 26 healthy participants with three established fMRI experiments (n-back working memory, emotional face-matching, resting state) and two parcellation schemes for node definition (AAL atlas, functional atlas proposed by Power et al.). We compared the intra-class correlation coefficients (ICCs) of five different data processing strategies and demonstrated a superior reliability of task-regression methods with condition-specific regressors. The between-task comparison revealed significantly higher ICCs for resting state relative to the active tasks, and a superiority of the n-back task relative to the face-matching task for global and local network properties. While the mean ICCs were typically lower for the active tasks, overall fair to good reliabilities were detected for global and local connectivity properties, and for the n-back task with both atlases, smallworldness. For all three tasks and atlases, low mean ICCs were seen for the local network properties. However, node-specific good reliabilities were detected for node degree in regions known to be critical for the challenged functions (resting-state: default-mode network nodes, n-back: fronto-parietal nodes, face-matching: limbic nodes). Between-atlas comparison demonstrated significantly higher reliabilities for the functional parcellations for global and local network properties. Our findings can inform the choice of processing strategies, brain atlases and outcome properties for fMRI studies using active tasks, graph theory methods, and within-subject designs, in particular future pharmaco-fMRI studies.

Introduction

The examination of the human brain functional connectome with functional magnetic resonance imaging (fMRI) and graph theory methods is an attractive research strategy in contemporary neuroscience. Here, whole-brain topological properties are calculated from pairwise connectivity matrices derived from the temporal correlations of blood-oxygen level dependent (BOLD) time series of spatially distant neural regions (Bullmore and Bassett, 2011, Bullmore and Sporns, 2009, Wig et al., 2011). By characterizing the connectivity structure in its entirety, this approach provides valuable insights into the configuration and efficiency of brain functional networks that are unattainable by conventional activation and connectivity analyses. Prior graph theory work suggests that the functional architecture of the human brain is modulated by age (Achard and Bullmore, 2007, Meunier et al., 2009, Wang et al., 2010), sex (Tian et al., 2011), intelligence (van den Heuvel et al., 2009), genetic predisposition (Fornito et al., 2011b), and brain disorders (Liu et al., 2008, Lynall et al., 2010, Sanz-Arigita et al., 2010). These data indicate the value, practicality, and broad applicability of this method in neuroimaging research, including potential future applications in drug development studies.

Many prior studies used fMRI and graph theory methods to study the brain functional connectome at rest, one of many possible neural functional states in humans. Less attention has been directed to the examination of brain topologies constructed from fMRI data collected during active task performance, which is important because certain connectivity features during a task may not be evident at rest (Bilek et al., 2013, Pezawas et al., 2005). Accordingly, a rising interest in this topic is evident in the recent literature. For instance, brain functional topologies have been examined during working memory (Ginestet and Simmons, 2011, He et al., 2012, Wang et al., 2010), visual stimulation (Moussa et al., 2011), motor learning (Bassett et al., 2011b, Heitger et al., 2012), auditory stimulation (Guo et al., 2011, Ma et al., 2012) and emotion processing (Kinnison et al., 2012). However, the combination of active fMRI tasks and graph-based analyses in neuropsychiatric research has just started to gain impetus (Fornito et al., 2011a, He et al., 2012, Heitger et al., 2012), and very few studies have used the potential of this approach to gather imaging biomarkers for therapy research (Giessing and Thiel, 2012).

At least three empirical questions remain to be addressed before the utility of graph theory methods for the analysis of active fMRI tasks can be fully appreciated. The first question relates to the test–retest reliability of topological estimates, which inevitably constrains the validity of within-subject designs such as those routinely used in drug development. Although prior studies indicated a fair to good robustness of resting-state brain graph measures (Braun et al., 2012, Deuker et al., 2009, Jin et al., 2011, Wang et al., 2011), little is known about the test–retest reliability of graph theoretical properties derived from active fMRI tasks. In our prior study, we showed relevant differences in the reliability of activation estimates derived from paradigms targeting cognitive, emotional and motivational processes (Plichta et al., 2012). Similar differences in robustness may apply for brain graphs constructed from active fMRI tasks that challenge different brain functional domains. To date, only one magnetoencephalography (MEG) study has compared the reliabilities of graph measurements derived from an active working memory task with those from a resting state experiment (Deuker et al., 2009). While this study suggested higher reliabilities during active task performance relative to rest, it remains unclear whether this translates to fMRI, and if so, whether there are differences in the reliabilities of graph theoretical properties derived from active tasks challenging different higher order functions.

The second question relates to the adequate preprocessing method for the fMRI time series of active tasks prior to graph construction. A fundamental step in graph analyses is the computation of temporal correlations (or other measures of shared variance) between pre-defined neural nodes. However, compared to resting state, fMRI time series from active tasks are impacted by additional sources of signal variability linked to the temporal structure of the experiment. These task-dependent signal fluctuations are, in a sense, the point of the activation experiment, but may complicate the interpretation of the resulting functional connectivity estimates (Gavrilescu et al., 2008, Horwitz, 2003, Jones et al., 2010). Specifically, it has been argued that in blocked fMRI designs, the systematic “ramping up and down” of BOLD signals in response to the on- and offsets of block conditions may lead to an artificial inflation of the derived connectivity measures (Gavrilescu et al., 2008, Jones et al., 2010). Consequently, the removal of block-to-block signal fluctuations has been proposed as a basic correction step (Gavrilescu et al., 2008, Meyer-Lindenberg, 2009, Muller et al., 2011). Accordingly, two procedures have been widely employed: linear regression of block-dependent activations or deactivations (Esslinger et al., 2009, Esslinger et al., 2011, Villalobos et al., 2005), and the splitting and concatenating of time segments that correspond to a given condition of interest (Honey et al., 2002, Just et al., 2004, Mostofsky et al., 2009). Both methods result in a substantial modification of the original time series and may affect the connectivity matrix of interacting neural nodes, and thus the properties of the derived brain graphs. To date, no final agreement has been reached regarding the optimal processing strategy for active fMRI tasks, and little is known about the effects of these procedures on the reliability of the outcome properties in graph theoretical analyses (Gavrilescu et al., 2008, Jones et al., 2010).

The third question relates to the choice of brain atlas used for node definition. While the impact of different parcellation strategies on the topological organization of brain networks is widely acknowledged (Fornito et al., 2010, Power et al., 2011, Wang et al., 2009, Zalesky et al., 2010), the question of which parcellation scheme to use for structural and/or functional neuroimaging data is an open one. Specifically, the effect of different node definitions on the test–retest reliability of graph theoretic investigations of active fMRI experiments is unknown.

Here, we aim to address these questions by studying the test–retest reliability of brain graphs calculated from three well-established fMRI experiments: an n-back working memory task, an emotional face-matching task, and a resting-state experiment. Consistent with published procedures, we calculated intra-class correlation coefficients (ICC) as index of interest (Bennett and Miller, 2010, Shrout and Fleiss, 1979) and compared the reliabilities of the derived network estimates between experiments. To answer the second question, we used five different strategies for the processing of the task-related time series (e.g., use of the original time series, removal of task effects by regression, removal of task effects by splitting and concatenation of epochs) and compared the reliabilities of the derived brain graphs between these procedures. Moreover, we calculated and compared the reliabilities of network estimates derived from two established methods for node definition: a widely-used automated anatomical labeling (AAL) template (Tzourio-Mazoyer et al., 2002) and a larger set of putative functional nodes proposed by a prior study combining meta-analytic data from active fMRI experiments and functional connectivity mapping (Power et al., 2011).

Section snippets

Subjects

Twenty-six healthy volunteers (mean age: 24.4 ± 2.8 years, 15 females, 2 smokers) participated in this study. All participants were recruited from communities in and around the city of Mannheim, Germany, and provided written informed consent for a protocol approved by the institutional review board of the University of Heidelberg. Exclusion criteria included a lifetime history of significant general medical, psychiatric, or neurological illness, prior psychotropic pharmacological treatment, head

Results

Details on the acquired physiological and behavioral measures are presented in Supplementary Table S3. No significant between-session differences in the participants' sleeping hours, cigarette consumption or coffee intake were observed (all P's > 0.28). Analyses of the performance data of the active fMRI tasks demonstrated stable results across sessions (all P's > 0.21). All calculated graphs displayed small-world network properties (σ = γ/λ > 1, range 1.16–2.67), as indicated by the presence of large

Discussion

In this study, we aimed to quantify the test–retest reliability of network properties derived from graph theory methods to inform and optimize the choice of data processing strategies, parcellation schemes and outcome diagnostics for use in fMRI research with active tasks and within-subject designs, in particular future pharmacological challenge studies. Overall, our results support three main conclusions. First, for the active fMRI experiments, the choice of processing strategy impacts the

Recommendations for future studies

From our data, several suggestions can be derived for future fMRI studies using graph theory methods and within-subject designs. First, if the main focus of the research is not on a specific cognitive domain, resting-state experiments are preferable over active tasks. Second, most global connectivity and global network properties are suitable outcome diagnostics in resting-state experiments. The same notion applies for some nodal diagnostics such as connectivity strength and node degree,

Limitations

The results reported in this work and the conclusions regarding best practices are dependent upon several methodological choices made early in the processing stream. Firstly, the statistical measure of association is the topic of some disagreements in recent work (David et al., 2004, Dawson et al., 2013, Gates and Molenaar, 2012, Smith et al., 2011). Since some prior studies suggest that the temporal correlation of time series derived from neuroimaging data is a sensitive measure (Smith et al.,

Acknowledgments

H.C. is a Ph.D. scholarship awardee of the Chinese Scholarship Council. A.M.-L. gratefully acknowledges the grant support by the NEWMEDS Innovative Medicines Initiative Joint Undertaking (IMI) under grant agreement No. 115008. H.T. gratefully acknowledges grant support by the German Federal Ministry of Education and Research (BMBF 01GQ1102). We thank Dagmar Gass, Daniela Mier and Carina Saur for the research assistance and Urs Braun for the valuable comments on our data.

Conflict of interest

The

References (99)

  • H. Fischer et al.

    Brain habituation during repeated exposure to fearful and neutral faces: a functional MRI study

    Brain Res. Bull.

    (2003)
  • A. Fornito et al.

    General and specific functional connectivity disturbances in first-episode schizophrenia during cognitive control performance

    Biol. Psychiatry

    (2011)
  • K.M. Gates et al.

    Group search algorithm recovers effective connectivity maps for individuals in homogeneous and heterogeneous samples

    Neuroimage

    (2012)
  • C.E. Ginestet et al.

    Statistical parametric network analysis of functional connectivity dynamics during a working memory task

    Neuroimage

    (2011)
  • W.B. Guo et al.

    Abnormal neural activities in first-episode, treatment-naive, short-illness-duration, and treatment-response patients with major depressive disorder: a resting-state fMRI study

    J. Affect. Disord.

    (2011)
  • M.H. Heitger et al.

    Motor learning-induced changes in functional brain connectivity as revealed by means of graph-theoretical network analysis

    Neuroimage

    (2012)
  • F. Homae et al.

    Selective enhancement of functional connectivity in the left prefrontal cortex during sentence processing

    Neuroimage

    (2003)
  • G.D. Honey et al.

    Effects of verbal working memory load on corticocortical connectivity modeled by path analysis of functional magnetic resonance imaging data

    Neuroimage

    (2002)
  • B. Horwitz

    The elusive concept of brain connectivity

    Neuroimage

    (2003)
  • T. Johnstone et al.

    Stability of amygdala BOLD response to fearful faces over multiple scan sessions

    Neuroimage

    (2005)
  • T.B. Jones et al.

    Sources of group differences in functional connectivity: an investigation applied to autism spectrum disorder

    Neuroimage

    (2010)
  • T. Koike et al.

    Connectivity pattern changes in default-mode network with deep non-REM and REM sleep

    Neurosci. Res.

    (2011)
  • J. Kong et al.

    Test–retest study of fMRI signal change evoked by electroacupuncture stimulation

    Neuroimage

    (2007)
  • S. Ma et al.

    Modulations of functional connectivity in the healthy and schizophrenia groups during task and rest

    Neuroimage

    (2012)
  • D. Meunier et al.

    Age-related changes in modular organization of human brain functional networks

    Neuroimage

    (2009)
  • S.P. Pantazatos et al.

    Cortical functional connectivity decodes subconscious, task-irrelevant threat-related emotion processing

    Neuroimage

    (2012)
  • M.M. Plichta et al.

    Test–retest reliability of evoked BOLD signals from a cognitive–emotive fMRI test battery

    Neuroimage

    (2012)
  • J.D. Power et al.

    Functional network organization of the human brain

    Neuron

    (2011)
  • M. Rubinov et al.

    Complex network measures of brain connectivity: uses and interpretations

    Neuroimage

    (2010)
  • M. Rubinov et al.

    Weight-conserving characterization of complex functional brain networks

    Neuroimage

    (2011)
  • A.J. Schwarz et al.

    Negative edges and soft thresholding in complex network analysis of resting state functional connectivity data

    Neuroimage

    (2011)
  • S.M. Smith et al.

    Network modelling methods for FMRI

    Neuroimage

    (2011)
  • L. Tian et al.

    Hemisphere- and gender-related differences in small-world brain networks: a resting-state functional MRI study

    Neuroimage

    (2011)
  • N. Tzourio-Mazoyer et al.

    Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain

    Neuroimage

    (2002)
  • M.E. Villalobos et al.

    Reduced functional connectivity between V1 and inferior frontal cortex associated with visuomotor performance in autism

    Neuroimage

    (2005)
  • L. Wang et al.

    Age-related changes in topological patterns of large-scale brain functional networks during memory encoding and recognition

    Neuroimage

    (2010)
  • C.G. Yan et al.

    A comprehensive assessment of regional variation in the impact of head micromovements on functional connectomics

    Neuroimage

    (2013)
  • A. Zalesky et al.

    Whole-brain anatomical networks: does the choice of nodes matter?

    Neuroimage

    (2010)
  • A. Zalesky et al.

    On the use of correlation as a measure of network connectivity

    Neuroimage

    (2012)
  • X.N. Zuo et al.

    The oscillating brain: complex and reliable

    Neuroimage

    (2010)
  • S. Achard et al.

    Efficiency and cost of economical brain functional networks

    PLoS Comput. Biol.

    (2007)
  • S. Achard et al.

    A resilient, low-frequency, small-world human brain functional network with highly connected association cortical hubs

    J. Neurosci.

    (2006)
  • D.S. Bassett et al.

    Small-world brain networks

    Neuroscientist

    (2006)
  • D.S. Bassett et al.

    Cognitive fitness of cost-efficient brain functional networks

    Proc. Natl. Acad. Sci. U. S. A.

    (2009)
  • D.S. Bassett et al.

    Dynamic reconfiguration of human brain networks during learning

    Proc. Natl. Acad. Sci. U. S. A.

    (2011)
  • D.S. Bassett et al.

    Robust detection of dynamic community structure in networks

    Chaos

    (2013)
  • C.M. Bennett et al.

    How reliable are the results from functional magnetic resonance imaging?

    Ann. N. Y. Acad. Sci.

    (2010)
  • E. Bilek et al.

    Application of high-frequency repetitive transcranial magnetic stimulation to the DLPFC alters human prefrontal–hippocampal functional interaction

    J. Neurosci.

    (2013)
  • R.L. Buckner et al.

    Cortical hubs revealed by intrinsic functional connectivity: mapping, assessment of stability, and relation to Alzheimer's disease

    J. Neurosci.

    (2009)
  • Cited by (183)

    • Cognitive control and dishonesty

      2022, Trends in Cognitive Sciences
    View all citing articles on Scopus
    View full text