Abstract
Brain functional connectivity under the naturalistic paradigm has been demonstrated to be better at predicting individual behaviors than other brain states, such as rest and task. Nevertheless, the state-of-the-art methods are difficult to achieve desirable results from movie-watching paradigm fMRI(mfMRI) induced brain functional connectivity, especially when the datasets are small, because it is difficult to quantify how much useful dynamic information can be extracted from a single mfMRI modality to describe the state of the brain. Eye tracking, becoming popular due to its portability and less expense, can provide abundant behavioral features related to the output of human’s cognition, and thus might supplement the mfMRI in observing subjects’ subconscious behaviors. However, there are very few works on how to effectively integrate the multimodal information to strengthen the performance by unified framework. To this end, an effective fusion approach with mfMRI and eye tracking, based on Convolution with Edge-Node Switching in Graph Neural Networks (CensNet), is proposed in this article, with subjects taken as nodes, mfMRI derived functional connectivity as node feature, different eye tracking features used to compute similarity between subjects to construct heterogeneous graph edges. By taking multiple graphs as different channels, we introduce squeeze-and-excitation attention module to CensNet (A-CensNet) to integrate graph embeddings from multiple channels into one. The experiments demonstrate the proposed model outperforms the one using single modality, single channel and state-of-the-art methods. The results suggest that brain functional activities and eye behaviors might complement each other in interpreting trait-like phenotypes. Our code will make public later.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Thiebaut de Schotten, M., Forkel, S.J.: The emergent properties of the connected brain. Science, 378(6619), 505–510 (2022)
Li, J., et al.: Global signal regression strengthens association between resting-state functional connectivity and behavior. Neuroimage 196, 126–141 (2019)
Huijbers, W., Van Dijk, K.R.A., Boenniger, M.M., Stirnberg, R., Breteler, M.M.: Less head motion during MRI under task than resting-state conditions. Neuroimage 147, 111–120 (2017)
Barch, D.M., et al.: Function in the human connectome: task-fMRI and individual differences in behavior. Neuroimage 80, 169–189 (2013)
Sonkusare, S., Breakspear, M., Guo, C.: Naturalistic stimuli in neuroscience: critically acclaimed. Trends Cogn. Sci. 23(8), 699–714 (2019)
Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., Malach, R.: Intersubject synchronization of cortical activity during natural vision. Science 303(5664), 1634–1640 (2004)
Finn, E.S., Scheinost, D., Finn, D.M., Shen, X., Papademetris, X., Constable, R.T.: Can brain state be manipulated to emphasize individual differences in functional con-nectivity? Neuroimage 160, 140–151 (2017)
Finn, E.S., Bandettini, P.A.: Movie-watching outperforms rest for functional connectivity-based prediction of behavior. Neuroimage 235, 117963 (2021)
He, T., et al.: Deep neural networks and kernel regression achieve comparable accuracies for functional connectivity prediction of behavior and de-mographics. Neuroimage 206, 116276 (2020)
Gal, S., Coldham, Y., Bernstein-Eliav, M.: Act natural: functional connectivity from naturalistic stimuli fMRI outperforms resting-state in predicting brain activity. bioRxiv (2021)
He, T., et al.: Meta-matching as a simple framework to translate phenotypic predictive models from big to small data. Nature Neurosci. 25, 1–10 (2022)
Lim, J.Z., Mountstephens, J., Teo, J.: Emotion recognition using eye-tracking: taxonomy, review and current challenges. Sensors 20(8), 2384 (2020)
Hess, E.H., Polt, J.M.: Pupil size as related to interest value of visual stimuli. Science 132(3423), 349–350 (1960)
Lohse, G.L., Johnson, E.J.: A comparison of two process tracing methods for choice tasks. Organ. Behav. Hum. Decis. Process. 68(1), 28–43 (1996)
Son, J., et al.: Evaluating fMRI-based estimation of eye gaze during naturalistic viewing. Cereb. Cortex 30(3), 1171–1184 (2020)
Gao, J., et al.: Prediction of cognitive scores by movie-watching FMRI connectivity and eye movement via spectral graph convolutions. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1–5. IEEE (2022)
Jiang, X., Ji, P., Li, S.: CensNet: convolution with edge-node switching in graph neural networks. In: IJCAI, pp. 2656–2662 (2019)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)
Elam, J.: https://www.humanconnectome.org/study/hcp-young-adult/article/first-release-of-7t-mr-image-data. Accessed 20 June 2016
Griffanti, L., et al.: ICA-based artefact removal and accelerated fMRI ac-quisition for improved resting state network imaging. Neuroimage 95, 232–247 (2014)
Glasser, M.F., et al., Wu-Minn HCP Consortium: The minimal preprocessing pipelines for the human connectome project. Neuroimage, 80, 105–124 (2013)
Destrieux, C., Fischl, B., Dale, A., Halgren, E.: Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature. Neuroimage 53(1), 1–15 (2010)
Kawahara, J., et al.: BrainNetCNN: Convolutional neural networks for brain networks; towards predicting neurodevelopment. Neuroimage 146, 1038–1049 (2017)
Tye, C., et al.: Neurophysiological responses to faces and gaze direction differentiate children with ASD, ADHD and ASD + ADHD. Dev. Cogn. Neurosci. 5, 71–85 (2013)
Acknowledgement
This work was supported by the National Natural Science Foundation of China (31971288, U1801265, 61936007, 62276050, 61976045, U20B2065, U1801265 and 61936007); the National Key R&D Program of China under Grant 2020AAA0105701; High-level researcher start-up projects (Grant No. 06100-22GH0202178); Innovation Foundation for Doctor Dissertation of Northwestern Polytechnical University CX2022052.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gao, J. et al. (2023). Prediction of Cognitive Scores by Joint Use of Movie-Watching fMRI Connectivity and Eye Tracking via Attention-CensNet. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14221. Springer, Cham. https://doi.org/10.1007/978-3-031-43895-0_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-43895-0_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43894-3
Online ISBN: 978-3-031-43895-0
eBook Packages: Computer ScienceComputer Science (R0)