Abstract
During task performance, brain states change dynamically and can appear recurrently. Recently, recurrent neural networks (RNN) have been used for identifying functional signatures underlying such brain states from task functional Magnetic Resonance Imaging (fMRI) data. While RNNs only model temporal dependence between time points, brain task decoding needs to model temporal dependencies of the underlying brain states. Furthermore, as only a subset of brain regions are involved in task performance, it is important to consider subsets of brain regions for brain decoding. To address these issues, we present a customised neural network architecture, Salient Patterns Over Time and Space (SPOTS), which not only captures dependencies of brain states at different time points but also pays attention to key brain regions associated with the task. On language and motor task data gathered in the Human Connectome Project, SPOTS improves brain state prediction by 17% to 40% as compared to the baseline RNN model. By spotting salient spatio-temporal patterns, SPOTS is able to infer brain states even on small time windows of fMRI data, which the present state-of-the-art methods struggle with. This allows for quick identification of abnormal task-fMRI scans, leading to possible future applications in task-fMRI data quality assurance and disease detection. Code is available at https://github.com/SCSE-Biomedical-Computing-Group/SPOTS.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
A sample refers to input data of length T. However, the model makes a prediction at every time point, thus accuracy is computed based on single time points.
References
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: ICLR (2015)
Barch, D.M., Burgess, G.C., Harms, M.P., Petersen, S.E., Schlaggar, B.L., Corbetta, M., Glasser, M.F., Curtiss, S., Dixit, S., Feldt, C., et al.: Function in the human connectome: task-fMRI and individual differences in behavior. Neuroimage 80, 169–189 (2013)
Binder, J.R., et al.: Mapping anterior temporal lobe language areas with fMRI: a multi-center normative study. NeuroImage 54(2), 1465 (2011)
Buckner, R.L., Krienen, F.M., Castellanos, A., Diaz, J.C., Yeo, B.T.: The organization of the human cerebellum estimated by intrinsic functional connectivity. J. Neurophysiol. 106(5), 2322–2345 (2011). https://doi.org/10.1152/jn.00339.2011
Chollet, F.: Deep Learning with Python and Keras: The Practical Guide from the Developer of the Keras Library. MITP-Verlags GmbH & Co. KG, Bonn (2018)
Glasser, M.F., et al.: The human connectome project’s neuroimaging approach. Nature Neurosci. 19(9), 1175 (2016)
Glasser, M.F., et al.: The minimal preprocessing pipelines for the human connectome project. Neuroimage 80, 105–124 (2013)
Goodfellow, I.J., Mirza, M., Xiao, D., Courville, A., Bengio, Y.: An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 (2013)
Gupta, S., Chan, Y.H., Rajapakse, J.C.: Decoding brain functional connectivity implicated in AD and MCI. In: Shen, D., et al. (eds.) MICCAI 2019. Lecture Notes in Computer Science, vol. 11766, pp. 781–789. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32248-9_87
Gupta, S., Chan, Y.H., Rajapakse, J.C.: Obtaining leaner deep neural networks for decoding brain functional connectome in a single shot. Neurocomput. (2020). (in Press)
Gupta, S., Rajapakse, J.C.: Iterative consensus spectral clustering improves detection of subject and group level brain functional modules. Sci. Rep. 10(1), 1–15 (2020)
Li, H., Fan, Y.: Brain decoding from functional MRI using long short-term memory recurrent neural networks. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. Lecture Notes in Computer Science, vol. 11072, pp. 320–328. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00931-1_37
Li, H., Fan, Y.: Interpretable, highly accurate brain decoding of subtly distinct brain states from functional MRI using intrinsic functional networks and long short-term memory recurrent neural networks. NeuroImage 202, 116059 (2019)
Liao, C.H., Worsley, K.J., Poline, J.B., Aston, J.A., Duncan, G.H., Evans, A.C.: Estimating the delay of the fMRI response. NeuroImage 16(3), 593–606 (2002)
Loula, J., Varoquaux, G., Thirion, B.: Decoding fMRI activity in the time domain improves classification performance. NeuroImage 180, 203–210 (2018)
Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems, pp. 4765–4774 (2017)
Power, J.D., et al.: Functional network organization of the human brain. Neuron 72(4), 665–678 (2011)
Serra, J., Suris, D., Miron, M., Karatzoglou, A.: Overcoming catastrophic forgetting with hard attention to the task. arXiv preprint arXiv:1801.01423 (2018)
Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. arXiv preprint arXiv:1704.02685 (2017)
Srivastava, N., Mansimov, E., Salakhutdinov, R.: Unsupervised learning of video representations using LSTMs. In: Proceedings of the 32nd International Conference on International Conference on Machine Learnings, vol. 37, pp. 843–852 (2015)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: NIPS (2014)
Acknowledgement
This work was partially supported by AcRF Tier 1 grant RG 116/19 of Ministry of Education, Singapore. Data were provided [in part] by the Human Connectome Project, WU-Minn Consortium (Principal Investigators: David Van Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Chan, Y.H., Gupta, S., Kasun, L.L.C., Rajapakse, J.C. (2020). Decoding Task States by Spotting Salient Patterns at Time Points and Brain Regions. In: Kia, S.M., et al. Machine Learning in Clinical Neuroimaging and Radiogenomics in Neuro-oncology. MLCN RNO-AI 2020 2020. Lecture Notes in Computer Science(), vol 12449. Springer, Cham. https://doi.org/10.1007/978-3-030-66843-3_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-66843-3_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-66842-6
Online ISBN: 978-3-030-66843-3
eBook Packages: Computer ScienceComputer Science (R0)
