Abstract
The transformation and transmission of brain stimuli reflect the dynamical brain activity in space and time. Compared with functional magnetic resonance imaging (fMRI), magneto- or electroencephalography (M/EEG) fast couples to the neural activity through generated magnetic fields. However, the MEG signal is inhomogeneous throughout the whole brain, which is affected by the signal-to-noise ratio, the sensors’ location and distance. Current non-invasive neuroimaging modalities such as fMRI and M/EEG excel high resolution in space or time but not in both. To solve the main limitations of current technique for brain activity recording, we propose a novel recurrent memory optimization approach to predict the internal behavioral states in space and time. The proposed method uses Optimal Polynomial Projections to capture the long temporal history with robust online compression. The training process takes the pairs of fMRI and MEG data as inputs and predicts the recurrent brain states through the Siamese network. In the testing process, the framework only uses fMRI data to generate the corresponding neural response in space and time. The experimental results with Human connectome project (HCP) show that the predicted signal could reflect the neural activity with high spatial resolution as fMRI and high temporal resolution as MEG signal. The experimental results demonstrate for the first time that the proposed method is able to predict the brain response in both milliseconds and millimeters using only fMRI signal.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, X., He, K.: Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566 (2020)
Cornblath, E.J., et al.: Temporal sequences of brain activity at rest are constrained by white matter structure and modulated by cognitive demands. Commun. Biol. 3(1), 1–12 (2020)
Gallego, J.A., Perich, M.G., Naufel, S.N., Ethier, C., Solla, S.A., Miller, L.E.: Cortical population activity within a preserved neural manifold underlies multiple motor behaviors. Nat. Commun. 9(1), 1–13 (2018)
Glasser, M.F., et al.: A multi-modal parcellation of human cerebral cortex. Nature 536(7615), 171–178 (2016)
Glasser, M.F., et al.: The minimal preprocessing pipelines for the human connectome project. Neuroimage 80, 105–124 (2013)
Gu, A., Dao, T., Ermon, S., Rudra, A., Ré, C.: Hippo: Recurrent memory with optimal polynomial projections. arXiv preprint arXiv:2008.07669 (2020)
Kriegeskorte, N., Mur, M., Bandettini, P.A.: Representational similarity analysis-connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008)
McIntosh, A.R., Mišić, B.: Multivariate statistical analyses for neuroimaging data. Annu. Rev. Psychol. 64, 499–525 (2013)
Musall, S., Kaufman, M.T., Juavinett, A.L., Gluf, S., Churchland, A.K.: Single-trial neural dynamics are dominated by richly varied movements. Nat. Neurosci. 22(10), 1677–1686 (2019)
Norman, K.A., Polyn, S.M., Detre, G.J., Haxby, J.V.: Beyond mind-reading: multi-voxel pattern analysis of FMRI data. Trends Cogn. Sci. 10(9), 424–430 (2006)
Saxena, S., Cunningham, J.P.: Towards the neural population doctrine. Curr. Opin. Neurobiol. 55, 103–111 (2019)
Shine, J.M., et al.: Human cognition involves the dynamic integration of neural activity and neuromodulatory systems. Nat. Neurosci. 22(2), 289–296 (2019)
Stringer, C., Pachitariu, M., Steinmetz, N., Carandini, M., Harris, K.D.: High-dimensional geometry of population responses in visual cortex. Nature 571(7765), 361–365 (2019)
Taghia, J., et al.: Uncovering hidden brain state dynamics that regulate performance and decision-making during cognition. Nat. Commun. 9(1), 1–19 (2018)
Tang, E., Mattar, M.G., Giusti, C., Lydon-Staley, D.M., Thompson-Schill, S.L., Bassett, D.S.: Effective learning is accompanied by high-dimensional and efficient representations of neural activity. Nat. Neurosci. 22(6), 1000–1009 (2019)
Tavares, R.M., et al.: A map for social navigation in the human brain. Neuron 87(1), 231–243 (2015)
Voelker, A.R., Kajić, I., Eliasmith, C.: Legendre memory units: Continuous-time representation in recurrent neural networks. In: Proceedings of the 33st International Conference on Neural Information Processing Systems (2019)
Zhao, C., Gao, X., Emery, W.J., Wang, Y., Li, J.: An integrated spatio-spectral-temporal sparse representation method for fusing remote-sensing images with different resolutions. IEEE Trans. Geosci. Remote Sens. 56(6), 3358–3370 (2018)
Zhao, C., Li, H., Jiao, Z., Du, T., Fan, Y.: A 3D convolutional encapsulated long short-term memory (3DConv-LSTM) model for denoising fMRI data. In: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (eds.) MICCAI 2020. LNCS, vol. 12267, pp. 479–488. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59728-3_47
Acknowledgement
This work was partially supported by NIH R01AG071243, R01MH125928, R01AG049371, U01AG068057, and NSF IIS 2045848, 1845666, 1852606, 1838627, 1837956, 1956002, IIA 2040588.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhao, C., Zhan, L., Thompson, P.M., Huang, H. (2022). Predicting Spatio-Temporal Human Brain Response Using fMRI. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13431. Springer, Cham. https://doi.org/10.1007/978-3-031-16431-6_32
Download citation
DOI: https://doi.org/10.1007/978-3-031-16431-6_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16430-9
Online ISBN: 978-3-031-16431-6
eBook Packages: Computer ScienceComputer Science (R0)
