Neural Video: A Novel Framework for Interpreting the Spatiotemporal Activities of the Human Brain

Abstract

Deep learning has become a powerful tool for brain image analysis. However, high dimensionality hinders the direct application of deep learning models to 4D functional magnetic resonance imaging (fMRI) data of the brain. Previous methods usually simplified fMRI data with Regions-Of-Interest (ROI) based methods, which may lead to information loss. To address this issue, we proposed a novel framework that converts 4D fMRI data into 3D “neural video” based on area-preserving geometry mapping. In detail, 3D cortical surface mesh constructed with FreeSurfer could be converted into 2D planar mesh by using area-preserving geometry mapping, and then each fMRI volume could be aligned to the 2D planar mesh and then converted into a 2D image. Thus, a 4D fMRI scan could be converted into a 3D neural video. We further constructed CNN+Transformer models to process the converted neural video. We evaluated the framework with gender (females vs. males) and brain age (22–25 years vs. 31–35 years) classification tasks using data from the S1200 data release of the Human Connectome Project (HCP). The classification accuracy are 0.8811 ± 0.0254, 0.8612 ± 0.0199 and 0.8996 ± 0.0278 for the left hemisphere, right hemisphere and whole brain in gender classification as well as 0.5996 ± 0.0396, 0.6369 ± 0.0387 and 0.6479 ± 0.0453 in brain age classification. The results suggest that the proposed framework may provide a new avenue for deep learning of 4D spatiotemporal data such as brain fMRI.