GCTN: A Novel Graph Convolutional and Transformer-based Network for Multimodal Neuroimaging Analysis and Neurodegenerative Disease Classification

The current use of unimodal data is insufficient for medical diagnosis, and the fusion of multimodal data is a promising direction for development, as it can combine the unique information from each modality. Deep neural networks have shown excellent classification performance in medical image analysis, effectively extracting deep features for classification. However, how to fuse information from different modalities has been an overlooked problem. This paper proposes a novel neural network model, GCTN, based on Graph Convolutional Network (GCN) and Transformer network, for the analysis of brain imaging data and disease classification. The GCTN model can simultaneously learn functional features and the topological structure of the structural network, forming new deep-level brain connectivity features. The model also employs a dynamic attention mechanism to adaptively adjust the attention allocation of the input data, thereby improving the classification performance. Experimental results show that the GCTN model achieves high accuracy, sensitivity, and specificity in distinguishing normal controls (NC) from mild cognitive impairment (MCI), providing valuable reference for the neuroimaging diagnosis of neurodegenerative diseases.