Abstract
Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by complex symptoms, which makes ASD difficult to be identified. Combining different brain imaging modalities to provide complementary information has been extensively used in the diagnosis of brain disorders. However, it is still very difficult to fully integrate different modalities by capturing the complex connections between different modalities. To solve this problem, we propose a deep low-rank multimodal fusion (DLMF) network that takes into account distribution discrepancy between different modalities. This network aims to learn the complex connections between rest-state functional magnetic resonance imaging (rs-fMRI) and structural magnetic resonance imaging (sMRI) in order to effectively perform multimodal identification of Autism Spectrum Disorder (ASD). Firstly, two different networks are used to extract the features that represent complex information in the rs-fMRI and sMRI data. Then, a measurement function is proposed to quantify distribution discrepancy between different modalities. This measurement function is then incorporated into the loss function of our low-rank multimodal fusion network.
Therefore, our method can reduce the distribution discrepancy between different modalities through joint learning from rs-fMRI and sMRI data. The classifier in our approach adopts Support Vector Machines (SVM). The proposed network was trained with the new loss function using an end-to-end training approach. We verify the effectiveness of our method on a publicly available multimodal dataset: ABIDE database. Experimental results show that our methods are superior to several of the most advanced ASD diagnostic methods currently available.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wang, J., Zhang, F., Jia, X., et al.: Multi-class ASD classification via label distribution learning with class-shared and class-specific decomposition. Med. Image Anal. 1892, 68–73 (2022)
Pappaianni, E., Siugzdaite, R., Vettori, S., et al.: Three shades of grey: detecting brain abnormalities in children with autism by using source-, voxel- and surface-based morphometry. Eur. J. Neurosci. 47(Pt 2) (2017)
Wang, J., Fu, K., Chen, L., et al.: Increased gray matter volume and resting-state functional connectivity in somatosensory cortex and their relationship with autistic symptoms in young boys with autism spectrum disorder. Frontiers Physiol. 588 (2017)
Yttredahl, A.A., Mcrobert, E., Sheler, B., et al.: Abnormal emotional and neural responses to romantic rejection and acceptance in depressed women. J. Affect. Disord. 234, 231 (2018)
McKinnon, M.C., Yucel, K., Nazarov, A., et al.: A meta-analysis examining clinical predictors of hippocampal volume in patients with major depressive disorder. J. Psychiatry Neuro sci. 34(1), 41–54 (2009)
Hajek, T., Kozeny, J., Kopecek, M., et al.: Reduced subgenual cingulate volumes in mood disorders: a meta-analysis. J. Psychiatry Neurosci. 33(2), 91–99 (2008)
Anderson, J.S., Nielsen, J.A., Froehlich, A.L., et al.: Functional connectivity magnetic resonance imaging classification of autism. Brain 134(12), 3742–3754 (2011)
Uddin, L.Q., Supekar, K., Lynch, C.J., et al.: Salience network-based classification and prediction of symptom severity in children with autism. JAMA Psychiat. 70(8), 1–11 (2013)
Hazlett, H.C., Poe, M., Gerig, G., et al.: Magnetic resonance imaging and head circumference study of brain size in autism: birth through age 2 years. Arch Gen Psychiatry 62(12), 1366–1376 (2005)
Redcay, E., Courchesne, E.: When is the brain enlarged in autism? A meta analysis of all brain size reports. Biolog. Psychiatry 58(1), 1–9 (2005)
Dvornek, N.C., Ventola, P., Pelphrey, K.A., et al.: Identifying autism from resting-state fMRI using long short-term memory networks. In: International Workshop on Machine Learning in Medical Imaging, pp. 362–370 (2017)
Wang, M.L., Lian, C.F., Yao, D.R., Zhang, D.Q., et al.: Spatial-temporal dependency modeling and network hub detection for functional MRI analysis via convolutional-recurrent network. IEEE Trans. Biomed. Eng., 2241–2252 (2020)
Wang, J., Wang, Q., Peng, J., et al.: Multi-task diagnosis for autism spectrum disorders using multi-modality features: a multi-center study. Hum. Brain Mapp. 38(6), 3081–3097 (2017)
Tan, W., Thitn, W., Xiang, P., et al.: Multimodal brain image fusion based on multi-level edge-preserving filtering. Biomed. Signal Process. Control 64(11), 102280 (2021)
Zhang, D., Wang, Y., Zhou, L., et al.: Multimodal classification of Alzheimer’s disease and mild cognitive impairment. Neuroimage 55(3), 856–867 (2011)
Dai, Z., Yan, C., Wang, Z., et al.: Discriminative analysis of early Alzheimer’s disease using multimodal imaging and multi-level characterization with multi-classifier (M3). Neuroimage 59(3), 2187–2195 (2012)
Liu, Z., Shen, Y., Lakshminarasimhan, V.B., et al.: Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064 (2018)
Zhu, X., Thung, K.-H., Adeli, E., Zhang, Y., Shen, D.: Maximum mean discrepancy based multiple kernel learning for incomplete multimodality neuroimaging data. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2017, pp. 72–80. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_9
Zhou, T., Thung, K.H., Zhu, X., et al.: Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis. Hum. Brain Mapp. 40(3), 1001–1016 (2018)
Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M.: Multimodal compact bilinear pooling for visual question answering and visual grounding (2016). arXiv preprint arXiv:1606.01847
Zadeh, A., Chen, M., Poria, S., Cambria, E., Morency, L.-P.: Tensor fusion network for multimodal sentiment analysis. In: Empirical Methods in Natural Language Processing, EMNLP (2017)
Chen, Z.G., Wan, Y.Q., Wang, Y.L., et al.: Prognosis of bacterial membrane mixture based on heterogeneous low rank multimodal fusion networks. Zhejiang University School J. (Eng. Ed.) 55(11), 2045–2053 (2021)
Cao, Y.L., Deng, Z.H., Hu, S.D., Wang, S.T.: Classification of Alzheimer’s disease based on personality features and fusion features [J/OL]. Comput. Sci. Explor., 1–13 (2023). http://kns.cnki.net/kcms/detail/11.5602.TP.20220322.1155.002.html
Yan, C.G., Zhang, Y.F.: DPARSF: a MATLAB toolbox for “pipeline” data analysis of resting-state fMRI. Frontiers Syst. Neurosci. 4, 1662–5137 (2010)
Rosasco, L., De Vito, E., Caponnetto, A., et al.: Are loss functions all the same? Neural Comput. 16(5), 1063–1076 (2004)
Li, Y.C., Yeh, C.C.: Some characterizations of convex function. Comput. Math. Appl. 59, 327–337 (2010)
Wu, W.X., Cai, B., Zhang, A.Y., et al.: Deep collaborative learning with application to the study of multimodal brain development. IEEE Trans. Biomed. Eng. 12(66), 3346–3359 (2019)
Andrew, G., Arora, R., Bilmes, J., et al.: Deep canonical correlation analysis. In: International Conference on International Conference on Machine Learning, pp. 1247–1255 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Xue, M., Wang, L., Shen, J., Wang, K., Wu, W., Fu, L. (2023). Deep Low-Rank Multimodal Fusion with Inter-modal Distribution Difference Constraint for ASD Diagnosis. In: Lu, H., et al. Image and Graphics . ICIG 2023. Lecture Notes in Computer Science, vol 14359. Springer, Cham. https://doi.org/10.1007/978-3-031-46317-4_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-46317-4_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46316-7
Online ISBN: 978-3-031-46317-4
eBook Packages: Computer ScienceComputer Science (R0)