Abstract
Magnetic resonance imaging (MRI) and positron emission tomography (PET) are increasingly used to forecast progression trajectories of cognitive decline caused by preclinical and prodromal Alzheimer’s disease (AD). Many existing studies have explored the potential of these two distinct modalities with diverse machine and deep learning approaches. But successfully fusing MRI and PET can be complex due to their unique characteristics and missing modalities. To this end, we develop a hybrid multimodality fusion (HMF) framework with cross-domain knowledge transfer for joint MRI and PET representation learning, feature fusion, and cognitive decline progression forecasting. Our HMF consists of three modules: 1) a module to impute missing PET images, 2) a module to extract multimodality features from MRI and PET images, and 3) a module to fuse the extracted multimodality features. To address the issue of small sample sizes, we employ a cross-domain knowledge transfer strategy from the ADNI dataset, which includes 795 subjects, to independent small-scale AD-related cohorts, in order to leverage the rich knowledge present within the ADNI. The proposed HMF is extensively evaluated in three AD-related studies with 272 subjects across multiple disease stages, such as subjective cognitive decline and mild cognitive impairment. Experimental results demonstrate the superiority of our method over several state-of-the-art approaches in forecasting progression trajectories of AD-related cognitive decline.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amieva, H., et al.: Prodromal Alzheimer’s disease: Successive emergence of the clinical symptoms. Annals of Neurology: Official Journal of the American Neurological Association and the Child Neurology Society 64(5), 492–498 (2008)
van Harten, A.C., et al.: Subjective cognitive decline and risk of MCI: the mayo clinic study of aging. Neurology 91(4), e300–e312 (2018)
Jessen, F., et al.: A conceptual framework for research on subjective cognitive decline in preclinical Alzheimer’s disease. Alzheimer’s Dementia 10(6), 844–852 (2014)
Kryscio, R.J., et al.: Self-reported memory complaints: implications from a longitudinal cohort with autopsies. Neurology 83(15), 1359–1365 (2014)
Petersen, R.C., Smith, G.E., Waring, S.C., Ivnik, R.J., Tangalos, E.G., Kokmen, E.: Mild cognitive impairment: clinical characterization and outcome. Arch. Neurol. 56(3), 303–308 (1999)
Dubois, B., et al.: Research criteria for the diagnosis of Alzheimer’s disease: revising the NINCDS-ADRDA criteria. Lancet Neurol. 6(8), 734–746 (2007)
Mitchell, A.J., Beaumont, H., Ferguson, D., Yadegarfar, M., Stubbs, B.: Risk of dementia and mild cognitive impairment in older people with subjective memory complaints: Meta-analysis. Acta Psychiatr. Scand. 130(6), 439–451 (2014)
Aisen, P., et al.: Report of the task force on designing clinical trials in early (predementia) AD. Neurology 76(3), 280–286 (2011)
Samanta, M.K., Wilson, B., Santhi, K., Kumar, K.S., Suresh, B.: Alzheimer disease and its management: a review. Am. J .Ther. 13(6), 516–526 (2006)
Grueso, S., Viejo-Sobera, R.: Machine learning methods for predicting progression from mild cognitive impairment to Alzheimer’s disease dementia: A systematic review. Alzheimer’s Res. Therapy 13, 1–29 (2021)
Hinrichs, C., Singh, V., Xu, G., Johnson, S.C.: Predictive markers for AD in a multi-modality framework: an analysis of MCI progression in the ADNI population. Neuroimage 55(2), 574–589 (2011)
Zhou, T., Thung, K.H., Liu, M., Shi, F., Zhang, C., Shen, D.: Multi-modal latent space inducing ensemble SVM classifier for early dementia diagnosis with neuroimaging data. Med. Image Anal. 60, 101630 (2020)
Pan, Y., Liu, M., Lian, C., Xia, Y., Shen, D.: Spatially-constrained fisher representation for brain disease identification with incomplete multi-modal neuroimages. IEEE Trans. Med. Imaging 39(9), 2965–2975 (2020)
Senanayake, U., Sowmya, A., Dawes, L.: Deep fusion pipeline for mild cognitive impairment diagnosis. In: IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE 2018, pp. 1394–1997 (2018)
Liu, M., Cheng, D., Wang, K., Wang, Y.: Multi-modality cascaded convolutional neural networks for Alzheimer’s disease diagnosis. Neuroinformatics 16, 295–308 (2018)
Dwivedi, S., Goel, T., Tanveer, M., Murugan, R., Sharma, R.: Multimodal fusion-based deep learning network for effective diagnosis of Alzheimer’s disease. IEEE Multimedia 29(2), 45–55 (2022)
Song, J., Zheng, J., Li, P., Lu, X., Zhu, G., Shen, P.: An effective multimodal image fusion method using MRI and PET for Alzheimer’s disease diagnosis. Front. Digital Jealth 3, 637386 (2021)
Jack Jr, C.R., et al.: The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods. J. Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine 27(4), 685–691(2008)
Xiao, S., et al.: The China longitudinal ageing study: overview of the demographic, psychosocial and cognitive data of the Shanghai sample. J. Mental Health 25(2), 31–136 (2016)
Saykin, A., Wishart, H., Rabin, L., Santulli, R., Flashman, L., West, J., McHugh, T., Mamourian, A.: Older adults with cognitive complaints show brain atrophy similar to that of amnestic MCI. Neurology 67(5), 834–842 (2006)
Goodfellow, I., et al.: Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020)
Liu, Y., Yue, L., Xiao, S., Yang, W., Shen, D., Liu, M.: Assessing clinical progression from subjective cognitive decline to mild cognitive impairment with incomplete multi-modal neuroimages. Med. Image Anal. 75, 102266 (2022)
Pan, Y., Liu, M., Xia, Y., Shen, D.: Disease-image-specific learning for diagnosis-oriented neuroimage synthesis with incomplete multi-modality data. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6839–6853 (2021)
Yu, M., Guan, H., Fang, Y., Yue, L., Liu, M.: Domain-prior-induced structural MRI adaptation for clinical progression prediction of subjective cognitive decline. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 24–33.Springer (2022)
Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823(2015)
Mensink, T., Verbeek, J., Perronnin, F., Csurka, G.: Metric learning for large scale image classification: generalizing to new classes at near-zero cost. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7573, pp. 488–501. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33709-3_35
Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: Feragen, A., Pelillo, M., Loog, M. (eds.) SIMBAD 2015. LNCS, vol. 9370, pp. 84–92. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24261-3_7
Vaswani, A., et al.: Attention is all you need. Advances in Neural Information Processing Systems 30 (2017)
Acknowledgements
M. Yu, A. Bozoki, and M. Liu were partly supported by NIH grant AG073297.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Yu, M. et al. (2023). Hybrid Multimodality Fusion with Cross-Domain Knowledge Transfer to Forecast Progression Trajectories in Cognitive Decline. In: Woo, J., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 Workshops. MICCAI 2023. Lecture Notes in Computer Science, vol 14394. Springer, Cham. https://doi.org/10.1007/978-3-031-47425-5_24
Download citation
DOI: https://doi.org/10.1007/978-3-031-47425-5_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47424-8
Online ISBN: 978-3-031-47425-5
eBook Packages: Computer ScienceComputer Science (R0)