Skip to main content

Hybrid Multimodality Fusion with Cross-Domain Knowledge Transfer to Forecast Progression Trajectories in Cognitive Decline

  • Conference paper
  • First Online:
Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 Workshops (MICCAI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14394))

Abstract

Magnetic resonance imaging (MRI) and positron emission tomography (PET) are increasingly used to forecast progression trajectories of cognitive decline caused by preclinical and prodromal Alzheimer’s disease (AD). Many existing studies have explored the potential of these two distinct modalities with diverse machine and deep learning approaches. But successfully fusing MRI and PET can be complex due to their unique characteristics and missing modalities. To this end, we develop a hybrid multimodality fusion (HMF) framework with cross-domain knowledge transfer for joint MRI and PET representation learning, feature fusion, and cognitive decline progression forecasting. Our HMF consists of three modules: 1) a module to impute missing PET images, 2) a module to extract multimodality features from MRI and PET images, and 3) a module to fuse the extracted multimodality features. To address the issue of small sample sizes, we employ a cross-domain knowledge transfer strategy from the ADNI dataset, which includes 795 subjects, to independent small-scale AD-related cohorts, in order to leverage the rich knowledge present within the ADNI. The proposed HMF is extensively evaluated in three AD-related studies with 272 subjects across multiple disease stages, such as subjective cognitive decline and mild cognitive impairment. Experimental results demonstrate the superiority of our method over several state-of-the-art approaches in forecasting progression trajectories of AD-related cognitive decline.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amieva, H., et al.: Prodromal Alzheimer’s disease: Successive emergence of the clinical symptoms. Annals of Neurology: Official Journal of the American Neurological Association and the Child Neurology Society 64(5), 492–498 (2008)

    Article  Google Scholar 

  2. van Harten, A.C., et al.: Subjective cognitive decline and risk of MCI: the mayo clinic study of aging. Neurology 91(4), e300–e312 (2018)

    Google Scholar 

  3. Jessen, F., et al.: A conceptual framework for research on subjective cognitive decline in preclinical Alzheimer’s disease. Alzheimer’s Dementia 10(6), 844–852 (2014)

    Article  Google Scholar 

  4. Kryscio, R.J., et al.: Self-reported memory complaints: implications from a longitudinal cohort with autopsies. Neurology 83(15), 1359–1365 (2014)

    Article  Google Scholar 

  5. Petersen, R.C., Smith, G.E., Waring, S.C., Ivnik, R.J., Tangalos, E.G., Kokmen, E.: Mild cognitive impairment: clinical characterization and outcome. Arch. Neurol. 56(3), 303–308 (1999)

    Article  Google Scholar 

  6. Dubois, B., et al.: Research criteria for the diagnosis of Alzheimer’s disease: revising the NINCDS-ADRDA criteria. Lancet Neurol. 6(8), 734–746 (2007)

    Article  Google Scholar 

  7. Mitchell, A.J., Beaumont, H., Ferguson, D., Yadegarfar, M., Stubbs, B.: Risk of dementia and mild cognitive impairment in older people with subjective memory complaints: Meta-analysis. Acta Psychiatr. Scand. 130(6), 439–451 (2014)

    Article  Google Scholar 

  8. Aisen, P., et al.: Report of the task force on designing clinical trials in early (predementia) AD. Neurology 76(3), 280–286 (2011)

    Google Scholar 

  9. Samanta, M.K., Wilson, B., Santhi, K., Kumar, K.S., Suresh, B.: Alzheimer disease and its management: a review. Am. J .Ther. 13(6), 516–526 (2006)

    Article  Google Scholar 

  10. Grueso, S., Viejo-Sobera, R.: Machine learning methods for predicting progression from mild cognitive impairment to Alzheimer’s disease dementia: A systematic review. Alzheimer’s Res. Therapy 13, 1–29 (2021)

    Google Scholar 

  11. Hinrichs, C., Singh, V., Xu, G., Johnson, S.C.: Predictive markers for AD in a multi-modality framework: an analysis of MCI progression in the ADNI population. Neuroimage 55(2), 574–589 (2011)

    Article  Google Scholar 

  12. Zhou, T., Thung, K.H., Liu, M., Shi, F., Zhang, C., Shen, D.: Multi-modal latent space inducing ensemble SVM classifier for early dementia diagnosis with neuroimaging data. Med. Image Anal. 60, 101630 (2020)

    Article  Google Scholar 

  13. Pan, Y., Liu, M., Lian, C., Xia, Y., Shen, D.: Spatially-constrained fisher representation for brain disease identification with incomplete multi-modal neuroimages. IEEE Trans. Med. Imaging 39(9), 2965–2975 (2020)

    Article  Google Scholar 

  14. Senanayake, U., Sowmya, A., Dawes, L.: Deep fusion pipeline for mild cognitive impairment diagnosis. In: IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE 2018, pp. 1394–1997 (2018)

    Google Scholar 

  15. Liu, M., Cheng, D., Wang, K., Wang, Y.: Multi-modality cascaded convolutional neural networks for Alzheimer’s disease diagnosis. Neuroinformatics 16, 295–308 (2018)

    Article  Google Scholar 

  16. Dwivedi, S., Goel, T., Tanveer, M., Murugan, R., Sharma, R.: Multimodal fusion-based deep learning network for effective diagnosis of Alzheimer’s disease. IEEE Multimedia 29(2), 45–55 (2022)

    Article  Google Scholar 

  17. Song, J., Zheng, J., Li, P., Lu, X., Zhu, G., Shen, P.: An effective multimodal image fusion method using MRI and PET for Alzheimer’s disease diagnosis. Front. Digital Jealth 3, 637386 (2021)

    Article  Google Scholar 

  18. Jack Jr, C.R., et al.: The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods. J. Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine 27(4), 685–691(2008)

    Google Scholar 

  19. Xiao, S., et al.: The China longitudinal ageing study: overview of the demographic, psychosocial and cognitive data of the Shanghai sample. J. Mental Health 25(2), 31–136 (2016)

    Google Scholar 

  20. Saykin, A., Wishart, H., Rabin, L., Santulli, R., Flashman, L., West, J., McHugh, T., Mamourian, A.: Older adults with cognitive complaints show brain atrophy similar to that of amnestic MCI. Neurology 67(5), 834–842 (2006)

    Article  Google Scholar 

  21. Goodfellow, I., et al.: Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020)

    Article  MathSciNet  Google Scholar 

  22. Liu, Y., Yue, L., Xiao, S., Yang, W., Shen, D., Liu, M.: Assessing clinical progression from subjective cognitive decline to mild cognitive impairment with incomplete multi-modal neuroimages. Med. Image Anal. 75, 102266 (2022)

    Article  Google Scholar 

  23. Pan, Y., Liu, M., Xia, Y., Shen, D.: Disease-image-specific learning for diagnosis-oriented neuroimage synthesis with incomplete multi-modality data. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6839–6853 (2021)

    Article  Google Scholar 

  24. Yu, M., Guan, H., Fang, Y., Yue, L., Liu, M.: Domain-prior-induced structural MRI adaptation for clinical progression prediction of subjective cognitive decline. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 24–33.Springer (2022)

    Google Scholar 

  25. Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823(2015)

    Google Scholar 

  26. Mensink, T., Verbeek, J., Perronnin, F., Csurka, G.: Metric learning for large scale image classification: generalizing to new classes at near-zero cost. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7573, pp. 488–501. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33709-3_35

    Chapter  Google Scholar 

  27. Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: Feragen, A., Pelillo, M., Loog, M. (eds.) SIMBAD 2015. LNCS, vol. 9370, pp. 84–92. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24261-3_7

    Chapter  Google Scholar 

  28. Vaswani, A., et al.: Attention is all you need. Advances in Neural Information Processing Systems 30 (2017)

    Google Scholar 

Download references

Acknowledgements

M. Yu, A. Bozoki, and M. Liu were partly supported by NIH grant AG073297.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Mingxia Liu .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yu, M. et al. (2023). Hybrid Multimodality Fusion with Cross-Domain Knowledge Transfer to Forecast Progression Trajectories in Cognitive Decline. In: Woo, J., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 Workshops. MICCAI 2023. Lecture Notes in Computer Science, vol 14394. Springer, Cham. https://doi.org/10.1007/978-3-031-47425-5_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-47425-5_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-47424-8

  • Online ISBN: 978-3-031-47425-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics