Abstract
Reliable and efficient transferability of 3D convolutional neural networks (3D-CNNs) is an important but extremely challenging issue in medical image analysis, due to small-sized samples and the domain shift problem (e.g., caused by the use of different scanners, protocols and/or subject populations in different sites/datasets). Although previous studies proposed to pretrain CNNs on ImageNet, models’ transferability is usually limited due to semantic gap between natural and medical images. In this work, we try to answer a key question: how to learn transferable 3D-CNNs from scratch based on a small (e.g., tens or hundreds) medical image dataset? We focus on the case of structural MRI-based brain disorder classification using four benchmark datasets (i.e., ADNI-1, ADNI-2, ADNI-3 and AIBL) to address this problem. (1) We explore the influence of different network architectures on model transferability, and find that appropriately deepening or widening a network can increase the transferability (e.g., with improved sensitivity). (2) We analyze the contributions of different parts of 3D-CNNs to the transferability, and verify that fine-tuning CNNs can significantly enhance the transferability. This is different from the previous finding that fine-tuning CNNs (pretrained on ImageNet) cannot improve the model transferability in 2D medical image analysis. (3) We also study the between-task transferability when a model is trained on a source task from scratch and applied to a related target task. Experimental results show that, compared to directly training CNN on related target tasks, CNN pretrained on a source task can yield significantly better performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., et al.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)
Mazurowski, M.A., Buda, M., Saha, A., Bashir, M.R.: Deep learning in radiology: an overview of the concepts and a survey of the state of the art with focus on MRI. J. Magn. Reson. Imaging 49, 939–954 (2019)
Guan, H., Liu, Y., Yang, E., Yap, P.T., Shen, D., Liu, M.: Multi-site MRI harmonization via attention-guided deep domain adaptation for brain disorder identification. Med. Image Anal. 71, 102076 (2021)
Morid, M.A., Borjali, A., Del Fiol, G.: A scoping review of transfer learning research on medical image analysis using ImageNet. Computers in Biology and Medicine (2020)
Cuingnet, R., Gerardin, E., Tessieras, J., et al.: Automatic classification of patients with Alzheimer’s disease from structural MRI: a comparison of ten methods using the ADNI database. NeuroImage 56(2), 766–781 (2011)
Tajbakhsh, N., et al.: Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Trans. Med. Imaging 35, 1299–1312 (2016)
Wen, J., Thibeau-Sutre, E., Diaz-Melo, M., et al.: Convolutional neural networks for classification of Alzheimer’s disease: overview and reproducible evaluation. Med. Image Anal. 63, 1–19 (2020)
Guan, Z., Kumar, R., Fung, Y.R., Wu, Y., Fiterau, M.: A comprehensive study of Alzheimer’s disease classification using convolutional neural networks. arXiv:1904.07950 (2019)
Liu, M., Zhang, J., Adeli, E., Shen, D.: Landmark-based deep multi-instance learning for brain disease diagnosis. Med. Image Anal. 43, 157–168 (2018)
Guan, H., Liu, M.: Domain adaptation for medical image analysis: a survey. arXiv:2102.09508 (2021)
AlBadawy, E.A., Saha, A., Mazurowski, M.A.: Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Med. Phys. 45(3), 1150–1158 (2018)
Pooch, E.H., Ballester, P.L., Barros, R.C.: Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification. arXiv:1909.01940 (2019)
Stacke, K., Eilertsen, G., Unger, J., Lundström, C.: A closer look at domain shift for deep learning in histopathology. arXiv:1909.11575 (2019)
Wang, M., Zhang, D., Huang, J., Yap, P.T., Shen, D., Liu, M.: Identifying autism spectrum disorder with multi-site fMRI via low-rank domain adaptation. IEEE Trans. Med. Imaging 39(3), 644–655 (2019)
Jack Jr, C.R., Bernstein, M.A., Fox, N.C., et al.: The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods. J. Magn. Reson. Imaging 27(4), 685–691 (2008)
Jack Jr, C.R., et al.: Magnetic resonance imaging in Alzheimer’s Disease Neuroimaging Initiative 2. Alzheimer’s Dementia 11(7), 740–756 (2015)
Weiner, M.W., et al.: The Alzheimer’s Disease Neuroimaging Initiative 3: Continued innovation for clinical trial improvement. Alzheimer’s Dementia 13(5), 561–571 (2017)
Ellis, K.A., Bush, A.I., Darby, D., et al.: The Australian Imaging, Biomarkers and lifestyle (AIBL) study of aging: Methodology and baseline characteristics of 1112 individuals recruited for a longitudinal study of Alzheimer’s disease. Int. Psychogeriatrics 21(4), 672–687 (2009)
Raghu, M., Zhang, C., Kleinberg, J., Bengio, S.: Transfusion: Understanding transfer learning for medical imaging. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 3347–3357 (2019)
Gauthier, S., et al.: Mild cognitive impairment. Lancet 367(9518), 1262–1270 (2006)
Sabbagh, M.N., et al.: Early detection of mild cognitive impairment (MCI) in primary care. J. Prev. Alzheimer’s Disease 7, 165–170 (2020)
Acknowledgements
This work was supported in part by NIH grants (Nos. AG041721, MH109773 and MH117943).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Guan, H., Wang, L., Yao, D., Bozoki, A., Liu, M. (2021). Learning Transferable 3D-CNN for MRI-Based Brain Disorder Classification from Scratch: An Empirical Study. In: Lian, C., Cao, X., Rekik, I., Xu, X., Yan, P. (eds) Machine Learning in Medical Imaging. MLMI 2021. Lecture Notes in Computer Science(), vol 12966. Springer, Cham. https://doi.org/10.1007/978-3-030-87589-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-87589-3_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87588-6
Online ISBN: 978-3-030-87589-3
eBook Packages: Computer ScienceComputer Science (R0)
