Skip to main content
SpringerLink
Log in
Book cover

International Conference on Health Information Science

HIS 2019: Health Information Science pp 193–200Cite as

Deep Learning in Multimodal Medical Image Analysis

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11837))

Abstract

Various imaging modalities (CT, MRI, PET, etc.) encompass abundant information which is different and complementary to each other. It is reasonable to combine images from multiple modalities to make a more accurate assessment. Multimodal medical imaging has shown notable achievements in improving clinical accuracy. Deep learning has achieved great success in image recognition, and also shown huge potential for multimodal medical imaging analysis. This paper gives a review of deep learning in multimodal medical imaging analysis, aiming to provide a starting point for people interested in this field, and highlight gaps and challenges of this topic. Based on the introduction of basic ideas of deep learning and medical imaging, the state-of-the-art multimodal medical image analysis is given, with emphasis on the fusion technique and feature extraction deep models. Multimodal medical image applications, especially cross-modality related, are also summarized.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Litjens, G., Kooi, T., Bejnordi, B.E., et al.: A survey on deep learning in medical image analysis. Med. Image Anal. 42, 60–88 (2017)

    Article  Google Scholar 

  2. Lundervold, A.S., Lundervold, A.: An overview of deep learning in medical imaging focusing on MRI. https://arxiv.org/abs/1811.10052 (2018)

  3. Liu, S., et al.: Deep learning in medical ultrasound analysis: a review, engineering (2019)

    Article  Google Scholar 

  4. James, A.P., Dasarathy, B.V.: Medical image fusion: a survey of the state of the art. Inf. Fusion 19, 4–19 (2014)

    Article  Google Scholar 

  5. Guo, Z., Li, X., Huang, H., Guo, N., Li, Q.: Deep learning-based image segmentation on multimodal medical imaging. IEEE Trans. Radiat. Plasma Med. Sci. 3(2), 162–169 (2019)

    Article  Google Scholar 

  6. Zhang, W., Li, R., Deng, H., Wang, L., Lin, W., et al.: Deep convolutional neural networks for multimodality isointense infant brain image segmentation. NeuroImage 108, 214–224 (2015)

    Article  Google Scholar 

  7. Liu, S., et al.: Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer’s disease. IEEE Trans. Biomed. Eng. 62(4), 1132–1140 (2015)

    Article  Google Scholar 

  8. Suk, H., Shen, D.: Deep learning-based feature representation for AD/MCI classification. In: Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 583–590 (2013)

    Chapter  Google Scholar 

  9. Shi, B., Chen, Y., Zhang, P., Smith, C.D., Liu, J.: Nonlinear feature transformation and deep fusion for Alzheimer’s disease staging analysis. Pattern Recognit. 63, 487–498 (2017)

    Article  Google Scholar 

  10. Suk, H.I., Lee, S.W., Shen, D.G., ADNI: Hierarchical feature representation and multimodal fusion with deep learning for AD MCI diagnosis. NeuroImage 101, 569–582 (2014)

    Article  Google Scholar 

  11. Li, F., Tran, L., Thung, K.H., Ji, S.W., Shen, D.G., Li, J.: A robust deep model for improved classification of AD/MCI patients. IEEE J. Biomed. Health Inform. 19(5), 1610–1616 (2015)

    Article  Google Scholar 

  12. Shi, J., Zheng, X., Li, Y., Zhang, Q., Ying, S.: Multimodal neuroimaging feature learning with multimodal stacked deep polynomial networks for diagnosis of Alzheimer’s disease. IEEE J. Biomed. Health Inform. 22(1), 173–183 (2018)

    Article  Google Scholar 

  13. Gupta, A., Ayhan, M., Maida, A.: Natural image bases to represent neuroimaging data. In: Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA, 16–21 June, pp. 987–994 (2013)

    Google Scholar 

  14. Carneiro, G., Nascimento, J., Bradley, A.P.: Unregistered multiview mammogram analysis with pre-trained deep learning models. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 652–660. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_78

    Chapter  Google Scholar 

  15. Hadad, O., Bakalo, R., Ben-Ari, R., et al.: Classification of breast lesions using cross-modal deep learning. In: IEEE International Symposium on Biomedical Imaging. IEEE (2017)

    Google Scholar 

  16. Nie, D., Wang, L., Gao, Y., Shen, D.: Fully convolutional networks for multi-modality isointense infant brain image segmentation. In: Proceedings of the 13th IEEE International Symposium on Biomedical Imaging, Washington, DC, pp. 1342–1345 (2016)

    Google Scholar 

  17. Zeng, G., Zheng, G.: Multi-stream 3D FCN with multi-scale deep supervision for multi-modality isointense infant brain MR image segmentation. https://arxiv.org/abs/1711.10212, March 2019

  18. Huo, Y., Xu, Z., Moon, H., et al.: SynSeg-Net: synthetic segmentation without target modality ground truth. IEEE Trans. Med. Imaging 38(4), 1016–1025 (2019)

    Article  Google Scholar 

  19. Zhang, Z., Yang, L., Zheng, Y.: Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network. https://arxiv.org/abs/1802.09655 (2018)

  20. Nie, D., Zhang, H., Adeli, E., Liu, L., Shen, D.: 3D deep learning for multi-modal imaging-guided survival time prediction of brain tumor patients. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 212–220. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8_25

    Chapter  Google Scholar 

  21. Wolterink, J.M., Dinkla, A.M., Savenije, M.H.F., Seevinck, P.R., van den Berg, C.A.T., Išgum, I.: Deep MR to CT synthesis using unpaired data. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2017. LNCS, vol. 10557, pp. 14–23. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68127-6_2

    Chapter  Google Scholar 

  22. Chartsias, A., Joyce, T., Dharmakumar, R., Tsaftaris, S.A.: Adversarial image synthesis for unpaired multi-modal cardiac data. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2017. LNCS, vol. 10557, pp. 3–13. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68127-6_1

    Chapter  Google Scholar 

  23. Van Nguyen, H., Zhou, K., Vemulapalli, R.: Cross-domain synthesis of medical images using efficient location-sensitive deep network. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9349, pp. 677–684. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24553-9_83

    Chapter  Google Scholar 

  24. Sevetlidis, V., Giuffrida, M.V., Tsaftaris, S.A.: Whole image synthesis using a deep encoder-decoder network. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2016. LNCS, vol. 9968, pp. 127–137. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46630-9_13

    Chapter  Google Scholar 

  25. Xiang, L., et al.: Deep embedding convolutional neural network for synthesizing CT image from T1-weighted MR image. Med. Image Anal. 47, 31–44 (2018)

    Article  Google Scholar 

  26. Nie, D., et al.: Medical image synthesis with context-aware generative adversarial networks. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10435, pp. 417–425. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66179-7_48

    Chapter  Google Scholar 

  27. Costa, P., et al.: Towards adversarial retinal image synthesis. https://arxiv.org/abs/1701.08974 (2017)

Download references

Author information

Authors and Affiliations

  1. China Academy of Information and Communications Technology, Beijing, 100045, China

    Yan Xu

Authors
  1. Yan Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yan Xu .

Editor information

Editors and Affiliations

  1. Victoria University, Footscray, VIC, Australia

    Hua Wang

  2. Victoria University, Footscray, VIC, Australia

    Siuly Siuly

  3. Swinburne University of Technology, Hawthorn, VIC, Australia

    Rui Zhou

  4. Instituto de Salud Carlos III, Madrid, Spain

    Fernando Martin-Sanchez

  5. Victoria University, Footscray, VIC, Australia

    Yanchun Zhang

  6. Vrije University, Amsterdam, The Netherlands

    Zhisheng Huang

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xu, Y. (2019). Deep Learning in Multimodal Medical Image Analysis. In: Wang, H., Siuly, S., Zhou, R., Martin-Sanchez, F., Zhang, Y., Huang, Z. (eds) Health Information Science. HIS 2019. Lecture Notes in Computer Science(), vol 11837. Springer, Cham. https://doi.org/10.1007/978-3-030-32962-4_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32962-4_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32961-7

  • Online ISBN: 978-3-030-32962-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation