Skip to main content

Advertisement

Springer Nature Link
Log in

Convolutional neural networks in medical image understanding: a survey

  • Review Article
  • Published:
Evolutionary Intelligence Aims and scope Submit manuscript

Abstract

Imaging techniques are used to capture anomalies of the human body. The captured images must be understood for diagnosis, prognosis and treatment planning of the anomalies. Medical image understanding is generally performed by skilled medical professionals. However, the scarce availability of human experts and the fatigue and rough estimate procedures involved with them limit the effectiveness of image understanding performed by skilled medical professionals. Convolutional neural networks (CNNs) are effective tools for image understanding. They have outperformed human experts in many image understanding tasks. This article aims to provide a comprehensive survey of applications of CNNs in medical image understanding. The underlying objective is to motivate medical image understanding researchers to extensively apply CNNs in their research and diagnosis. A brief introduction to CNNs has been presented. A discussion on CNN and its various award-winning frameworks have been presented. The major medical image understanding tasks, namely image classification, segmentation, localization and detection have been introduced. Applications of CNN in medical image understanding of the ailments of brain, breast, lung and other organs have been surveyed critically and comprehensively. A critical discussion on some of the challenges is also presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Abadi M, Agarwal A, Barham P (2016) Tensorflow: large-scale machine learning on heterogeneous distributed systems. CoRR Arxiv: 1603.04467

  2. Akkus Z, Galimzianova A, Hoogi A, Rubin DL, Erickson BJ (2017) Deep learning for brain MRI segmentation: state of the art and future directions. J Digit Imaging 30(4):449–459

    Article  Google Scholar 

  3. Al-Rfou R, Alain G, Almahairi A, Angermueller C, Bahdanau D, Ballas N, Bengio Y (2016) Theano: a python framework for fast computation of mathematical expressions. arXiv:1605.02688

  4. Anthimopoulos M, Christodoulidis S, Ebner L, Christe A, Mougiakakou SG (2016) Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imaging 35(5):1207–1216

    Article  Google Scholar 

  5. Apou G, Schaadt NS, Naegel B (2016) Structures in normal breast tissue. Comput Biol Med 74:91–102

    Article  Google Scholar 

  6. Bassi PAS, Attux R (2020) A deep convolutional neural network for COVID-19 detection using chest X-rays

  7. Bayramoglu N, Kannala J, Heikkilä J (2015) Human epithelial type 2 cell classification with convolutional neural networks. In: Proceedings of the 15th IEEE international conference on bioinformatics and bioengineering, BIBE, pp 1–6

  8. Brosch T, Tang LYW, Yoo Y, Li DKB, Traboulsee A, Tam RC (2016) Deep 3-D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation. IEEE Trans Med Imaging 35(5):1229–1239

    Article  Google Scholar 

  9. CS231N convolutional neural network for visual recogntion (2019). http://cs231n.stanford.edu/. Accessed 24 June 2019

  10. Chen H, Ni D, Qin J (2015) Standard plane localization in fetal ultrasound via domain transferred deep neural networks. IEEE J Biomed Health Inform 19(5):1627–1636

    Article  Google Scholar 

  11. Chollet F (2018) Keras: the Python deep learning library. https://keras.io/. Accessed 24 June 2019

  12. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, (CVPR), pp 1800–1807

  13. Chowdhury NK, Rahman MM, Kabir MA (2020) Pdcovidnet: a parallel-dilated convolutional neural network architecture for detecting COVID-19 from chest X-ray images

  14. Ciompi F, de Hoop B, van Riel SJ (2015) Automatic classification of pulmonary peri-fissural nodules in computed tomography using an ensemble of 2-D views and a convolutional neural network out-of-the-box. Med Image Anal 26(1):195–202. https://doi.org/10.1016/j.media.2015.08.001

    Article  Google Scholar 

  15. Collobert R, Kavukcuoglu K, Farabet C (2011) Torch7: a Matlab-like environment for machine learning. In: BigLearn, NIPS workshop, EPFL-CONF-192376

  16. Dasgupta A, Singh S (2017) A fully convolutional neural network based structured prediction approach towards the retinal vessel segmentation. In: Proceedings of the 14th IEEE international symposium on biomedical imaging (ISBI), pp 248–251

  17. Emad O, Yassine IA, Fahmy AS (2015) Automatic localization of the left ventricle in cardiac MRI images using deep learning. In: Proceedings of the 37th IEEE annual international conference on engineering in medicine and biology society (EMBC), pp 683–686

  18. Gao M, Bagci U, Lu L, Wu A, Buty M (2018) Holistic classification of CT attenuation patterns for interstitial lung diseases via deep convolutional neural networks. CMBBE Imaging Vis 6(1):1–6

    Google Scholar 

  19. Gao XW, Hui R (2016)A deep learning based approach to classification of CT brain images. In: Proceedings of the SAI computing conference, pp 28–31

  20. Gao XW, Li W, Loomes M, Wang L (2017) A fused deep learning architecture for viewpoint classification of echocardiography. Inf Fusion 36:103–113

    Article  Google Scholar 

  21. Gao Z, Zhang J, Zhou L, Wang L (2014) HEp-2i cell image classification with deep convolutional neural networks. IEEE J Biomed Health Inform 21:416–428

    Article  Google Scholar 

  22. Ghoshal B, Tucker A (2020) Estimating uncertainty and interpretability in deep learning for coronavirus (COVID-19) detection arXiv:2003.10769

  23. Gordienko Y, Gang P, Hui J, Zeng W, Kochura Y, Alienin O, Rokovyi O, Stirenko S (2017) Deep learning with lung segmentation and bone shadow exclusion techniques for chest X-ray analysis of lung cancer. CoRR arXiv:1712.07632

  24. Hasan MK, Alam MA, Elahi MTE, Roy S, Wahid SR (2020) CVR-NET: a deep convolutional neural network for coronavirus recognition from chest radiography images

  25. Havaei M, Davy A, Warde-Farley D (2017) Brain tumor segmentation with deep neural networks. Med Image Anal 35:18–31

    Article  Google Scholar 

  26. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, (CVPR), pp 770–778

  27. Hesamian M, Jia W, He X (2019) Deep learning techniques for medical image segmentation: achievements and challenges. J Digit Imaging 32:582–596

    Article  Google Scholar 

  28. Ho TK (1995) Random decision forests. In: Proceedings of the 3rd IEEE international conference on document analysis and recognition, vol 1, pp 278–282

  29. Hossain MZ, Sohel F, Shiratuddin MF, Laga H (2018) A comprehensive survey of deep learning for image captioning. CoRR arXiv:1810.04020

  30. Hubel D, Wiesel T (1959) Receptive fields of single neurones in the cat’s striate cortex. J Physiol 148(3):574–591

    Article  Google Scholar 

  31. ICPR 2018 international conference on pattern recognition (2019). http://www.icpr2018.org. Accessed 24 June 2019

  32. ISLES challenge 2018 ischemic stroke lesion segmentation (2018). http://www.isles-challenge.org. Accessed 24 June 2019

  33. Jia, Y., Shelhamer E, Donahue J (2014) Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22th ACM international conference on multimedia, MM ’14, pp 675–678. ACM, New York

  34. Jiao Z, Gao X, Wang Y, Li J (2016) A deep feature based framework for breast masses classification. Neurocomputing 197:221–231

    Article  Google Scholar 

  35. Jusman Y, Ng SC, Abu Osman NA (2014) Intelligent screening systems for cervical cancer. Sci World J 2014:810368

    Article  Google Scholar 

  36. Kallenberg M, Petersen K, Nielsen M (2016) Unsupervised deep learning applied to breast density segmentation and mammographic risk scoring. IEEE Trans Med Imaging 35(5):1322–1331

    Article  Google Scholar 

  37. Kamnitsas K, Ferrante E, Parisot S (2016) Deepmedic for brain tumor segmentation. In: Proceedings of the international workshop on brainlesion: glioma, multiple sclerosis, stroke and traumatic brain injuries. Springer, pp 138–149

  38. Kendall A, Gal Y (2017) What uncertainties do we need in bayesian deep learning for computer vision? CoRR arXiv:1703.04977

  39. Kermany DS, Goldbaum M, Cai W, Valentim CC, Liang H, Baxter SL, McKeown A, Yang G, Wu X, Yan F et al (2018) Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172(5):1122–1131

    Article  Google Scholar 

  40. Kiranyaz S, Ince T, Gabbouj M (2016) Real-time patient-specific ECG classification by 1-D convolutional neural networks. IEEE Trans Biomed Eng 63(3):664–675

    Article  Google Scholar 

  41. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems 25: proceedings of the 26th annual conference on neural information processing systems, pp 1106–1114

  42. Larochelle H, Bengio Y (2008) Classification using discriminative restricted Boltzmann machines. In: Proceedings of the 25th international conference on machine learning, pp 536–543

  43. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  44. Li Q, Cai W, Wang X, Zhou Y, Feng DD, Chen M (2014) Medical image classification with convolutional neural network. In: Proceedings of the 13th international conference on control, automation, robotics and vision (ICARCV), pp 844–848

  45. Maghdid HS, Asaad AT, Ghafoor KZ, Sadiq AS, Khan MK (2020)Diagnosing COVID-19 pneumonia from X-ray and CT images using deep learning and transfer learning algorithms. CoRR arXiv:2004.00038

  46. Maji D, Santara A, Mitra P (2016) Ensemble of deep convolutional neural networks for learning to detect retinal vessels in fundus images. arXiv preprint arXiv:1603.04833

  47. Mavroforakis ME, Georgiou HV, Dimitropoulos N (2006) Mammographic masses characterization based on localized texture and dataset fractal analysis using linear, neural and support vector machine classifiers. Artif Intell Med 37(2):145–162

    Article  Google Scholar 

  48. Menze BH, Jakab A, Bauer S (2015) The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans Med Imaging 34(10):1993–2024

    Article  Google Scholar 

  49. Moeskops P, Viergever MA, Mendrik AM, de Vries LS, Benders MJNL, Isgum I (2016) Automatic segmentation of MR brain images with a convolutional neural network. IEEE Trans Med Imaging 35(5):1252–1261

    Article  Google Scholar 

  50. Nasr-Esfahani E, Samavi S, Karimi N (2016) Melanoma detection by analysis of clinical images using convolutional neural network. In: Proceedings of the 38th IEEE annual international conference on engineering in medicine and biology society, EMBC, pp 1373–1376

  51. Nervana Systems Inc (2018) neon. http://neon.nervanasys.com/docs/latest/. Accessed 24 June 2019

  52. Ovalle JEA, González FA, Ramos-Pollán R, Oliveira JL, Guevara-López MÁ (2016) Representation learning for mammography mass lesion classification with convolutional neural networks. Comput Methods Prog Biomed 127:248–257

    Article  Google Scholar 

  53. Paszke A, Gross S, Chintala S, Chanan G (2018) Tensors and dynamic neural networks in Python with strong GPU acceleration. https://pytorch.org/. Accessed 24 June 2019

  54. Pereira S, Pinto A, Alves V, Silva CA (2015) Deep convolutional neural networks for the segmentation of gliomas in multi-sequence MRI. In: Brainlesion: glioma, multiple sclerosis, stroke and traumatic brain injuries—first international workshop, Brainles 2015, held in conjunction with MICCAI 2015, Munich, Germany, October 5, 2015, revised selected papers, pp 131–143

  55. Pereira S, Pinto A, Alves V, Silva CA (2016) Brain tumor segmentation using convolutional neural networks in MRI images. IEEE Trans Med Imaging 35(5):1240–1251

    Article  Google Scholar 

  56. Phillips NA, Rajpurkar P, Sabini, M, Krishnan R, Zhou, S, Pareek, A, Phu NM, Wang C, Ng AY, Lungren MP (2020) Chexphoto: 10,000+ smartphone photos and synthetic photographic transformations of chest X-rays for benchmarking deep learning robustness. CoRR arXiv:2007.06199

  57. Pratt H, Coenen F, Broadbent DM, Harding SP, Zheng Y (2016) Convolutional neural networks for diabetic retinopathy. In: Proceedings of the 20th conference on medical image understanding and analysis, MIUA, pp 200–205

  58. Rajpurkar P, Irvin J, Zhu K, Yang B, Mehta H, Duan T, Ding DY, Bagul A, Langlotz C, Shpanskaya KS, Lungren MP, Ng AY (2017) Chexnet: radiologist-level pneumonia detection on chest X-rays with deep learning. CoRR arXiv:1711.05225

  59. Ranzato M, Hinton GE, LeCun Y (2015) Guest editorial: deep learning. Int J Comput Vis 113(1):1–2

    Article  MathSciNet  Google Scholar 

  60. Ravi D, Wong C, Deligianni F, Berthelot M, Pérez JA, Lo B, Yang G (2017) Deep learning for health informatics. IEEE J Biomed Health Inform 21(1):4–21

    Article  Google Scholar 

  61. Ribeiro E, Uhl A, Häfner M (2016) Colonic polyp classification with convolutional neural networks. In: Proceedings of the 29th IEEE international symposium on computer-based medical systems, (CBMS), pp 253–258

  62. Sajjad M, Khan S, Muhammad K, Wu W, Ullah A, Baik SW (2019) Multi-grade brain tumor classification using deep CNN with extensive data augmentation. J Comput Sci 30:174–182. https://doi.org/10.1016/j.jocs.2018.12.003

    Article  Google Scholar 

  63. Sarraf S, Tofighi G (2016) Classification of Alzheimer’s disease using fMRI data and deep learning convolutional neural networks. Computer Research Repository. arXiv:1603.08631

  64. Seide F, Agarwal A (2016) CNTK: Microsoft’s open-source deep-learning toolkit. In: Proceedings of the 22nd ACM international conference on knowledge discovery and data mining, p 2135

  65. Sermanet P, Eigen D, Zhang X, Mathieu M, Fergus R, LeCun Y (2013) Overfeat: integrated recognition, localization and detection using convolutional networks. Computing Research Repository. arXiv:1312.6229

  66. Shakeri M, Tsogkas S, Ferrante E, Lippe S, Kadoury S, Paragios N, Kokkinos I (2016) Sub-cortical brain structure segmentation using F-CNNs. In: Proceedigns of the IEEE 13th international symposium on biomedical imaging (ISBI). IEEE, pp 269–272

  67. Shankar K, Zhang Y, Liu Y, Wu L, Chen CH (2020) Hyperparameter tuning deep learning for diabetic retinopathy fundus image classification. IEEE Access 8:118164–118173

    Article  Google Scholar 

  68. Shen W, Zhou M (2017) Multi-crop convolutional neural networks for lung nodule malignancy suspiciousness classification. Pattern Recognit 61:663–673

    Article  Google Scholar 

  69. Shkolyar A, Gefen A, Benayahu D, Greenspan H (2015) Automatic detection of cell divisions (mitosis) in live-imaging microscopy images using convolutional neural networks. In: Proceedings of the 37th annual international conference of the IEEE engineering in medicine and biology society (EMBC), pp 743–746

  70. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Computer Research Repository. arXiv:1409.1556

  71. Sirinukunwattana K, Raza SEA, Tsang Y, Snead DRJ, Cree IA, Rajpoot NM (2016) Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans Med Imaging 35(5):1196–1206

    Article  Google Scholar 

  72. Su H, Liu F, Xie Y (2015) Region segmentation in histopathological breast cancer images using deep convolutional neural network. In: Proceedings of the 12th IEEE international symposium on biomedical imaging (ISBI), pp 55–58

  73. Sun W, Tseng TB, Zhang J, Qian W (2017) Enhancing deep convolutional neural network scheme for breast cancer diagnosis with unlabeled data. Comput Med Imaging Graph 57:4–9

    Article  Google Scholar 

  74. Szegedy C, Liu W, Jia Y, Sermanet P, Reed SE (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1–9

  75. van Grinsven MJ, van Ginneken B, Hoyng CB (2016) Fast convolutional neural network training using selective data sampling: application to hemorrhage detection in color fundus images. IEEE Trans Med Imaging 35(5):1273–1284

    Article  Google Scholar 

  76. Wang D, Khosla A, Gargeya R, Irshad H, Beck AH (2016) Deep learning for identifying metastatic breast cancer. Computer Research Repository. arXiv:1606.05718

  77. Xing F, Xie Y, Yang L (2016) An automatic learning-based framework for robust nucleus segmentation. IEEE Trans Med Imaging 35(2):550–566

    Article  Google Scholar 

  78. Xu J, Luo X, Wang G, Gilmore H, Madabhushi A (2016) A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing 191:214–223

    Article  Google Scholar 

  79. Zeiler MD, Fergus R (2013) Visualizing and understanding convolutional networks. Computer Research Repository. arXiv:1311.2901

  80. Zhao L, Jia K (2016) Multiscale CNNs for brain tumor segmentation and diagnosis. Comput Math Methods Med 2016:8356294:1–8356294:7

    Article  Google Scholar 

  81. Zilly JG, Buhmann JM, Mahapatra D (2017) Glaucoma detection using entropy sampling and ensemble learning for automatic optic cup and disc segmentation. Comput Med Imaging Graph 55:28–41

    Article  Google Scholar 

  82. Zou Y, Li L, Wang Y (2015) Classifying digestive organs in wireless capsule endoscopy images based on deep convolutional neural network. In: Proceedings of the IEEE international conference on digital signal processing, DSP, pp 1274–1278

  83. Zreik M, Leiner T, de Vos BD, van Hamersvelt RW, Viergever MA, Isgum I (2016) Automatic segmentation of the left ventricle in cardiac CT angiography using convolutional neural networks. In: Proceedings of the 13th IEEE international symposium on biomedical imaging (ISBI), pp 40–43

Download references

Acknowledgements

The authors acknowledge with gratitude the support received from REVA University, Bengaluru, and M. S. Ramaiah University of Applied Sciences, Bengaluru, India.

Author information

Authors and Affiliations

  1. REVA University, Bengaluru, India

    D. R. Sarvamangala

  2. Ramaiah University of Applied Sciences, Bengaluru, India

    Raghavendra V. Kulkarni

Authors
  1. D. R. Sarvamangala
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Raghavendra V. Kulkarni
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to D. R. Sarvamangala.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sarvamangala, D.R., Kulkarni, R.V. Convolutional neural networks in medical image understanding: a survey. Evol. Intel. 15, 1–22 (2022). https://doi.org/10.1007/s12065-020-00540-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12065-020-00540-3

Keywords