Elsevier

Knowledge-Based Systems

Volume 210, 27 December 2020, 106445
Knowledge-Based Systems

A descriptive framework for the field of deep learning applications in medical images

https://doi.org/10.1016/j.knosys.2020.106445Get rights and content

Abstract

Deep learning in medical image analysis is a typical interdisciplinary application, which needs support and cooperation of computer techniques and medical experience, and has broad application prospects. Since 2017, the number of related published articles has increased exponentially, imposing a burden on the literature review in this field. In this survey, we clustered 2068 retrieved articles into 15 topics through Latent Dirichlet Allocation (LDA) and provided a rough overview on the application of deep learning in medical images. On this basis, we conducted a detailed review with 77 top representative articles. We built a descriptive review framework based on LDA and discussed classification, object detection, segmentation, and image generation applications in medical images for the field of deep learning from the perspective of image modalities. We ended with discussing current challenges and future research directions of deep learning in medical images analysis.

Introduction

Medical images play an important role in disease prevention, diagnosis, and prognosis. Its analysis has always been a common problem for academic and medical communities. The traditional medical image-based analysis way is manual reading, which is labor-intensive and time-consuming. In addition, due to different experience levels of readers, there are inconsistencies among observers. To solve this problem, some scholars have tried to recognize medical images automatically by traditional machine learning methods (such as support vector machine). These methods have been mainstream for quite a long time. However, manual feature engineering that relies on expert experience makes these approaches less intelligent. In contrast, deep learning, as an advanced knowledge-based method, especially convolutional neural networks, can automatically extract hidden patterns and features in medical images. With the rapid development of computer technology, deep learning has a better ability to mine medical image knowledge. At present, deep learning has achieved great success in computer-aided diagnosis systems and artificial intelligence decision-making systems, such as skin lesion diagnosis system [1], eye disease diagnosis system [2], [3], [4], thoracic disease diagnosis system [5], [6], [7], etc. To further improve diagnostic performance and enhance clinical workflow efficiency, deep learning also demonstrates its powerful knowledge discovery capabilities in applications like organ and structure segmentation [8], [9], or image quality enhancement [10]. Therefore, a comprehensive review of the applications of deep learning in medical images is valuable and will help promote the advancement of knowledge-based systems in the medical field.

AlexNet [11], one of the most popular convolutional neural networks, won the first place in the ImageNet challenge in 2012, and has since captured widespread attention from researchers on deep learning for images. Subsequently, deeper networks such as GoogleNet [12], VGG [13] and ResNet [14] were proposed. VGG uses multiple consecutive 3 × 3 kernels to improve kernels with larger size (11 × 11, 7 × 7, 5 × 5) in AlexNet, so as to reduce the number of parameters and achieve better performance for training deeper networks. VGG16 and VGG19 are two common VGG network architectures. However, some studies have shown that although deeper networks are beneficial to achieve better generalization performance, the model training will encounter problems such as gradient disappearance and network degradation as the depth increases, making deep networks difficult to train. In this context, ResNet was proposed, identity shortcut connection was introduced to directly skip one or more layers to achieve deep network training. Since its inception, it has aroused great attention in computer vision, and many subsequent studies have benefited from it. For example, DenseNet [15], where the feature map of each layer will be used as inputs to all subsequent layers. Both ResNet and DenseNet adopted the idea of short paths, but the main difference between the two is that DenseNet improves performance by concatenating feature maps of different layers, rather than adding. However, the parameters of the deep network are huge, and there are high requirements for computing resources during training and testing. When there is a need for rapid decision-making, there is often a certain communication delay. Later, some people focused on building lightweight convolutional neural networks that can maintain high performance at the same time. SqueezeNet [16] is a typical lightweight network that can achieve accuracy comparable to AlexNet with 50 times fewer parameters. MobileNet [17], and ShuffleNet [18], etc. are also representatives of lightweight networks. Generally, convolutional neural networks are used in image-based classification, which is not sufficient for medical image analysis. In 2015, Ronneberger et al. [19] proposed a well-known architecture named U-Net for medical image segmentation. This architecture improves segmentation performance by concatenating feature maps in the upsampling path with the corresponding cropped feature map in the downsampling path via skip connection. At present, convolutional neural networks are backbones of these architectures and have achieved performance close to human experts in the analysis of medical images.

Deep learning in medical image analysis is a typical interdisciplinary application. There are already some relevant reviews. In one review [20], we think that the authors paid more attention to deep learning methods, and the applications reviewed were less comprehensive and somewhat outdated, which may not be suitable for physicians without computer technology background and algorithm engineers without medical experience. Litjens et al. [21] have conducted a detailed review of the application of deep learning in medical images, but only covers the period from 2012 to 2017. There is an urgent need to supplement relevant literature for a wider range of applications, due to the rapid increase in the number of literatures in the next two years. In the most recent reviews, Seo et al. [22] made a detailed review of the traditional machine learning and deep learning techniques for a single task (i.e. segmentation) in biomedical images, and focused more on the introduction of the methods. Arif et al. [23] and Altinkaya et al. [24] discussed the application of deep learning based on single-mode medical images (i.e. MRI) in the classification of prostate cancer and the detection of Alzheimer’s disease and dementia states, respectively. Compared with these reviews, this survey focuses more on the multiple applications of deep learning based on multi-mode medical images in multiple anatomic sites, and is supplemented by the introduction of corresponding deep learning techniques, so that readers can have a more holistic and comprehensive understanding of applications of deep learning in medical images. In addition, we must note that as the number of articles grows exponentially, reviewing massive articles one by one is challenging, or even impossible. It will be valuable if a novel framework can be proposed to improve the review efficiency.

Delen and Crossland [25] proposed that extracting related words and terms for text mining can help produce a categorization of target literature. On this basis, Moro et al. [26] employed latent Dirichlet allocation (LDA) method to mine the full text of 219 articles on business intelligence in banking, which could help determine which topics are attracting more attention from researchers. In addition, LDA for text knowledge mining has the merits of being simpler, more robust, and more interpretable than advanced techniques like deep learning [27]. Inspired by this, this survey used LDA as an auxiliary analysis tool. To our best knowledge, it is the first time to use LDA to review literature on the application of deep learning in medical images. But please note that our main purpose is still to review the applications of deep learning in medical images, and LDA is only used as an auxiliary tool to provide us with rough categorization results of articles, which will be more conducive to the following in-depth analysis.

Compared with the traditional way of literature review, the advantages of using LDA as an aid are: first, the traditional way relies entirely on manual reading and summarizing, and cannot provide readers with any prior information about the research field before reading. By comparison, LDA can quickly cluster articles and enable readers to understand the research field from a macro level; Secondly, once the previous way is adopted for literature review, forgetting speed will increase as the number of articles accumulates, making the analysis more difficult. In contrast, the use of LDA in this survey can cluster articles with high topic relevance, thereby saving the analysis time of readers. However, it should be noted that the topic categories provided by LDA are a rough result. To learn more detailed information about applications of deep learning in medical images, this survey selected representative articles for specific analysis on the basis of LDA.

This survey is organized as follows. Section 2 mainly introduces the implementation of LDA and construction of the descriptive review framework. Section 3 describes basic deep learning concepts. Section 4 provides a detailed review for applications in medical images based on deep learning. Section 5 summarizes conclusions, challenges and future research directions.

Section snippets

Article selection

We retrieved articles from PubMed on and before June 26, 2020, based on keywords (Medical image) AND (Deep learning). Sorted by matching degree, there were 2897 articles being retrieved initially. However, not all articles belonged to our research scope and were in line with our purpose of conducting a systematic review of deep learning techniques in medical images. Therefore, we read abstracts of all articles and eliminated irrelevant literature according to the following criteria:

  • Exclude

Basic concepts of deep learning

To have a deeper understanding of applications of deep learning in medical images, this section mainly introduces the basic concepts of deep learning from two aspects of learning patterns and commonly used neural networks. According to the deep learning patterns, deep learning can be divided into supervised learning, weakly supervised learning and unsupervised learning. The main difference between these three learning schemes is the proportion and granularity discrepancy of the annotated labels

Applications in medical images

This section summarizes the application of 77 representative articles for the field of deep learning in medical image classification, detection, segmentation and generation.

Discussion

The analysis of medical images is an important tool for clinical diagnosis. In this survey, 2068 articles published on PubMed on and before June 26, 2020 about deep learning in medical images were reviewed. The number of articles has experienced exponential growth since 2017. A term set was constructed using keywords and all articles were clustered into 15 topics using LDA. To our best knowledge, LDA was initially used to review the literature on application of deep learning in medical images.

CRediT authorship contribution statement

Yingjie Tian: Conceptualization, Writing - review & editing, Supervision, Project administration, Funding acquisition. Saiji Fu: Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing - original draft.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work has been partially supported by grants from National Natural Science Foundation of China (No. 12071458, 71731009, 61472390).

References (131)

  • DelenDursun et al.

    Seeding the survey and analysis of research literature with text mining

    Expert Syst. Appl.

    (2008)
  • MoroSérgio et al.

    Business intelligence in banking: A literature analysis from 2002 to 2013 using text mining and latent dirichlet allocation

    Expert Syst. Appl.

    (2015)
  • ChenChaotao et al.

    Forum latent dirichlet allocation for user interest discovery

    Knowl.-based Syst.

    (2017)
  • GuanQingji et al.

    Multi-label chest x-ray image classification via category-wise residual attention learning

    Pattern Recognit. Lett.

    (2020)
  • GuanQingji et al.

    Thorax disease classification with attention guided convolutional neural network

    Pattern Recognit. Lett.

    (2020)
  • HaenssleHolger A et al.

    Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists

    Ann. Oncol.

    (2018)
  • HeklerAchim et al.

    Superior skin cancer classification by the combination of human and artificial intelligence

    Eur. J. Cancer

    (2019)
  • BetancurJulian et al.

    Deep learning for prediction of obstructive disease from fast myocardial perfusion spect: a multicenter study

    JACC: Cardiovascular Imaging

    (2018)
  • ChenPeng-Jen et al.

    Accurate classification of diminutive colorectal polyps using computer-aided analysis

    Gastroenterology

    (2018)
  • GargeyaRishab et al.

    Automated identification of diabetic retinopathy using deep learning

    Ophthalmology

    (2017)
  • LiZhixi et al.

    Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs

    Ophthalmology

    (2018)
  • GrassmannFelix et al.

    A deep learning algorithm for prediction of age-related eye disease study severity scale for age-related macular degeneration from color fundus photography

    Ophthalmology

    (2018)
  • SayresRory et al.

    Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy

    Ophthalmology

    (2019)
  • PengYifan et al.

    Deepseenet: A deep learning model for automated classification of patient-based age-related macular degeneration severity from color fundus photographs

    Ophthalmology

    (2019)
  • SonJaemin et al.

    Development and validation of deep learning models for screening multiple abnormal findings in retinal fundus images

    Ophthalmology

    (2020)
  • KusumotoDai et al.

    Automated deep learning-based system to identify endothelial cells derived from induced pluripotent stem cells

    Stem Cell Rep.

    (2018)
  • HorieYoshimasa et al.

    Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks

    Gastrointest. Endosc.

    (2019)
  • NakagawaKentaro et al.

    Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists

    Gastrointest. Endosc.

    (2019)
  • LuoGongning et al.

    Commensal correlation network between segmentation and direct area estimation for bi-ventricle quantification

    Med. Image Anal.

    (2020)
  • BaskaranLohendran et al.

    Identification and quantification of cardiovascular structures from ccta: an end-to-end, rapid, pixel-wise, deep-learning method

    JACC: Cardiovascular Imaging

    (2020)
  • DongSuyu et al.

    Deep atlas network for efficient 3d left ventricle segmentation on echocardiography

    Med. Image Anal.

    (2020)
  • KamnitsasKonstantinos et al.

    Efficient multi-scale 3d cnn with fully connected CRF for accurate brain lesion segmentation

    Med. Image Anal.

    (2017)
  • DubostFlorian et al.

    Multi-atlas image registration of clinical data with automated quality assessment using ventricle segmentation

    Med. Image Anal.

    (2020)
  • HavaeiMohammad et al.

    Brain tumor segmentation with deep neural networks

    Med. Image Anal.

    (2017)
  • SchleglThomas et al.

    Fully automated detection and quantification of macular fluid in oct using deep learning

    Ophthalmology

    (2018)
  • YangQingsong et al.

    Low-dose ct image denoising using a generative adversarial network with wasserstein distance and perceptual loss

    IEEE Trans. Med. Imaging

    (2018)
  • KrizhevskyAlex et al.

    Imagenet classification with deep convolutional neural networks

  • Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent...
  • SimonyanKaren et al.

    Very deep convolutional networks for large-scale image recognition

    (2014)
  • Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, Deep residual learning for image recognition, in: Proceedings of the...
  • Gao Huang, Zhuang Liu, Laurens Van Der Maaten, Kilian Q Weinberger, Densely connected convolutional networks, in:...
  • IandolaForrest N et al.

    Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size

    (2016)
  • HowardAndrew G et al.

    Mobilenets: Efficient convolutional neural networks for mobile vision applications

    (2017)
  • Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, Jian Sun, Shufflenet: An extremely efficient convolutional neural network for...
  • RonnebergerOlaf et al.

    U-net: Convolutional networks for biomedical image segmentation

  • ShenDinggang et al.

    Deep learning in medical image analysis

    Annual Rev. Biomed. Eng.

    (2017)
  • SeoHyunseok et al.

    Machine learning techniques for biomedical image segmentation: An overview of technical aspects and introduction to state-of-art applications

    Med. Phys.

    (2020)
  • ArifMuhammad et al.

    Automated classification of significant prostate cancer on MRI: A systematic review on the performance of machine learning applications

    Cancers

    (2020)
  • AltinkayaEmre et al.

    Detection of alzheimer’s disease and dementia states based on deep learning from MRI images: A comprehensive review

    J. Inst. Electron. Comput.

    (2020)
  • ZechJohn R et al.

    Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study

    PLoS Med.

    (2018)
  • Cited by (31)

    View all citing articles on Scopus
    View full text