A descriptive framework for the field of deep learning applications in medical images
Introduction
Medical images play an important role in disease prevention, diagnosis, and prognosis. Its analysis has always been a common problem for academic and medical communities. The traditional medical image-based analysis way is manual reading, which is labor-intensive and time-consuming. In addition, due to different experience levels of readers, there are inconsistencies among observers. To solve this problem, some scholars have tried to recognize medical images automatically by traditional machine learning methods (such as support vector machine). These methods have been mainstream for quite a long time. However, manual feature engineering that relies on expert experience makes these approaches less intelligent. In contrast, deep learning, as an advanced knowledge-based method, especially convolutional neural networks, can automatically extract hidden patterns and features in medical images. With the rapid development of computer technology, deep learning has a better ability to mine medical image knowledge. At present, deep learning has achieved great success in computer-aided diagnosis systems and artificial intelligence decision-making systems, such as skin lesion diagnosis system [1], eye disease diagnosis system [2], [3], [4], thoracic disease diagnosis system [5], [6], [7], etc. To further improve diagnostic performance and enhance clinical workflow efficiency, deep learning also demonstrates its powerful knowledge discovery capabilities in applications like organ and structure segmentation [8], [9], or image quality enhancement [10]. Therefore, a comprehensive review of the applications of deep learning in medical images is valuable and will help promote the advancement of knowledge-based systems in the medical field.
AlexNet [11], one of the most popular convolutional neural networks, won the first place in the ImageNet challenge in 2012, and has since captured widespread attention from researchers on deep learning for images. Subsequently, deeper networks such as GoogleNet [12], VGG [13] and ResNet [14] were proposed. VGG uses multiple consecutive 3 × 3 kernels to improve kernels with larger size (11 × 11, 7 × 7, 5 × 5) in AlexNet, so as to reduce the number of parameters and achieve better performance for training deeper networks. VGG16 and VGG19 are two common VGG network architectures. However, some studies have shown that although deeper networks are beneficial to achieve better generalization performance, the model training will encounter problems such as gradient disappearance and network degradation as the depth increases, making deep networks difficult to train. In this context, ResNet was proposed, identity shortcut connection was introduced to directly skip one or more layers to achieve deep network training. Since its inception, it has aroused great attention in computer vision, and many subsequent studies have benefited from it. For example, DenseNet [15], where the feature map of each layer will be used as inputs to all subsequent layers. Both ResNet and DenseNet adopted the idea of short paths, but the main difference between the two is that DenseNet improves performance by concatenating feature maps of different layers, rather than adding. However, the parameters of the deep network are huge, and there are high requirements for computing resources during training and testing. When there is a need for rapid decision-making, there is often a certain communication delay. Later, some people focused on building lightweight convolutional neural networks that can maintain high performance at the same time. SqueezeNet [16] is a typical lightweight network that can achieve accuracy comparable to AlexNet with 50 times fewer parameters. MobileNet [17], and ShuffleNet [18], etc. are also representatives of lightweight networks. Generally, convolutional neural networks are used in image-based classification, which is not sufficient for medical image analysis. In 2015, Ronneberger et al. [19] proposed a well-known architecture named U-Net for medical image segmentation. This architecture improves segmentation performance by concatenating feature maps in the upsampling path with the corresponding cropped feature map in the downsampling path via skip connection. At present, convolutional neural networks are backbones of these architectures and have achieved performance close to human experts in the analysis of medical images.
Deep learning in medical image analysis is a typical interdisciplinary application. There are already some relevant reviews. In one review [20], we think that the authors paid more attention to deep learning methods, and the applications reviewed were less comprehensive and somewhat outdated, which may not be suitable for physicians without computer technology background and algorithm engineers without medical experience. Litjens et al. [21] have conducted a detailed review of the application of deep learning in medical images, but only covers the period from 2012 to 2017. There is an urgent need to supplement relevant literature for a wider range of applications, due to the rapid increase in the number of literatures in the next two years. In the most recent reviews, Seo et al. [22] made a detailed review of the traditional machine learning and deep learning techniques for a single task (i.e. segmentation) in biomedical images, and focused more on the introduction of the methods. Arif et al. [23] and Altinkaya et al. [24] discussed the application of deep learning based on single-mode medical images (i.e. MRI) in the classification of prostate cancer and the detection of Alzheimer’s disease and dementia states, respectively. Compared with these reviews, this survey focuses more on the multiple applications of deep learning based on multi-mode medical images in multiple anatomic sites, and is supplemented by the introduction of corresponding deep learning techniques, so that readers can have a more holistic and comprehensive understanding of applications of deep learning in medical images. In addition, we must note that as the number of articles grows exponentially, reviewing massive articles one by one is challenging, or even impossible. It will be valuable if a novel framework can be proposed to improve the review efficiency.
Delen and Crossland [25] proposed that extracting related words and terms for text mining can help produce a categorization of target literature. On this basis, Moro et al. [26] employed latent Dirichlet allocation (LDA) method to mine the full text of 219 articles on business intelligence in banking, which could help determine which topics are attracting more attention from researchers. In addition, LDA for text knowledge mining has the merits of being simpler, more robust, and more interpretable than advanced techniques like deep learning [27]. Inspired by this, this survey used LDA as an auxiliary analysis tool. To our best knowledge, it is the first time to use LDA to review literature on the application of deep learning in medical images. But please note that our main purpose is still to review the applications of deep learning in medical images, and LDA is only used as an auxiliary tool to provide us with rough categorization results of articles, which will be more conducive to the following in-depth analysis.
Compared with the traditional way of literature review, the advantages of using LDA as an aid are: first, the traditional way relies entirely on manual reading and summarizing, and cannot provide readers with any prior information about the research field before reading. By comparison, LDA can quickly cluster articles and enable readers to understand the research field from a macro level; Secondly, once the previous way is adopted for literature review, forgetting speed will increase as the number of articles accumulates, making the analysis more difficult. In contrast, the use of LDA in this survey can cluster articles with high topic relevance, thereby saving the analysis time of readers. However, it should be noted that the topic categories provided by LDA are a rough result. To learn more detailed information about applications of deep learning in medical images, this survey selected representative articles for specific analysis on the basis of LDA.
This survey is organized as follows. Section 2 mainly introduces the implementation of LDA and construction of the descriptive review framework. Section 3 describes basic deep learning concepts. Section 4 provides a detailed review for applications in medical images based on deep learning. Section 5 summarizes conclusions, challenges and future research directions.
Section snippets
Article selection
We retrieved articles from PubMed on and before June 26, 2020, based on keywords (Medical image) AND (Deep learning). Sorted by matching degree, there were 2897 articles being retrieved initially. However, not all articles belonged to our research scope and were in line with our purpose of conducting a systematic review of deep learning techniques in medical images. Therefore, we read abstracts of all articles and eliminated irrelevant literature according to the following criteria:
- •
Exclude
Basic concepts of deep learning
To have a deeper understanding of applications of deep learning in medical images, this section mainly introduces the basic concepts of deep learning from two aspects of learning patterns and commonly used neural networks. According to the deep learning patterns, deep learning can be divided into supervised learning, weakly supervised learning and unsupervised learning. The main difference between these three learning schemes is the proportion and granularity discrepancy of the annotated labels
Applications in medical images
This section summarizes the application of 77 representative articles for the field of deep learning in medical image classification, detection, segmentation and generation.
Discussion
The analysis of medical images is an important tool for clinical diagnosis. In this survey, 2068 articles published on PubMed on and before June 26, 2020 about deep learning in medical images were reviewed. The number of articles has experienced exponential growth since 2017. A term set was constructed using keywords and all articles were clustered into 15 topics using LDA. To our best knowledge, LDA was initially used to review the literature on application of deep learning in medical images.
CRediT authorship contribution statement
Yingjie Tian: Conceptualization, Writing - review & editing, Supervision, Project administration, Funding acquisition. Saiji Fu: Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing - original draft.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work has been partially supported by grants from National Natural Science Foundation of China (No. 12071458, 71731009, 61472390).
References (131)
- et al.
Adaptive melanoma diagnosis using evolving clustering, ensemble and deep neural networks
Knowl.-Based Syst.
(2020) - et al.
Deep membrane systems for multitask segmentation in diabetic retinopathy
Knowl.-Based Syst.
(2019) - et al.
Glaucoma diagnosis based on both hidden features and domain knowledge through deep learning models
Knowl.-Based Syst.
(2018) - et al.
Automated identification and grading system of diabetic retinopathy using deep neural networks
Knowl.-Based Syst.
(2019) - et al.
Deepln: A framework for automatic lung nodule detection using multi-resolution CT screening images
Knowl.-Based Syst.
(2020) - et al.
Automated detection of coronary artery disease using different durations of ecg segments with convolutional neural network
Knowl.-Based Syst.
(2017) - et al.
Detection and localization of myocardial infarction based on a convolutional autoencoder
Knowl.-Based Syst.
(2019) - et al.
Dunet: A deformable network for retinal vessel segmentation
Knowl.-Based Syst.
(2019) - et al.
Automatic segmentation of left ventricle using parallel end-end deep convolutional neural networks framework
Knowl.-Based Syst.
(2020) - et al.
A survey on deep learning in medical image analysis
Med. Image Anal.
(2017)
Seeding the survey and analysis of research literature with text mining
Expert Syst. Appl.
Business intelligence in banking: A literature analysis from 2002 to 2013 using text mining and latent dirichlet allocation
Expert Syst. Appl.
Forum latent dirichlet allocation for user interest discovery
Knowl.-based Syst.
Multi-label chest x-ray image classification via category-wise residual attention learning
Pattern Recognit. Lett.
Thorax disease classification with attention guided convolutional neural network
Pattern Recognit. Lett.
Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists
Ann. Oncol.
Superior skin cancer classification by the combination of human and artificial intelligence
Eur. J. Cancer
Deep learning for prediction of obstructive disease from fast myocardial perfusion spect: a multicenter study
JACC: Cardiovascular Imaging
Accurate classification of diminutive colorectal polyps using computer-aided analysis
Gastroenterology
Automated identification of diabetic retinopathy using deep learning
Ophthalmology
Efficacy of a deep learning system for detecting glaucomatous optic neuropathy based on color fundus photographs
Ophthalmology
A deep learning algorithm for prediction of age-related eye disease study severity scale for age-related macular degeneration from color fundus photography
Ophthalmology
Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy
Ophthalmology
Deepseenet: A deep learning model for automated classification of patient-based age-related macular degeneration severity from color fundus photographs
Ophthalmology
Development and validation of deep learning models for screening multiple abnormal findings in retinal fundus images
Ophthalmology
Automated deep learning-based system to identify endothelial cells derived from induced pluripotent stem cells
Stem Cell Rep.
Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks
Gastrointest. Endosc.
Classification for invasion depth of esophageal squamous cell carcinoma using a deep neural network compared with experienced endoscopists
Gastrointest. Endosc.
Commensal correlation network between segmentation and direct area estimation for bi-ventricle quantification
Med. Image Anal.
Identification and quantification of cardiovascular structures from ccta: an end-to-end, rapid, pixel-wise, deep-learning method
JACC: Cardiovascular Imaging
Deep atlas network for efficient 3d left ventricle segmentation on echocardiography
Med. Image Anal.
Efficient multi-scale 3d cnn with fully connected CRF for accurate brain lesion segmentation
Med. Image Anal.
Multi-atlas image registration of clinical data with automated quality assessment using ventricle segmentation
Med. Image Anal.
Brain tumor segmentation with deep neural networks
Med. Image Anal.
Fully automated detection and quantification of macular fluid in oct using deep learning
Ophthalmology
Low-dose ct image denoising using a generative adversarial network with wasserstein distance and perceptual loss
IEEE Trans. Med. Imaging
Imagenet classification with deep convolutional neural networks
Very deep convolutional networks for large-scale image recognition
Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size
Mobilenets: Efficient convolutional neural networks for mobile vision applications
U-net: Convolutional networks for biomedical image segmentation
Deep learning in medical image analysis
Annual Rev. Biomed. Eng.
Machine learning techniques for biomedical image segmentation: An overview of technical aspects and introduction to state-of-art applications
Med. Phys.
Automated classification of significant prostate cancer on MRI: A systematic review on the performance of machine learning applications
Cancers
Detection of alzheimer’s disease and dementia states based on deep learning from MRI images: A comprehensive review
J. Inst. Electron. Comput.
Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study
PLoS Med.
Cited by (31)
ECMS-NET:A multi-task model for early endometrial cancer MRI sequences classification and segmentation of key tumor structures
2024, Biomedical Signal Processing and ControlStratMed: Relevance stratification between biomedical entities for sparsity on medication recommendation
2024, Knowledge-Based SystemsA novel Residual and Gated Network for prostate segmentation on MR images
2024, Biomedical Signal Processing and ControlLinear-exponential loss incorporated deep learning for imbalanced classification
2023, ISA TransactionsSmart IoMT-based segmentation of coronavirus infections using lung CT scans
2023, Alexandria Engineering Journal