Elsevier

Information Fusion

Volume 66, February 2021, Pages 111-137
Information Fusion

Full length article
A survey on deep learning in medicine: Why, how and when?

https://doi.org/10.1016/j.inffus.2020.09.006Get rights and content

Highlights

  • We review the state-of-the-art focusing on the application of DL in medicine.

  • We expose a categorization of Deep Learning models used and applied in medicine.

  • We classify medicine-related DL applications into macro-areas and sub-areas.

  • We thoroughly discuss the recent and open challenges related to DL in medicine.

Abstract

New technologies are transforming medicine, and this revolution starts with data. Health data, clinical images, genome sequences, data on prescribed therapies and results obtained, data that each of us has helped to create. Although the first uses of artificial intelligence (AI) in medicine date back to the 1980s, it is only with the beginning of the new millennium that there has been an explosion of interest in this sector worldwide. We are therefore witnessing the exponential growth of health-related information with the result that traditional analysis techniques are not suitable for satisfactorily management of this vast amount of data. AI applications (especially Deep Learning), on the other hand, are naturally predisposed to cope with this explosion of data, as they always work better as the amount of training data increases, a phase necessary to build the optimal neural network for a given clinical problem. This paper proposes a comprehensive and in-depth study of Deep Learning methodologies and applications in medicine. An in-depth analysis of the literature is presented; how, where and why Deep Learning models are applied in medicine are discussed and reviewed. Finally, current challenges and future research directions are outlined and analysed.

Introduction

In the coming years, Artificial Intelligence (AI) will have an increasingly important role in the field of medicine, where it is already making a difference today. Medicine based on the observation of events has been, for many years, ever since the time of Hippocrates, the epistemological guiding criterion of the healthcare profession. This approach has evolved, with the progress of medicine, into what is termed Evidence-Based Medicine (EBM). Today, indeed, medicine based on signs which cannot be observed by any human doctor but can become evident with the use of Big Data and Deep Learning (DL) techniques has been developed. Such techniques are able to consider and process much more information than is possible for any human. State-of-the-art Deep Neural Networks (DNNs), also known as DL modes [1] have demonstrated remarkable results in image processing, classification and data analysis.

DL is increasingly attracting the interest of researchers in the medical and healthcare sectors, since, by using medical data, it is possible to increase the accuracy of medical applications. In particular, DL is rapidly replacing classic neural network techniques, named artificial neural networks (ANNs), whose goal is to mimic the human brain. This trend can be motivated by the following reasons. Firstly, DL can provide a better interpretation of a very complex phenomenon than classic statistical approaches if high-dimensional datasets are available, and the performance of a DL is directly proportional to the input size. This is a common scenario in medicine, where, as pointed out in [2], large amounts of data (about 15 to 20 TB) are collected and stored in optimized databases every day, also by using Cloud computing platforms [3]. Furthermore, DL is characterized by a high degree of flexibility. Medical data include different types of unstructured data, such as images, signals, genetic expressions and text data. Thanks to the complexity of their architectures, DL frameworks are able to benefit from this heterogeneity by achieving high levels of abstraction in data analysis. Finally, the high level of automation [4]. ML algorithms require a manual intervention to select the fundamental information from the input data and the corresponding transformation rules [5]. This is a crucial challenge because an experts’ decision is needed and, therefore, there is a corresponding increase in the time and costs for a diagnosis [6]. However, DL can determine these elements by using large samples of examples. There are two main consequences of this facility. Firstly, there is a significant reduction in the cost and time of treatment and diagnosis, Secondly, the independence of the diagnosis means that patients can talk directly to data scientists and, by running software, can understand the cause of their disease and obtain the best treatment.

The application of AI in real contexts can result in numerous potential advantages, such as the execution speed, potential reduction in costs, both direct and indirect, better diagnostic accuracy, greater clinical and operational efficiency (“algorithms don’t sleep”) and the possibility of providing access to the clinical information even for people who cannot otherwise benefit from this for geographical, political and economic reasons.

A great number of publications and surveys have addressed the use of DL in medicine, focusing on specific challenges or medical fields [7], [8], [9], [10]. Nevertheless, most of these works are lacking in details, difficult to compare and do not provide the reader with a comprehensive overview of the applications of DL in the general medical area. Fig. 1 presents a keywords-generated tree-map extracted from a Scopus1 dataset composed of papers related to “Deep Learning” and “Medicine” as input words. From 2016 until now, more than 1200 papers have been considered within this dataset. The tree-map has been generated by using the bibliometrix R-package,2 an open-source tool for quantitative research in scientometrics and bibliometrics that includes all the main bibliometric methods of analysis. By analysing the tree-map (starting from the left side and considering the biggest squares), it is evident that DL in Medicine is mostly applied in the task of image processing, with a great focus on diagnostics. By continuing the analysis towards the right, some crucial keywords can be observed, such as “aged”, “personalized medicine”, and “classification”. Summarizing the results of the keyword-based tree-map, it is possible to have an overview of the main medical fields, the principal tasks performed, and the most frequently used algorithms relating to DL in medicine during the last few years.

Starting from the above considerations, in this paper our aim is to provide an extensive analysis of DL applications in medicine, also categorizing DL models in relation to their applications in different medical areas. Afterwards, a comprehensive study of the state-of-the-art DL in medicine will be performed, taking into account existing DL surveys focused on specific medical fields. In summary, with this paper we aim to make the following contribution:

  • 1.

    We will review the state-of-the-art in papers and surveys, especially of recent years, focusing on the application of DL in medicine, including all medical areas.

  • 2.

    We will present a categorization of the DL models used and applied in medicine and give clear definitions of each, also providing an overview of hybrid DL architectures.

  • 3.

    We will comprehensively classify medicine-related DL applications into macro-areas, also describing their sub-areas and the key aspects of the applied DL models.

  • 4.

    We will analyse and discuss the recent and open challenges related to DL in medicine, also addressing future research directions, in order to provide the reader with a clear overview of the real-world scenario.

The rest of the paper is organized as follows, as depicted in Fig. 2. In Section 2, the various DL models are described and analysed in depth, with hybrid architectures also presented. In Section 3, a comprehensive overview and classification of DL applications in medicine is provided, including also a description of the main properties of the applied DL models. Section 4 presents a review of the kinds of medical data and hyperparameter optimization techniques. The current challenges in relation to the application of DL in medicine and future research directions are outlined in Section 5. Finally, in Section 6 our conclusions are presented.

Section snippets

Deep learning models

In this section we will provide a comprehensive overview of DL models applied in medicine. Starting from an in-depth study of the literature, we will present the main families of DL architectures: Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Autoencoders (AE), Generative Adversarial Networks (GAN), Deep Belief Networks (DBN) and Hybrid Architectures (HA) (see Fig. 3).

In the following subsections of this section, the various DL models will be discussed: Section 2.1 the

Applications fields of DL in medicine

DL achieves a remarkable accuracy and quality of its results thanks to its multi-layer architecture, which is able to obtain a high level of abstraction by working with large data samples. For this reason, it is gaining a great popularity in all fields where the process of information extraction from data involves various problems, such as the medical sector [93], [94], [95].

In detail, the analysis of medical data encounters three main issues, which are summarized in the following discussion.

Data structure and hyperparameters optimization

DL methodologies in medicine, as well as in all in the other fields where these techniques can be applied, often require the analysis of some “algorithmic” problems that can arise from data or from the algorithm itself. For these reasons, in the following Section a brief discussion on two of the main issues involved in the usage of Neural network approaches are discussed: on one side the format of stored data usually affect the class of algorithms that can be used on a specific problem, so in

Challenges and future research directions

DL is becoming the new paradigm in the analysis of medical data, as confirmed by the results discussed in Section 3. In addition, in recent years many other medical fields are beginning to benefit from the ability of these models to extract information from very different kinds of data. However, the complexity of DL models, the heterogeneity of medical data and the necessary interaction between machines and humans pose several issues, which must be taken into account in any assessment of future

Conclusions

DL is changing the cultural paradigm of medicine: its applications could become increasingly indispensable in terms of providing answers in contexts of high complexity and uncertainty and in order to allow doctors to have more time to take care of the medical needs of their patients. However, data are not values; any intervention based on data must be personalized, also taking into account the frequently contradictory nature of the knowledge provided in the literature. DL will be useful mainly

CRediT authorship contribution statement

Francesco Piccialli: Conceptualization, Methodology, Investigation, Writing, Visualization, Supervision. Vittorio Di Somma: Data curation, Writing, Investigation. Fabio Giampaolo: Writing, Resources, Review & editing. Salvatore Cuomo: Formal analysis, Writing - review & editing. Giancarlo Fortino: Writing - review & editing, Validation.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The authors dedicate this work to their friend and colleague Prof. Antonio Picariello, who passed away prematurely. The memory of his wonderful and kind soul will always remain in our hearts.

This work was supported by the CUP-in-un-click (CUP-in-One-Click) research, Italy project [Regione Campania - Bando RIS3 2018 - Fase 2]. The authors would like to thank the M.O.D.A.L. research laboratory (http://www.labdma.unina.it/index.php/modal/) for their efforts and support.

References (305)

  • PhamT. et al.

    Predicting healthcare trajectories from medical records: A deep learning approach

    J. Biomed. Inform.

    (2017)
  • AckleyD.H. et al.

    A learning algorithm for Boltzmann machines

    Cogn. Sci.

    (1985)
  • RibeiroM. et al.

    A study of deep convolutional auto-encoders for anomaly detection in videos

    Pattern Recognit. Lett.

    (2018)
  • HintonG.E.

    Learning multiple layers of representation

    Trends Cogn. Sci.

    (2007)
  • HassanM.M. et al.

    Human emotion recognition using deep belief network architecture

    Inf. Fusion

    (2019)
  • ZreikM. et al.

    Deep learning analysis of the myocardium in coronary CT angiography for identification of patients with functionally significant coronary artery stenosis

    Med. Image Anal.

    (2018)
  • CaballoM. et al.

    Deep learning-based segmentation of breast masses in dedicated breast CT imaging: Radiomic feature stability between radiologists and artificial intelligence

    Comput. Biol. Med.

    (2020)
  • ZhaoR. et al.

    Deep learning and its applications to machine health monitoring

    Mech. Syst. Signal Process.

    (2019)
  • KulkarniS. et al.

    Artificial intelligence in medicine: where are we now?

    Acad. Radiol.

    (2020)
  • GoodfellowI. et al.

    Deep Learning

    (2016)
  • YueL. et al.

    Deep learning for heterogeneous medical data analysis

    World Wide Web

    (2020)
  • EstevaA. et al.

    A guide to deep learning in healthcare

    Nat. Med.

    (2019)
  • ChenD. et al.

    Deep learning and alternative learning strategies for retrospective real-world clinical data

    NPJ Digit. Med.

    (2019)
  • AkayA. et al.

    Deep learning: current and emerging applications in medicine and technology

    IEEE J. Biomed. Health Inf.

    (2019)
  • ZhangZ. et al.

    Deep learning in omics: a survey and guideline

    Brief. Funct. Genom.

    (2019)
  • NielsenM.A.

    Neural Networks and Deep Learning, Vol. 2018

    (2015)
  • LeCunY. et al.

    Handwritten digit recognition with a back-propagation network

  • ChengG. et al.

    When deep learning meets metric learning: Remote sensing image scene classification via learning discriminative CNNs

    IEEE Trans. Geosci. Remote Sens.

    (2018)
  • ZhaoZ.-Q. et al.

    Object detection with deep learning: A review

    IEEE Trans. Neural Netw. Learn. Syst.

    (2019)
  • ChenL.-C. et al.

    Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2017)
  • RawatW. et al.

    Deep convolutional neural networks for image classification: A comprehensive review

    Neural Comput.

    (2017)
  • AhmedI. et al.

    Exploring deep learning models for overhead view multiple object detection

    IEEE Internet Things J.

    (2020)
  • KarpathyA. et al.

    Large-scale video classification with convolutional neural networks

  • KrizhevskyA. et al.

    Imagenet classification with deep convolutional neural networks

  • RonnebergerO. et al.

    U-net: Convolutional networks for biomedical image segmentation

  • AkhtarN. et al.

    Interpretation of intelligence in CNN-pooling processes: a methodological survey

    Neural Comput. Appl.

    (2019)
  • J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE...
  • N. Suda, V. Chandra, G. Dasika, A. Mohanty, Y. Ma, S. Vrudhula, J.-s. Seo, Y. Cao, Throughput-optimized OpenCL-based...
  • SimonyanK. et al.

    Very deep convolutional networks for large-scale image recognition

  • MengD. et al.

    Liver fibrosis classification based on transfer learning and FCNet for ultrasound images

    Ieee Access

    (2017)
  • TangY. et al.

    Scene text detection and segmentation based on cascaded convolution neural networks

    IEEE Trans. Image Process.

    (2017)
  • K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference...
  • F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, X. Tang, Residual attention network for image...
  • HersheyS. et al.

    CNN architectures for large-scale audio classification

  • DongH. et al.

    Automatic brain tumor detection and segmentation using u-net based fully convolutional networks

  • ÇiçekÖ. et al.

    3D U-Net: learning dense volumetric segmentation from sparse annotation

  • ZhouZ. et al.

    Unet++: A nested u-net architecture for medical image segmentation

  • BadrinarayananV. et al.

    Segnet: A deep convolutional encoder-decoder architecture for image segmentation

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2017)
  • GirshickR. et al.

    Rich feature hierarchies for accurate object detection and semantic segmentation

  • DaiJ. et al.

    R-fcn: Object detection via region-based fully convolutional networks

  • Cited by (226)

    • Contour-induced parallel graph reasoning for liver tumor segmentation

      2024, Biomedical Signal Processing and Control
    View all citing articles on Scopus
    View full text