Skip to main content
Springer Nature Link
Log in

On the Necessity of Fine-Tuned Convolutional Neural Networks for Medical Imaging

  • Chapter
  • First Online:
Deep Learning and Convolutional Neural Networks for Medical Image Computing

Part of the book series: Advances in Computer Vision and Pattern Recognition ((ACVPR))

  • 8877 Accesses

Abstract

This study aims to address two central questions. First, are fine-tuned convolutional neural networks (CNNs) necessary for medical imaging applications? In response, we considered four medical vision tasks from three different medical imaging modalities, and studied the necessity of fine-tuned CNNs under varying amounts of training data. Second, to what extent the knowledge is to be transferred? In response, we proposed a layer-wise fine-tuning scheme to examine how the extent or depth of fine-tuning contributes to the success of knowledge transfer. Our experiments consistently showed that the use of a pre-trained CNN with adequate fine-tuning outperformed or, in the worst case, performed as well as a CNN trained from scratch. The performance gap widened when reduced training sets were used for training and fine-tuning. Our results further revealed that the required level of fine-tuning differed from one application to another, suggesting that neither shallow tuning nor deep tuning may be the optimal choice for a particular application. Layer-wise fine-tuning may offer a practical way to reach the best performance for the application at hand based on the amount of available data. We conclude that knowledge transfer from natural images is necessary and that the level of tuning should be chosen experimentally.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    http://polyp.grand-challenge.org/.

References

  1. Azizpour H, Razavian AS, Sullivan J, Maki A, Carlsson S (2014) From generic to specific deep representations for visual recognition. arXiv preprint arXiv:1406.5774

  2. Penatti OA, Nogueira K, dos Santos JA (2015) Do deep features generalize from everyday objects to remote sensing and aerial scenes domains? In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 44–51

    Google Scholar 

  3. Razavian AS, Azizpour H, Sullivan J, Carlsson S (2014) CNN features off-the-shelf: an astounding baseline for recognition. In: 2014 IEEE conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 512–519

    Google Scholar 

  4. Arevalo J, Gonzalez F, Ramos-Pollan R, Oliveira J, Guevara Lopez M (2015) Convolutional neural networks for mammography mass lesion classification. In: 2015 37th Annual international conference of the IEEE engineering in medicine and biology society (EMBC), pp 797–800. doi:10.1109/EMBC.2015.7318482

  5. Bar Y, Diamant I, Wolf L, Greenspan H (2015) Deep learning with non-medical training used for chest pathology identification. In: SPIE medical imaging. International Society for Optics and Photonics, pp 94,140V–94,140V

    Google Scholar 

  6. Chen H, Dou Q, Ni D, Cheng JZ, Qin J, Li S, Heng PA (2015) Automatic fetal ultrasound standard plane detection using knowledge transferred recurrent neural networks. In: Medical image computing and computer-assisted intervention–MICCAI 2015. Springer, Berlin, pp 507–514

    Google Scholar 

  7. van Ginneken B, Setio AA, Jacobs C, Ciompi F (2015) Off-the-shelf convolutional neural network features for pulmonary nodule detection in computed tomography scans. In: 2015 IEEE 12th International symposium on biomedical imaging (ISBI), pp 286–289

    Google Scholar 

  8. Shin HC, Lu L, Kim L, Seff A, Yao J, Summers RM (2015) Interleaved text/image deep mining on a very large-scale radiology database. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1090–1099

    Google Scholar 

  9. Carneiro G, Nascimento J, Bradley A (2015) Unregistered multiview mammogram analysis with pre-trained deep learning models. In: Navab N, Hornegger J, Wells WM, Frangi AF (eds) Medical image computing and computer-assisted intervention MICCAI 2015. Lecture notes in computer science, vol 9351. Springer International Publishing, Berlin, pp 652–660

    Google Scholar 

  10. Shin HC, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J, Mollura D, Summers RM (2016) Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans Med Imaging PP(99):1–1. doi:10.1109/TMI.2016.2528162

    Google Scholar 

  11. Ciompi F, de Hoop B, van Riel SJ, Chung K, Scholten ET, Oudkerk M, de Jong PA, Prokop M, van Ginneken B (2015) Automatic classification of pulmonary peri-fissural nodules in computed tomography using an ensemble of 2D views and a convolutional neural network out-of-the-box. Med Image Anal 26(1):195–202

    Article  Google Scholar 

  12. Hua KL, Hsu CH, Hidayati SC, Cheng WH, Chen YJ (2015) Computer-aided classification of lung nodules on computed tomography images via deep learning technique. Onco Targets Ther 8

    Google Scholar 

  13. Shen W, Zhou M, Yang F, Yang C, Tian J (2015) Multi-scale convolutional neural networks for lung nodule classification. In: Ourselin S, Alexander DC, Westin CF, Cardoso MJ (eds) Information processing in medical imaging. Lecture notes in computer science, vol 9123. Springer International Publishing, pp 588–599. doi:10.1007/978-3-319-19992-4_46

  14. Roth H, Lu L, Liu J, Yao J, Seff A, Cherry K, Kim L, Summers R (2015) Improving computer-aided detection using convolutional neural networks and random view aggregation. IEEE Trans Med Imaging

    Google Scholar 

  15. Tajbakhsh N, Gurudu SR, Liang J (2015) Automatic polyp detection in colonoscopy videos using an ensemble of convolutional neural networks. In: 2015 IEEE 12th international symposium on biomedical imaging (ISBI). IEEE, pp 79–83

    Google Scholar 

  16. Tajbakhsh N, Gurudu SR, Liang J (2015) A comprehensive computer-aided polyp detection system for colonoscopy videos. In: Information processing in medical imaging. Springer, Berlin, pp 327–338

    Google Scholar 

  17. Tajbakhsh N, Gotway MB, Liang J (2015) Computer-aided pulmonary embolism detection using a novel vessel-aligned multi-planar image representation and convolutional neural networks. In: Medical image computing and computer-assisted intervention MICCAI 2015

    Google Scholar 

  18. Roth H, Lu L, Seff A, Cherry K, Hoffman J, Wang S, Liu J, Turkbey E, Summers R (2014) A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations. In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical image computing and computer-assisted intervention MICCAI 2014. Lecture notes in computer science, vol 8673. Springer International Publishing, Berlin, pp 520–527. doi:10.1007/978-3-319-10404-1_65

  19. Chen T, Chefdhotel C (2014) Deep learning based automatic immune cell detection for immunohistochemistry images. In: Machine learning in medical imaging. Springer, Berlin, pp 17–24

    Google Scholar 

  20. Cireşan DC, Giusti A, Gambardella LM, Schmidhuber J (2013) Mitosis detection in breast cancer histology images with deep neural networks. In: Medical image computing and computer-assisted intervention–MICCAI 2013. Springer, Berlin, pp 411–418

    Google Scholar 

  21. Prasoon A, Petersen K, Igel C, Lauze F, Dam E, Nielsen M (2013) Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. In: Medical image computing and computer-assisted intervention–MICCAI 2013. Springer, Berlin, pp 246–253

    Google Scholar 

  22. Roth HR, Farag A, Lu L, Turkbey EB, Summers RM (2015) Deep convolutional networks for pancreas segmentation in CT imaging. In: SPIE medical imaging. International Society for Optics and Photonics, pp 94,131G–94,131G

    Google Scholar 

  23. Havaei M, Davy A, Warde-Farley D, Biard A, Courville A, Bengio Y, Pal C, Jodoin PM, Larochelle H (2015) Brain tumor segmentation with deep neural networks. arXiv preprint arXiv:1505.03540

  24. Kleesiek J, Urban G, Hubert A, Schwarz D, Maier-Hein K, Bendszus M, Biller A (2016) Deep MRI brain extraction: a 3D convolutional neural network for skull stripping. NeuroImage 129:460

    Article  Google Scholar 

  25. Zhang W, Li R, Deng H, Wang L, Lin W, Ji S, Shen D (2015) Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. NeuroImage 108:214–224

    Article  Google Scholar 

  26. Xu J, Luo X, Wang G, Gilmore H, Madabhushi A (2016) A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing 191:214

    Google Scholar 

  27. Margeta J, Criminisi A, Cabrera Lozoya R, Lee DC, Ayache N (2015) Fine-tuned convolutional neural nets for cardiac MRI acquisition plane recognition. Comput Methods Biomech Biomed Eng Imaging Vis 1–11

    Google Scholar 

  28. Miao S, Wang WJ, Liao R (2016) A CNN regression approach for real-time 2D/3D registration. IEEE Trans Med Imaging PP(99):1–1. doi:10.1109/TMI.2016.2521800

    Google Scholar 

  29. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093

  30. Tajbakhsh N, Gurudu SR, Liang J (2016) Automated polyp detection in colonoscopy videos using shape and context information. IEEE Trans Med Imaging 35(2):630–644

    Article  Google Scholar 

  31. Liang J, Bi J (2007) Computer aided detection of pulmonary embolism with tobogganing and multiple instance classification in CT pulmonary angiography. In: Information processing in medical imaging. Springer, Berlin, pp 630–641

    Google Scholar 

  32. Tajbakhsh N, Chi C, Sharma H, Wu Q, Gurudu SR, Liang J (2014) Automatic assessment of image informativeness in colonoscopy. In: Abdominal imaging. Computational and clinical applications. Springer, Berlin, pp 151–158

    Google Scholar 

  33. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  34. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2014) Going deeper with convolutions. arXiv preprint arXiv:1409.4842

Download references

Acknowledgements

This research has been supported by NIH (R01HL128785) and ASU-Mayo seed program (pulmonary embolism); Mayo discovery translation program (carotid intima-media thickness); and ASU-Mayo seed program (colonoscopy). The content is solely the responsibility of the authors and does not necessarily represent the official views of NIH or ASU-Mayo funding programs.

Author information

Authors and Affiliations

  1. Department of Biomedical Informatics, Arizona State University, 13212 E. Shea Blvd., Scottsdale, AZ, 85259, USA

    Nima Tajbakhsh, Jae Y. Shin & Jianming Liang

  2. Division of Gastroenterology and Hepatology, Mayo Clinic, 13400 E. Shea Blvd., Scottsdale, AZ, 85259, USA

    Suryakanth R. Gurudu

  3. Division of Cardiovascular Diseases, Mayo Clinic, 13400 E. Shea Blvd., Scottsdale, AZ, 85259, USA

    R. Todd Hurst & Christopher B. Kendall

  4. Department of Radiology, Mayo Clinic, 13400 E. Shea Blvd., Scottsdale, AZ, 85259, USA

    Michael B. Gotway

Authors
  1. Nima Tajbakhsh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jae Y. Shin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Suryakanth R. Gurudu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. R. Todd Hurst
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christopher B. Kendall
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Michael B. Gotway
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jianming Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nima Tajbakhsh .

Editor information

Editors and Affiliations

  1. NIH Clinical Center, Bethesda, Maryland, USA

    Le Lu

  2. Siemens Healthcare Technology Center, Princeton, New Jersey, USA

    Yefeng Zheng

  3. University of Adelaide, Adelaide, South Australia, Australia

    Gustavo Carneiro

  4. University of Florida, Gainesville, Florida, USA

    Lin Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Tajbakhsh, N. et al. (2017). On the Necessity of Fine-Tuned Convolutional Neural Networks for Medical Imaging. In: Lu, L., Zheng, Y., Carneiro, G., Yang, L. (eds) Deep Learning and Convolutional Neural Networks for Medical Image Computing. Advances in Computer Vision and Pattern Recognition. Springer, Cham. https://doi.org/10.1007/978-3-319-42999-1_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-42999-1_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-42998-4

  • Online ISBN: 978-3-319-42999-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics