Skip to main content
SpringerLink
Log in

Transfer learning for histopathology images: an empirical study

  • S.I.: TAM-LHR
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Histopathology imaging is one of the key methods used to determine the presence of cancerous cells. However, determining the results from such medical images is a tedious task because of their size, which may cause a delay in results for days. Even though CNNs are widely used to analyze medical images, they can only learn short-term dependency and ignore long-term dependency, which could be crucial in processing higher dimensional histology images. Transformers, however, make use of a self-attention mechanism, which might be helpful to learn dependencies across an entire set of features. To process histology images, deep learning models require a large number of images, which is usually not available. Transfer learning, which is often used to deal with this issue, involves fine-tuning a trained model for use with medical images by adding features. In context, it is essential to analyze which CNNs or transformers are more conducive to transfer learning. In this study, we performed an empirical study to evaluate the performance of different pre-trained deep learning models for the classification of lung and colon cancer on histology images. Vision transformer and CNN models pre-trained on image-net are analyzed for the classification of histopathology images. We performed an experiment on the LC25000 dataset for the evaluation of models. The dataset consists of five classes, two belong to colon and three belong to lung cancer. The insights and observations obtained from an ablation study performed on different pre-trained models show vision transformers perform better than CNN based models for histopathology image classification using transfer learning. Moreover, the vision transformer with more layers of ViT-L32 performs better than ViTB32 with fewer layers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Farmer P, Frenk J, Knaul FM, Shulman LN, Alleyne G, Armstrong L, Atun R, Blayney D, Chen L, Feachem R et al (2010) Expansion of cancer care and control in countries of low and middle income: a call to action. The Lancet 376(9747):1186–1193

    Article  Google Scholar 

  2. Pakistan G (2021) “International Agency on Research for Cancer.” https://gco.iarc.fr/today/data/factsheets/populations/586-pakistan-fact-sheets.pdf/, [Online; accessed 05-October-2021]

  3. Scheel BI, Holtedahl K (2015) Symptoms, signs, and tests: The general practitioner’s comprehensive approach towards a cancer diagnosis. Scandinavian journal of primary health care 33(3):170–177

    Article  Google Scholar 

  4. Whitaker K (2020) Earlier diagnosis: the importance of cancer symptoms. The Lancet Oncology 21(1):6–8

    Article  Google Scholar 

  5. Thakor AS, Gambhir SS (2013)“Nanooncology: the future of cancer diagnosis and therapy,” CA: a cancer journal for clinicians, vol.63, no.6, pp.395–418

  6. Frysh P (2021)“Can You Find Cancer Without a Biopsy?.” https://www.webmd.com/cancer/detect-cancer-without-biopsy, [Online; accessed 05-October-2021]

  7. Santoso JT, Coleman RL, Voet RL, Bernstein SG, Lifshitz S, Miller D (1998) Pathology slide review in gynecologic oncology. Obstetrics & Gynecology 91(5):730–734

    Google Scholar 

  8. Bhatt C, Kumar I, Vijayakumar V, Singh KU, Kumar A (2021) The state of the art of deep learning models in medical science and their challenges. Multimedia Systems 27(4):599–613

    Article  Google Scholar 

  9. Lipton ZC, Berkowitz J, Elkan C (2015)“A critical review of recurrent neural networks for sequence learning,” arXiv preprint arXiv:1506.00019

  10. Lipton ZC, Kale DC, Elkan C, Wetzel R (2015)“Learning to diagnose with lstm recurrent neural networks,” arXiv preprint arXiv:1511.03677

  11. Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classification with deep convolutional neural networks. Communications of the ACM 60(6):84–90

    Article  Google Scholar 

  12. He K, Zhang X, Ren S, Sun J (2016) “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, (Las Vegas, Nevada, USA.), pp.770–778

  13. Redmon J, Divvala S, Girshick R, Farhadi A (2016)“You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, (Las Vegas, Nevada, USA.), pp. 779–788

  14. Ronneberger O, Fischer P, Brox T (2015) “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention, (Munich, Germany), pp. 234–241, Springer

  15. Abbas MA, Bukhari SUK. Syed A, Shah SSH (2020)“The histopathological diagnosis of adenocarcinoma & squamous cells carcinoma of lungs by artificial intelligence: A comparative study of convolutional neural networks,” medRxiv, https://doi.org/10.1101/2020.05.02.20044602

  16. Bukhari SUK, Asmara S, Bokhari SKA, Hussain SS, Armaghan SU, Shah SSH (2020)“The histological diagnosis of colonic adenocarcinoma by applying partial self supervised learning,” medRxiv, https://doi.org/10.1101/2020.08.15.20175760

  17. Simonyan K, Zisserman A (2014)“Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556

  18. Nishio M, Nishio M, Jimbo N, Nakane K (2021) Homology-based image processing for automatic classification of histopathological images of lung tissue. Cancers 13(6):1192

    Article  Google Scholar 

  19. Fan J, Lee J, Lee Y (2021)“A transfer learning architecture based on a support vector machine for histopathology image classification, ” Applied Sciences, vol. 11, no.14, p.6380

  20. Toğaçar M (2021) Disease type detection in lung and colon cancer images using the complement approach of inefficient sets. Computers inBiology and Medicine 137:104827

    Article  Google Scholar 

  21. Raghu M, Zhang C, Kleinberg J, Bengio S (2019) “Transfusion: Understanding transfer learning for medical imaging,” arXiv preprint arXiv:1902.07208

  22. Hekler A, Utikal JS, Enk AH, Solass W, Schmitt M, Klode J, Schadendorf D, Sondermann W, Franklin C, Bestvater F et al (2019) Deep learning outperformed 11 pathologists in the classification of histopathological melanoma images. European Journal of Cancer 118:91–96

    Article  Google Scholar 

  23. Vo DM, Nguyen N-Q, Lee S-W (2019) Classification of breast cancer histology images using incremental boosting convolution networks. Information Sciences 482:123–138

    Article  Google Scholar 

  24. Khan S, Islam N, Jan Z, Din IU, Rodrigues JJC (2019) A novel deep learning based framework for the detection and classification of breast cancer using transfer learning. Pattern Recognition Letters 125:1–6

    Article  Google Scholar 

  25. Gupta K, Chawla N (2020) Analysis of histopathological images for prediction of breast cancer using traditional classifiers with pre-trained cnn. Procedia Computer Science 167:878–889

    Article  Google Scholar 

  26. Chen H, Li C, Li X, Rahaman MM, Hu W, Li Y, Liu W, Sun C, Sun H, Huang X et al (2022) Il-mcam: An interactive learning and multi-channel attention mechanism-based weakly supervised colorectal histopathology image classification approach. Computers in Biology and Medicine 143:105265

    Article  Google Scholar 

  27. Teramoto A, Yamada A, Kiriyama Y, Tsukamoto T, Yan K, Zhang L, Imaizumi K, Saito K, Fujita H (2019) Automated classification of benign and malignant cells from lung cytological images using deep convolutional neural network. Informatics in Medicine Unlocked 16:100205

    Article  Google Scholar 

  28. Wang C, Gong W, Cheng J, Qian Y (2022) Dblcnn: Dependency-based lightweight convolutional neural network for multi-classification of breast histopathology images. Biomedical Signal Processing and Control 73:103451

    Article  Google Scholar 

  29. Masud M, Sikder N, Nahid A-A, Bairagi AK, AlZain MA (2021) A machine learning approach to diagnosing lung and colon cancer using a deep learning-based classification framework. Sensors 21(3):748

    Article  Google Scholar 

  30. Angayarkanni SP (2022)“Hybrid convolution neural network in classification of cancer in histopathology images,” Journal of Digital Imaging, pp.1–10

  31. Adu K, Yu Y, Cai J,Owusu-Agyemang K, Twumasi BA, Wang X (2021)“Dhs-capsnet: Dual horizontal squash capsule networks for lung and colon cancer classification from whole slide histopathological images,” International Journal of Imaging Systems and Technology

  32. Naeem Abid MM, Zia T, Ghafoor M, Windridge D (2021) Multi-view convolutional recurrent neural networks for lung cancer nodule identification. Neurocomputing 453:299–311

    Article  Google Scholar 

  33. Garg S, Garg S (2020)“Prediction of lung and colon cancer through analysis of histopathological images by utilizing pre-trained cnn models with visualization of class activation and saliency maps,” in 2020 3rd Artificial Intelligence and Cloud Computing Conference, (Kyoto, Japan.), pp.38–45

  34. Roy Medhi BB (2020) Lung Cancer Classification from Histologic Images using Capsule Networks. PhD thesis, Dublin, National College of Ireland

  35. Sarwinda D, Bustamam A, Paradisa RH, Argyadiva T, Mangunwardoyo W (2020)“Analysis of deep feature extraction for colorectal cancer detection,” in 2020 4th International Conference on Informatics and Computational Sciences (ICICoS), (Semarang, Indonesia.), pp.1–5, IEEE

  36. Chougrad H, Zouaki H, Alheyane O (2017) “Convolutional neural networks for breast cancer screening: transfer learning with exponential decay,” arXiv preprint arXiv:1711.10752

  37. Kassani SH, Kassani PH, Wesolowski MJ, Schneider KA, Deters R (2019)“Classification of histopathological biopsy images using ensemble of deep learning networks,” arXiv preprint arXiv:1909.11870

  38. Gao X, Qian Y, Gao A (2021)“Covid-vit: Classification of covid-19 from ct chest images based on vision transformer models,” arXiv preprint arXiv:2107.01682

  39. Zhou M, Mo S (2021)“Shoulder implant x-ray manufacturer classification: Exploring with vision transformer,” arXiv preprint arXiv:2104.07667

  40. Shao Z, Bian HY, Chen Wang Y, Zhang J, Ji X, Zhang Y (2021)“Transmil: Transformer based correlated multiple instance learning for whole slide image classication,” arXiv preprint arXiv:2106.00908

  41. Duong LT, Le NH, Tran TB, Ngo VM, Nguyen PT (2021) Detection of tuberculosis from chest x-ray images: Boosting the performance with vision transformer and transfer learning. Expert Systems with Applications 184:115519

    Article  Google Scholar 

  42. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D,Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold, G, Gelly S et al (2020) “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929

  43. Javed S, Mahmood A, Fraz MM, Koohbanani NA, Benes K, Tsang Y-W, Hewitt K, Epstein D, Snead D, Rajpoot N (2020) Cellular community detection for tissue phenotyping in colorectal cancer histology images. Medical Image Analysis 63:101696

    Article  Google Scholar 

  44. Borkowski AA, Bui MM, Thomas LB, Wilson CP, DeLand LA, Mastorides SM (2019) Lung and colon cancer histopathological image dataset (lc25000). arXiv preprint arXiv:1912.12142

Download references

Author information

Authors and Affiliations

  1. COMSATS University Islamabad, Islamabad, Pakistan

    Tayyab Aitazaz & Tehseen Zia

  2. College of Technological Innovation, Zayed University, Abu Dhabi, UAE

    Abdullah Tubaishat, Feras Al-Obeidat & Babar Shah

  3. Medical Imaging and Diagnostic Lab, National Center for Artificial Intelligence, Comsats University, Islamabad, Pakistan

    Ali Tariq

Authors
  1. Tayyab Aitazaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abdullah Tubaishat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Feras Al-Obeidat
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Babar Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tehseen Zia
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ali Tariq
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tehseen Zia.

Ethics declarations

Conflict of interest

Tayyab Aitazaz, Abdullah Tubaishat, Feras Al-Obeidat, Babar Shah, Tehseen Zia and Ali Tariq declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aitazaz, T., Tubaishat, A., Al-Obeidat, F. et al. Transfer learning for histopathology images: an empirical study. Neural Comput & Applic 35, 7963–7974 (2023). https://doi.org/10.1007/s00521-022-07516-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07516-7

Keyword

Search

Navigation