Skip to main content
SpringerLink
Log in

BF2SkNet: best deep learning features fusion-assisted framework for multiclass skin lesion classification

  • S.i.: Deep Learning in Multimodal Medical Imaging for Cancer Detection
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The convolutional neural network showed considerable success in medical imaging with explainable AI for cancer detection and recognition. However, the irrelevant and large number of features increases the computational time and decreases the accuracy. This work proposes a deep learning and fuzzy entropy slime mould algorithm-based architecture for multiclass skin lesion classification. In the first step, we employed the data augmentation technique to increase the training data and further utilized it for training two fine-tuned deep learning models such as Inception-ResNetV2 and NasNet Mobile. Then, we used transfer learning on augmented datasets to train both models and obtained two feature vectors from newly fine-tuned models. Later, we applied a fuzzy entropy slime mould algorithm on both vectors to get optimal features that are finally fused using the Serial-Threshold fusion technique and classified using several machine learning classifiers. Eventually, the explainable AI technique named Gradcam opted for the visualization of the lesion region. The experimental process was conducted on two datasets, such as HAM10000 and ISIC 2018, and achieved 97.1 and 90.2% accuracy, better than the other techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The datasets used in this work are publically available: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/DBW86T; https://challenge.isic-archive.com/data/#2018.

References

  1. Spyridonos P, Gaitanis G, Likas A, Bassukas I (2021) Characterizing malignant melanoma clinically resembling seborrheic keratosis using deep knowledge transfer. Cancers 13:6300

    Google Scholar 

  2. Mahbod A, Tschandl P, Langs G, Ecker R, Ellinger I (2020) The effects of skin lesion segmentation on the performance of dermatoscopic image classification. Comput Methods Programs Biomed 197:105725

    Google Scholar 

  3. Lakhan MK, Lynch M (2021) Skin pigmentation. Medicine 49:447–452

    Google Scholar 

  4. Iqbal I, Younus M, Walayat K, Kakar MU, Ma J (2021) Automated multi-class classification of skin lesions through deep convolutional neural network with dermoscopic images. Comput Med Imaging Graph 88:101843

    Google Scholar 

  5. Mahbod A, Schaefer G, Wang C, Dorffner G, Ecker R, Ellinger I (2020) Transfer learning using a multi-scale and multi-network ensemble for skin lesion classification. Comput Methods Programs Biomed 193:105475

    Google Scholar 

  6. Sevli O (2021) A deep convolutional neural network-based pigmented skin lesion classification application and experts evaluation. Neural Comput Appl 33:12039–12050

    Google Scholar 

  7. Garg N, Sharma V, Kaur P (2018) Melanoma skin cancer detection using image processing. In: Sensors and image processing. Springer, pp 111–119

  8. Saladi RN, Persaud AN (2005) The causes of skin cancer: a comprehensive review. Drugs Today 41:37–54

    Google Scholar 

  9. A. C. Society (2021) Melanoma of the skin. Cancer Statistics Centre. https://cancerstatisticscenter.cancer.org/?_ga=2.121293320.1263676073.1630612387-522677707.1629187726#!/cancer-site/Melanoma%20of%20the%20skin

  10. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM et al (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542:115–118

    Google Scholar 

  11. Bi L, Feng DD, Fulham M, Kim J (2020) Multi-label classification of multi-modality skin lesion via hyper-connected convolutional neural network. Pattern Recognit 107:107502

    Google Scholar 

  12. Brinker TJ, Hekler A, Utikal JS, Grabe N, Schadendorf D, Klode J et al (2018) Skin cancer classification using convolutional neural networks: systematic review. J Med Internet Res 20:e11936

    Google Scholar 

  13. Mahbod A, Schaefer G, Wang C, Ecker R, Dorffner G, Ellinger I (2021) Investigating and exploiting image resolution for transfer learning-based skin lesion classification. In: 2020 25th international conference on pattern recognition (ICPR), pp 4047–4053

  14. Codella N, Cai J, Abedini M, Garnavi R, Halpern A, Smith JR (2015) Deep learning, sparse coding, and SVM for melanoma recognition in dermoscopy images. In: International workshop on machine learning in medical imaging, pp 118–126

  15. Kassem MA, Hosny KM, Fouad MM (2020) Skin lesions classification into eight classes for ISIC 2019 using deep convolutional neural network and transfer learning. IEEE Access 8:114822–114832

    Google Scholar 

  16. Tschandl P, Rosendahl C, Kittler H (2018) The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci Data 5:1–9

    Google Scholar 

  17. Slowinska M, Kaminska-Winciorek G, Kowalska-Oledzka E, Czarnecka I, Czarnecki R, Nasierowska-Guttmejer A et al (2021) Dermoscopy of small diameter melanomas with the diagnostic feasibility of selected algorithms—a clinical retrospective multicenter study. Cancers 13:6095

    Google Scholar 

  18. Dildar M, Akram S, Irfan M, Khan HU, Ramzan M, Mahmood AR et al (2021) Skin cancer detection: a review using deep learning techniques. Int J Environ Res Public Health 18:5479

    Google Scholar 

  19. Abbasi NR, Shaw HM, Rigel DS, Friedman RJ, McCarthy WH, Osman I et al (2004) Early diagnosis of cutaneous melanoma: revisiting the ABCD criteria. JAMA 292:2771–2776

    Google Scholar 

  20. Jaworek-Korjakowska J, Brodzicki A, Cassidy B, Kendrick C, Yap MH (2021) Interpretability of a deep learning based approach for the classification of skin lesions into main anatomic body sites. Cancers 13:6048

    Google Scholar 

  21. Oliveira RB, Papa JP, Pereira AS, Tavares JMR (2018) Computational methods for pigmented skin lesion classification in images: review and future trends. Neural Comput Appl 29:613–636

    Google Scholar 

  22. Nishio M, Nishio M, Jimbo N, Nakane K (2021) Homology-based image processing for automatic classification of histopathological images of lung tissue. Cancers 13:1192

    Google Scholar 

  23. Xu Y, Li Y, Shin B-S (2020) Medical image processing with contextual style transfer. HCIS 10:1–16

    Google Scholar 

  24. Lu S-Y, Wang S-H, Zhang Y-D (2022) SAFNet: a deep spatial attention network with classifier fusion for breast cancer detection. Comput Biol Med 148:105812

    Google Scholar 

  25. Bommert A, Sun X, Bischl B, Rahnenführer J, Lang M (2020) Benchmark for filter methods for feature selection in high-dimensional classification data. Comput Stat Data Anal 143:106839

    MathSciNet  MATH  Google Scholar 

  26. Khan MA, Javed MY, Sharif M, Saba T, Rehman A (2019) Multi-model deep neural network based features extraction and optimal selection approach for skin lesion classification. In: 2019 international conference on computer and information sciences (ICCIS), pp 1–7

  27. Karczmarek P, Pedrycz W, Kiersztyn A, Rutka P (2017) A study in facial features saliency in face recognition: an analytic hierarchy process approach. Soft Comput 21:7503–7517

    Google Scholar 

  28. Jinnai S, Yamazaki N, Hirano Y, Sugawara Y, Ohe Y, Hamamoto R (2020) The development of a skin cancer classification system for pigmented skin lesions using deep learning. Biomolecules 10:1123

    Google Scholar 

  29. O’Mahony N, Campbell S, Carvalho A, Harapanahalli S, Hernandez GV, Krpalkova L et al (2019) Deep learning vs. traditional computer vision. In: Science and information conference, pp 128–144

  30. Mridha MF, Hamid M, Monowar MM, Keya AJ, Ohi AQ, Islam M et al (2021) A comprehensive survey on deep-learning-based breast cancer diagnosis. Cancers 13:6116

    Google Scholar 

  31. Bibi S, Abbasi A, Haq IU, Baik SW, Ullah A (2021) Digital image forgery detection using deep autoencoder and CNN features. Hum Cent Comput Inf Sci 11:1–17

    Google Scholar 

  32. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105

    Google Scholar 

  33. Celebi ME, Iyatomi H, Schaefer G, Stoecker WV (2009) Lesion border detection in dermoscopy images. Comput Med Imaging Graph 33:148–153

    Google Scholar 

  34. Mostafa RR, Ewees AA, Ghoniem RM, Abualigah L, Hashim FA (2022) Boosting chameleon swarm algorithm with consumption AEO operator for global optimization and feature selection. Knowl-Based Syst 246:108743

    Google Scholar 

  35. Ewees AA, Mostafa RR, Ghoniem RM, Gaheen MA (2022) Improved seagull optimization algorithm using Lévy flight and mutation operator for feature selection. Neural Comput Appl 34:7437–7472

    Google Scholar 

  36. Adjed F, Gardezi SJS, Ababsa F, Faye I, Dass SC (2018) Fusion of structural and textural features for melanoma recognition. IET Comput Vis 12:185–195

    Google Scholar 

  37. Hagerty JR, Stanley RJ, Almubarak HA, Lama N, Kasmi R, Guo P et al (2019) Deep learning and handcrafted method fusion: higher diagnostic accuracy for melanoma dermoscopy images. IEEE J Biomed Health Inform 23:1385–1391

    Google Scholar 

  38. Codella N, Rotemberg V, Tschandl P, Celebi ME, Dusza S, Gutman D et al (2019) Skin lesion analysis toward melanoma detection 2018: a challenge hosted by the international skin imaging collaboration (ISIC). arXiv preprint arXiv:1902.03368

  39. Abbas Q, Celebi ME (2019) DermoDeep-A classification of melanoma-nevus skin lesions using multi-feature fusion of visual features and deep neural network. Multimedia Tools Appl 78:23559–23580

    Google Scholar 

  40. Harangi B, Baran A, Hajdu A (2020) Assisted deep learning framework for multi-class skin lesion classification considering a binary classification support. Biomed Signal Process Control 62:102041

    Google Scholar 

  41. Xiao F, Wu Q (2020) Visual saliency based global–local feature representation for skin cancer classification. IET Image Proc 14:2140–2148

    Google Scholar 

  42. Qin Z, Liu Z, Zhu P, Xue Y (2020) A GAN-based image synthesis method for skin lesion classification. Comput Methods Programs Biomed 195:105568

    Google Scholar 

  43. Afza F, Sharif M, Saba T, Rehman A, Javed MY (2020) Skin lesion classification: an optimized framework of optimal color features selection. In: 2020 2nd international conference on computer and information sciences (ICCIS), pp 1–6

  44. Huang HW, Hsu BWY, Lee CH, Tseng VS (2020) Development of a light-weight deep learning model for cloud applications and remote diagnosis of skin cancers. J Dermatol 48:310–316

    Google Scholar 

  45. Tang P, Liang Q, Yan X, Xiang S, Zhang D (2020) GP-CNN-DTEL: global-part cnn model with data-transformed ensemble learning for skin lesion classification. IEEE J Biomed Health Inform 24:2870–2882

    Google Scholar 

  46. Liu L, Mou L, Zhu XX, Mandal M (2020) Automatic skin lesion classification based on mid-level feature learning. Comput Med Imaging Graph 84:101765

    Google Scholar 

  47. Sharif M, Akram T, Damaševičius R, Maskeliūnas R (2021) Skin lesion segmentation and multiclass classification using deep learning features and improved moth flame optimization. Diagnostics 11:811

    Google Scholar 

  48. Arshad M, Khan MA, Tariq U, Armghan A, Alenezi F, Younus Javed M et al (2021) A computer-aided diagnosis system using deep learning for multiclass skin lesion classification. Comput Intell Neurosci 2021

  49. Akram T, Zhang Y-D, Sharif M (2021) Attributes based skin lesion detection and recognition: A mask RCNN and transfer learning-based deep learning framework. Pattern Recognit Lett 143:58–66

    Google Scholar 

  50. Sevli O (2021) A deep convolutional neural network-based pigmented skin lesion classification application and experts evaluation. Neural Comput Appl 33:1–12

    Google Scholar 

  51. Sharif MI, Raza M, Anjum A, Saba T, Shad SA (2019) Skin lesion segmentation and classification: a unified framework of deep neural network features fusion and selection. Expert Syst 39:e12497

    Google Scholar 

  52. Mikołajczyk A, Grochowski M (2018) Data augmentation for improving deep learning in image classification problem. In: 2018 international interdisciplinary PhD workshop (IIPhDW), pp 117–122

  53. Zhang Y-D, Sharif M, Akram T (2021) Pixels to classes: intelligent learning framework for multiclass skin lesion localization and classification. Comput Electr Eng 90:106956

    Google Scholar 

  54. Hussain N, Khan MA, Sharif M, Khan SA, Albesher AA, Saba T et al (2020) A deep neural network and classical features based scheme for objects recognition: an application for machine inspection. Multimedia Tools Appl 45:1–23

    Google Scholar 

  55. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958

    MathSciNet  MATH  Google Scholar 

  56. Weiss K, Khoshgoftaar TM, Wang D (2016) A survey of transfer learning. J Big Data 3:1–40

    Google Scholar 

  57. Akram T, Laurent B, Naqvi SR, Alex MM, Muhammad N (2018) A deep heterogeneous feature fusion approach for automatic land-use classification. Inf Sci 467:199–218

    Google Scholar 

  58. Zoph B, Vasudevan V, Shlens J, Le QV (2018) Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8697–8710

  59. Saleem F, Khan MA, Alhaisoni M, Tariq U, Armghan A, Alenezi F et al (2021) Human gait recognition: a single stream optimal deep learning features fusion. Sensors 21:7584

    Google Scholar 

  60. Howard FL (1931) The life history of Physarum polycephalum. Am J Bot 116–133

  61. Camp WG (1936) A method of cultivating myxomycete plasmodia. Bull Torrey Bot Club 205–210

  62. Li S, Chen H, Wang M, Heidari AA, Mirjalili S (2020) Slime mould algorithm: a new method for stochastic optimization. Futur Gener Comput Syst 111:300–323

    Google Scholar 

  63. Muhammad K, Sharif M, Akram T, Kadry S (2021) Intelligent fusion-assisted skin lesion localization and classification for smart healthcare. Neural Comput Appl 78:1–16

    Google Scholar 

  64. Akram T, Sharif M, Kadry S, Nam Y (2021) Computer decision support system for skin cancer localization and classification. Comput Mater Contin 68:1041–1064

    Google Scholar 

  65. Sharif M, Akram T, Kadry S, Hsu CH (2021) A two‐stream deep neural network‐based intelligent system for complex skin cancer types classification. Int J Intell Syst

  66. Deng L, Wang S-H, Zhang Y-D (2022) ELMGAN: A GAN-based efficient lightweight multi-scale-feature-fusion multi-task model. Knowl-Based Syst 252:109434

    Google Scholar 

Download references

Acknowledgements

Authors are thankful to Computer Vision lab of HITEC University Taxila

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, COMSATS University Islamabad, Wah Campus, Rawalpindi, Pakistan

    Muhammad Ajmal & Tallha Akram

  2. Department of Computer Science, HITEC University, Taxila, Pakistan

    Muhammad Attique Khan

  3. College of Computer Engineering and Sciences, Prince Sattam Bin Abdulaziz University, Al-Kharj, Saudi Arabia

    Abdullah Alqahtani

  4. Computer Sciences Department, College of Computer and Information Sciences, Princess Nourah Bint Abdulrahman University, Riyadh, 11671, Saudi Arabia

    Majed Alhaisoni

  5. Department of Electrical Engineering, College of Engineering, Jouf University, Sakakah, Saudi Arabia

    Ammar Armghan & Fayadh Alenezi

  6. Department of Computer Science, College of Computer and Information Sciences, Majmaah University, Al-Majmaah, 11952, Saudi Arabia

    Sara A. Althubiti

Authors
  1. Muhammad Ajmal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Muhammad Attique Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tallha Akram
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Abdullah Alqahtani
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Majed Alhaisoni
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ammar Armghan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Sara A. Althubiti
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Fayadh Alenezi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed equally in this work. All authors have read and agreed to the published version of the manuscript.

Corresponding authors

Correspondence to Muhammad Attique Khan or Ammar Armghan.

Ethics declarations

Conflict of interest

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ajmal, M., Khan, M.A., Akram, T. et al. BF2SkNet: best deep learning features fusion-assisted framework for multiclass skin lesion classification. Neural Comput & Applic 35, 22115–22131 (2023). https://doi.org/10.1007/s00521-022-08084-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-08084-6

Keywords

Search

Navigation