Skip to main content
SpringerLink
Log in

Visually Aware Metadata-Guided Supervision for Improved Skin Lesion Classification Using Deep Learning

  • Conference paper
  • First Online:
Ethical and Philosophical Issues in Medical Imaging, Multimodal Learning and Fusion Across Scales for Clinical Decision Support, and Topological Data Analysis for Biomedical Imaging (EPIMI 2022, ML-CDS 2022, TDA4BiomedicalImaging 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13755))

Abstract

Nowadays, skin cancer has become a common disease and is growing worldwide at an increasing rate. Its manual examination by dermatologists demands significant time and cost in terms of instruments. Also, practical diagnosis demands experienced and skilled dermatologists. These challenges show the impracticality of manual diagnosis over the increased rate of skin cancer patients and thus demand robust end-to-end computer-aided diagnosis (CAD) methods. This paper proposes a deep learning-based skin lesion classification approach that utilizes the visual attention-based mechanism over Convolutional Neural Networks (CNNs) to improve visual context. We use the information from skin lesion images and patient demographics to enhance visual attention, which further improves classification. The proposed method accurately classifies deadly melanoma skin cancer for the PAD-UFES-20 dataset, an essential but challenging task. Our proposed approach has been evaluated over multimodel data, i.e., clinical and dermoscopic images, using two publicly available datasets named PAD-UFES-20 and ISIC-2019. During experimentation, our approach surpasses the available state-of-the-art techniques over five commonly used Convolutional Neural Networks (CNNs) architectures which validate its generalizability and applicability in different scenarios. Our approach achieved efficient performance for small datasets like PAD-UFES-20 using a lightweight model (MobileNet), making it suitable for the CAD system. The effectiveness of our method has been shown by various quantitative and qualitative measures, which demonstrate its efficacy in addressing challenging lesion diagnoses. Our source code is publicly available to reproduce the work.

A. Pundhir, A. Agarwal and S. Dadhich—Equal Contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/AnshulPundhir/VisualMetaGuidedSkinLesionClassification.

References

  1. Haenssle, H.A., et al.: Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Ann. Oncol. 29(8), 1836–1842 (2018)

    Article  Google Scholar 

  2. Liu, Y., et al.: A deep learning system for differential diagnosis of skin diseases. Nat. Med. 26(6), 900–908 (2020)

    Article  Google Scholar 

  3. Pacheco, A.G., et al.: PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphones. Data Brief 32, 106221 (2020)

    Article  Google Scholar 

  4. Arroyo, R., Alcantarilla, P.F., Bergasa, L.M., Romera, E.: Fusion and binarization of CNN features for robust topological localization across seasons. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4656–4663. IEEE (2016)

    Google Scholar 

  5. Atrey, P.K., Hossain, M.A., El Saddik, A., Kankanhalli, M.S.: Multimodal fusion for multimedia analysis: a survey. Multimed. Syst. 16(6), 345–379 (2010)

    Article  Google Scholar 

  6. Brinker, T.J., et al.: Deep learning outperformed 136 of 157 dermatologists in a head-to-head dermoscopic melanoma image classification task. Eur. J. Cancer 113, 47–54 (2019)

    Article  Google Scholar 

  7. Brinker, T.J., et al.: Skin cancer classification using convolutional neural networks: systematic review. J. Med. Internet Res. 20(10), e11936 (2018)

    Google Scholar 

  8. Celebi, M.E., Codella, N., Halpern, A.: Dermoscopy image analysis: overview and future directions. IEEE J. Biomed. Health Inform. 23(2), 474–478 (2019)

    Article  Google Scholar 

  9. Codella, N.C., et al.: Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 168–172. IEEE (2018)

    Google Scholar 

  10. Codella, N.C., et al.: Deep learning ensembles for melanoma recognition in dermoscopy images. IBM J. Res. Dev. 61(4/5), 5–1 (2017)

    Google Scholar 

  11. Combalia, M., et al.: BCN20000: dermoscopic lesions in the wild. arXiv preprint arXiv:1908.02288 (2019)

  12. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

    Google Scholar 

  13. Esteva, A., et al.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115–118 (2017)

    Article  Google Scholar 

  14. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  15. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  16. ISIC: Skin lesion analysis towards melanoma detection. International skin imaging collaboration (2019). https://www.isic-archive.com. Accessed 26 Feb 2022

  17. Jetley, S., Lord, N.A., Lee, N., Torr, P.H.: Learn to pay attention. In: International Conference on Learning Representations (2018)

    Google Scholar 

  18. Kharazmi, P., Kalia, S., Lui, H., Wang, Z., Lee, T.: A feature fusion system for basal cell carcinoma detection through data-driven feature learning and patient profile. Skin Res. Technol. 24(2), 256–264 (2018)

    Article  Google Scholar 

  19. Li, W., Zhuang, J., Wang, R., Zhang, J., Zheng, W.S.: Fusing metadata and dermoscopy images for skin disease diagnosis. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 1996–2000. IEEE (2020)

    Google Scholar 

  20. Lin, T.Y., RoyChowdhury, A., Maji, S.: Bilinear CNN models for fine-grained visual recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1449–1457 (2015)

    Google Scholar 

  21. Liu, Y., Chen, X., Cheng, J., Peng, H.: A medical image fusion method based on convolutional neural networks. In: 2017 20th International Conference on Information Fusion (Fusion), pp. 1–7. IEEE (2017)

    Google Scholar 

  22. Pacheco, A.G., Krohling, R.A.: Recent advances in deep learning applied to skin cancer detection. arXiv preprint arXiv:1912.03280 (2019)

  23. Pacheco, A.G., Krohling, R.A.: The impact of patient clinical information on automated skin cancer detection. Comput. Biol. Med. 116, 103545 (2020)

    Google Scholar 

  24. Pacheco, A.G.C., Krohling, R.: An attention-based mechanism to combine images and metadata in deep learning models applied to skin cancer classification. IEEE J. Biomed. Health Inform. 25, 3554–3563 (2021)

    Article  Google Scholar 

  25. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  26. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV 2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)

    Google Scholar 

  27. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  28. Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)

    Google Scholar 

  29. Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data 5(1), 1–9 (2018)

    Article  Google Scholar 

  30. WHO: Ultraviolet (UV) radiation and skin cancer (2022). https://www.who.int/news-room/questions-and-answers/item/radiation-ultraviolet-(uv)-radiation-and-skin-cancer. Accessed on 26.02.2022

  31. Yu, Z., Jiang, X., Zhou, F., Qin, J., Ni, D., Chen, S., Lei, B., Wang, T.: Melanoma recognition in dermoscopy images via aggregated deep convolutional features. IEEE Trans. Biomed. Eng. 66(4), 1006–1016 (2018)

    Article  Google Scholar 

Download references

Acknowledgment

This work was supported by Indian Institute of Technology Roorkee and University Grants Commission (UGC) INDIA with grant number: 190510040512.

Author information

Authors and Affiliations

  1. Indian Institute of Technology, Roorkee, Roorkee, India

    Anshul Pundhir, Ananya Agarwal, Saurabh Dadhich & Balasubramanian Raman

Authors
  1. Anshul Pundhir
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ananya Agarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Saurabh Dadhich
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Balasubramanian Raman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anshul Pundhir .

Editor information

Editors and Affiliations

  1. Université de Rennes 1, Rennes, France

    John S. H. Baxter

  2. Imperial College London, London, UK

    Islem Rekik

  3. Western University, London, ON, Canada

    Roy Eagleson

  4. University of Sydney, Sydney, NSW, Australia

    Luping Zhou

  5. IBM Almaden Research Center, San Jose, CA, USA

    Tanveer Syeda-Mahmood

  6. IBM Almaden Research Center, San Jose, CA, USA

    Hongzhi Wang

  7. University of San Francisco, San Francisco, CA, USA

    Mustafa Hajij

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pundhir, A., Agarwal, A., Dadhich, S., Raman, B. (2022). Visually Aware Metadata-Guided Supervision for Improved Skin Lesion Classification Using Deep Learning. In: Baxter, J.S.H., et al. Ethical and Philosophical Issues in Medical Imaging, Multimodal Learning and Fusion Across Scales for Clinical Decision Support, and Topological Data Analysis for Biomedical Imaging. EPIMI ML-CDS TDA4BiomedicalImaging 2022 2022 2022. Lecture Notes in Computer Science, vol 13755. Springer, Cham. https://doi.org/10.1007/978-3-031-23223-7_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-23223-7_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-23222-0

  • Online ISBN: 978-3-031-23223-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation