Skip to main content
SpringerLink
Log in

CR-Conformer: a fusion network for clinical skin lesion classification

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Deep convolutional neural network (DCNN) models have been widely used to diagnose skin lesions, and some of them have achieved diagnostic results comparable to or even better than dermatologists. Most publicly available skin lesion datasets used to train DCNN were dermoscopic images. Expensive dermoscopic equipment is rarely available in rural clinics or small hospitals in remote areas. Therefore, it is of great significance to rely on clinical images for computer-aided diagnosis of skin lesions. This paper proposes an improved dual-branch fusion network called CR-Conformer. It integrates a DCNN branch that can effectively extract local features and a Transformer branch that can extract global features to capture more valuable features in clinical skin lesion images. In addition, we improved the DCNN branch to extract enhanced features in four directions through the convolutional rotation operation, further improving the classification performance of clinical skin lesion images. To verify the effectiveness of our proposed method, we conducted comprehensive tests on a private dataset named XJUSL, which contains ten types of clinical skin lesions. The test results indicate that our proposed method reduced the number of parameters by 11.17 M and improved the accuracy of clinical skin lesion image classification by 1.08%. It has the potential to realize automatic diagnosis of skin lesions in mobile devices.

Graphical Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Yue G, Wei P, Zhou T et al (2022) Toward multicenter skin lesion classification using deep neural network with adaptively weighted balance loss. IEEE Trans Med Imaging 42(1):119–131

    Article  PubMed  Google Scholar 

  2. Vestergaard M, Macaskill P, Holt P, Menzies S (2008) Dermoscopy compared with naked eye examination for the diagnosis of primary melanoma: a meta-analysis of studies performed in a clinical setting. Br J Dermatol 159(3):669–676

    PubMed  CAS  Google Scholar 

  3. Feng H, Berk-Krauss J, Feng PW, Stein JA (2018) Comparison of dermatologist density between urban and rural counties in the United States. JAMA Dermatol 154(11):1265–1271

    Article  PubMed  PubMed Central  Google Scholar 

  4. Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M et al (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  PubMed  Google Scholar 

  5. Adegun A, Viriri S (2021) Deep learning techniques for skin lesion analysis and melanoma cancer detection: a survey of state-of-the-art. Artif Intell Rev 54(2):811–841

    Article  Google Scholar 

  6. Yang J, Sun X, Liang J, and Rosin P L (2018) Clinical skin lesion diagnosis using representations inspired by dermatologist criteria[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1258–1266

  7. Satheesha T, Satyanarayana F, Prasad MG, Dhruve KD (2017) Melanoma is skin deep: a 3D reconstruction technique for computerized dermoscopic skin lesion classification. IEEE J Transl Eng Health Med 5:1–17

    Article  Google Scholar 

  8. Gessert N, Sentker T, Madesta F, Schmitz R, Kniep H, Baltruschat I et al (2019) Skin lesion classification using CNNs with patch-based attention and diagnosis-guided loss weighting. IEEE Trans Biomed Eng 67(2):495–503

    Article  PubMed  Google Scholar 

  9. Xie Y, Zhang J, Xia Y, Shen C (2020) A mutual bootstrapping model for automated skin lesion segmentation and classification. IEEE Trans Med Imaging 39(7):2482–2493

    Article  PubMed  Google Scholar 

  10. Yuan Y, Chao M, Lo YC (2017) Automatic skin lesion segmentation using deep fully convolutional networks with Jaccard distance. IEEE Trans Med Imaging 36(9):1876–1886

    Article  PubMed  Google Scholar 

  11. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM et al (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639):115–118

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  12. Liu Y, Jain A, Eng C, Way DH, Lee K, Bui P et al (2020) A deep learning system for differential diagnosis of skin diseases[J]. Nat Med 26(6):900–908

    Article  PubMed  CAS  Google Scholar 

  13. Pan SJ, Yang Q (2009) A survey on transfer learning[J]. IEEE Trans Knowl Data Eng 22(10):1345–1359

    Article  Google Scholar 

  14. Kawahara J, Hamarneh G (2016) Multi-resolution-tract CNN with hybrid pretrained and skin-lesion trained layers[C]//International workshop on machine learning in medical imaging. Springer, Cham, pp 164–171

    Google Scholar 

  15. Simonyan K, and Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  16. He K, Zhang X, Ren S, and Sun J (2016) Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778

  17. Hu J, Shen L, and Sun G (2018) Squeeze-and-excitation networks[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 7132–7141

  18. Zhang J, Xie Y, Xia Y, Shen C (2019) Attention residual learning for skin lesion classification. IEEE Trans Med Imaging 38(9):2092–2103

    Article  PubMed  Google Scholar 

  19. Wei Z, Li Q, Song H (2022) Dual attention based network for skin lesion classification with auxiliary learning. Biomed Signal Process Control 74:103549

    Article  Google Scholar 

  20. Yap J, Yolland W, Tschandl P (2018) Multimodal skin lesion classification using deep learning. Exp Dermatol 27(11):1261–1267

    Article  PubMed  Google Scholar 

  21. Kawahara J, Daneshvar S, Argenziano G et al (2018) Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE J Biomed Health Inform 23(2):538–546

    Article  Google Scholar 

  22. Bi L, Feng DD, Fulham M et al (2020) Multi-label classification of multi-modality skin lesion via hyper-connected convolutional neural network. Pattern Recogn 107:107502

    Article  Google Scholar 

  23. Ge Z,Demyanov S, Chakravorty R, et al. (2017) Skin disease recognition using deep saliency features and multimodal learning of dermoscopy and clinical images[C]//Medical Image Computing and Computer Assisted Intervention− MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III 20. Springer International Publishing 250-258

  24. Wang Y, Feng Y, Zhang L et al (2022) Adversarial multimodal fusion with attention mechanism for skin lesion classification using clinical and dermoscopic images. Med Image Anal 81:102535

    Article  PubMed  Google Scholar 

  25. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, et al. (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

  26. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, et al. (2021) Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 10012–10022

  27. Wang W, Xie E, Li X, Fan D P, Song K, Liang D, et al. Pyramid vision transformer: a versatile backbone for dense prediction without convolutions[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 568–578

  28. Yue G, Han W, Jiang B et al (2022) Boundary constraint network with cross layer feature integration for polyp segmentation. IEEE J Biomed Health Inform 26(8):4090–4099

    Article  PubMed  Google Scholar 

  29. Yue G, Li S, Cong R et al (2023) Attention-guided pyramid context network for polyp segmentation in colonoscopy images. IEEE Trans Instrum Meas 72:1–13

    Google Scholar 

  30. Lei H, Liu W, Xie H et al (2021) Unsupervised domain adaptation based image synthesis and feature alignment for joint optic disc and cup segmentation. IEEE J Biomed Health Inform 26(1):90–102

    Article  Google Scholar 

  31. Wang L, Zhang L, Shu X et al (2023) Intra-class consistency and inter-class discrimination feature learning for automatic skin lesion classification. Med Image Anal 85:102746

    Article  PubMed  Google Scholar 

  32. Nakai K, Chen YW, Han XH (2022) Enhanced deep bottleneck transformer model for skin lesion classification. Biomed Signal Process Control 78:103997

    Article  Google Scholar 

  33. MMCV Contributors. MMCV: OpenMMLab computer vision foundation. https://github.com/open-mmlab/mmcv. Accessed Oct 14 2022

  34. Selvaraju R R, Cogswell M, Das A, Vedantam R, Parikh D, and Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization[C]//Proceedings of the IEEE international conference on computer vision. 618–626

Download references

Funding

This work is partially supported by Xinjiang Uygur Autonomous Region Key R & D program under Grant 2021B03001-4 and National Natural Science Foundation of China 62362061.

Author information

Authors and Affiliations

  1. Department of Dermatology and Venereology, People’s Hospital of Xinjiang Uygur Autonomous Region, Urumqi, 830000, China

    Dezhi Zhang, Weidong Wu & Xiaojing Kang

  2. Xinjiang Clinical Research Center for Dermatologic Diseases, Urumqi, China

    Dezhi Zhang, Weidong Wu & Xiaojing Kang

  3. Xinjiang Key Laboratory of Dermatology Research (XJYS1707), Urumqi, China

    Dezhi Zhang, Weidong Wu & Xiaojing Kang

  4. School of Information Science and Engineering, Xinjiang University, Urumqi, China

    Aolun Li, Long Yu & Xiangzuo Huo

Authors
  1. Dezhi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aolun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weidong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Long Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaojing Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiangzuo Huo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weidong Wu.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Dezhi Zhang and Aolun Li have contributed equally to this work, should be regarded as co-first authorship.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, D., Li, A., Wu, W. et al. CR-Conformer: a fusion network for clinical skin lesion classification. Med Biol Eng Comput 62, 85–94 (2024). https://doi.org/10.1007/s11517-023-02904-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-023-02904-0

Keywords

Search

Navigation