Abstract
The ISIC archive is an open dermoscopy dataset containing thousands of images so that new Deep Learning skin classifiers can be trained. ISIC Challenges attract many participants to build a model that will bring the best performance to the ISIC test dataset. The question is whether such a model has consistent behavior in different datasets and other clinical images. In this work, we build and study the performance of a classifier trained in the ISIC 2019 dataset in three different cases: the performance during the cross-validation training process, the performance in the separate ISIC 2019 test dataset, and dermoscopy images taken from the SYGGROS skin disease hospital. The results show a stable performance compared to the metric F1 score for the categories in which there are more than 3000 images in the training dataset. In addition, we identify the factors that make it difficult to transfer and use classifiers from a competitive to a clinical setting.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Skin Cancer Facts & Statistics, https://www.skincancer.org/skin-cancer-information/skin-cancer-facts. Accessed 08 Mar 2019
USCS Data Visualizations. https://gis.cdc.gov/Cancer/USCS/#/AtAGlance/. Accessed 09 Oct 2021
Lucius, M., et al.: Deep neural frameworks improve the accuracy of general practitioners in the classification of pigmented skin lesions. Diagnostics 10, 969 (2020). https://doi.org/10.3390/diagnostics10110969
ISIC Archive, https://www.isic-archive.com/#!/topWithHeader/tightContentTop/about/aboutIsicOverview. Accessed 08 Oct 2021
ISIC Challenge History, https://www.isic-archive.com/#!/topWithHeader/tightContentTop/about/isicChallengesHistory. Accessed 09 Oct 2021
Valle, E., et al.: Data, depth, and design: learning reliable models for skin lesion analysis. Neurocomputing 383, 303–313 (2020). https://doi.org/10.1016/j.neucom.2019.12.003
Lio, P.A., Nghiem, P.: Interactive Atlas of Dermoscopy: Giuseppe Argenziano, H. Peter Soyer, Vincenzo De Giorgio, Domenico Piccolo, Paolo Carli, Mario Delfino, Angela Ferrari, Rainer Hofmann-Wellenhof, Daniela Massi, Giampiero Mazzocchetti, Massimiliano Scalvenzi, and Ingrid H. Wolfpages. J. Am. Acad. Dermatol. 50, 807–808 (2004). https://doi.org/10.1016/j.jaad.2003.07.029.ISBN 88–86457–30–8
Maron, R.C., et al.: A benchmark for neural network robustness in skin cancer classification. Eur. J. Cancer. 155, 191–199 (2021). https://doi.org/10.1016/j.ejca.2021.06.047
Nahata, H., Singh, S.P.: Deep learning solutions for skin cancer detection and diagnosis. In: Jain, V., Chatterjee, J.M. (eds.) Machine Learning with Health Care Perspective. LAIS, vol. 13, pp. 159–182. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-40850-3_8
Tschandl, P., Rosendahl, C., Kittler, H.: The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Sci. Data. 5, 180161 (2018). https://doi.org/10.1038/sdata.2018.161
Zhu, C.-Y., et al.: A deep learning based framework for diagnosing multiple skin diseases in a clinical environment. Front. Med. 8, 626369 (2021). https://doi.org/10.3389/fmed.2021.626369
ISIC 2019, https://challenge2019.isic-archive.com/. Accessed 08 Oct 2021
Module: tf.data | TensorFlow Core v2.7.0, https://www.tensorflow.org/api_docs/python/tf/data. Accessed 03 Feb 2022
Codella, N.C.F., et al.: Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC). ArXiv171005006 Cs. (2017). https://arxiv.org/abs/1710.05006v3
Combalia, M., et al.: BCN20000: Dermoscopic Lesions in the Wild. ArXiv E-Prints. 1908, arXiv:1908.02288 (2019)
Wasikowski, J.: Stratified Group k-Fold Cross-Validation. https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation
Tan, M., Le, Q.V.: EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ArXiv190511946 Cs Stat. (2020). http://arxiv.org/abs/1905.11946
Ha, Q., Liu, B., Liu, F.: Identifying Melanoma Images using EfficientNet Ensemble: Winning Solution to the SIIM-ISIC Melanoma Classification Challenge. ArXiv201005351 Cs. (2020). http://arxiv.org/abs/2010.05351
Gessert, N., Nielsen, M., Shaikh, M., Werner, R., Schlaefer, A.: Skin lesion classification using ensembles of multi-resolution EfficientNets with meta data. MethodsX 7, 100864 (2020). https://doi.org/10.1016/j.mex.2020.100864
Zhou, S., Zhuang, Y., Meng, R.: Multi-Category Skin Lesion Diagnosis Using Dermoscopy Images and Deep CNN Ensembles (2019). https://challenge.isic-archive.com/leaderboards/2019/
Kontogianni, G., Maglogiannis, I.: A review on state-of-the-art computer-based approaches for the early recognition of malignant melanoma. In: Maglogiannis, I., Brahnam, S., Jain, L.C. (eds.) Advanced Computational Intelligence in Healthcare-7. SCI, vol. 891, pp. 81–101. Springer, Heidelberg (2020). https://doi.org/10.1007/978-3-662-61114-2_6
Random-Erasing-tensorflow, https://github.com/uranusx86/Random-Erasing-tensorflow/blob/master/random_erasing.py
Shawn, N.: Shades of Grey
Barata, C., Celebi, M.E., Marques, J.S.: Improving dermoscopy image classification using color constancy. IEEE J. Biomed. Health Inform. 19, 1146–1152 (2015). https://doi.org/10.1109/JBHI.2014.2336473
Delibasis, K., Georgakopoulos, S.V., Tasoulis, S.K., Maglogiannis, I., Plagianakos, V.P.: On image prefiltering for skin lesion characterization utilizing deep transfer learning. In: Iliadis, L., Angelov, P.P., Jayne, C., Pimenidis, E. (eds.) EANN 2020. PINNS, vol. 2, pp. 377–388. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-48791-1_29
Georgakopoulos, S.V., Kottari, K., Delibasis, K., Plagianakos, V.P., Maglogiannis, I.: Detection of malignant melanomas in dermoscopic images using convolutional neural network with transfer learning. In: Boracchi, G., Iliadis, L., Jayne, C., Likas, A. (eds.) EANN 2017. CCIS, vol. 744, pp. 404–414. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-65172-9_34
tfa.optimizers.RectifiedAdam|TensorFlow Addons, https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/RectifiedAdam. Accessed 08 Oct 2021
sklearn.metrics.classification_report, https://scikit-learn/stable/modules/generated/sklearn.metrics.classification_report.html. Accessed 08 Oct 2021
sklearn.metrics.precision_recall_fscore_support. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html. Accessed 08 Oct 2021
ISIC Challenge 2019 Leaderboard - Lesion Diagnosis - Images Only, https://challenge.isic-archive.com/leaderboards/2019/. Accessed 12 Oct 2021
Tziomaka, M., Maglogiannis, I.: Ensembles of deep convolutional neural networks for detecting melanoma in dermoscopy images. In: Nguyen, N.T., Iliadis, L., Maglogiannis, I., Trawiński, B. (eds.) ICCCI 2021. LNCS (LNAI), vol. 12876, pp. 523–535. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88081-1_39
Popescu, D., El-Khatib, M., El-Khatib, H., Ichim, L.: New trends in melanoma detection using neural networks: a systematic review. Sensors 22, 496 (2022). https://doi.org/10.3390/s22020496
Acknowledgments
We would like to thank Alexios Zarras MD and Professor Alexander J. Stratigos MD, at the Department of Dermatology-Venereology, University of Athens Medical School, Andreas Sygros Hospital (Athens, Greece) for providing the SYGGROS Dataset used in this paper, as well as information on the technical specifications and collection procedures of the dataset.
Funding
This work was supported by the National Project TRANSITION – Translating the diagnostic complexity of melanoma into rational therapeutic stratification – Hellenic General Secretariat of Research and Technology, [Τ1ΕΔΚ-01385] co-funded by the European Union.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 IFIP International Federation for Information Processing
About this paper
Cite this paper
Moutselos, K., Maglogiannis, I. (2022). On the Reusability of ISIC Data for Training DL Classifiers Applied on Clinical Skin Images. In: Maglogiannis, I., Iliadis, L., Macintyre, J., Cortez, P. (eds) Artificial Intelligence Applications and Innovations. AIAI 2022 IFIP WG 12.5 International Workshops. AIAI 2022. IFIP Advances in Information and Communication Technology, vol 652. Springer, Cham. https://doi.org/10.1007/978-3-031-08341-9_17
Download citation
DOI: https://doi.org/10.1007/978-3-031-08341-9_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-08340-2
Online ISBN: 978-3-031-08341-9
eBook Packages: Computer ScienceComputer Science (R0)
