Skip to main content

Advertisement

Log in

Performing Melanoma Diagnosis by an Effective Multi-view Convolutional Network Architecture

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Despite the proved effectiveness of deep learning models in solving complex problems, the melanoma diagnosis remains as a challenging task mainly due to the high levels of inter and intra-class variability present in images of moles. Aimed at filling this gap, in this work we propose a novel architecture for melanoma diagnosis which is inspired on multi-view learning and data augmentation. In order to make the model more transformation-invariant, the proposed architecture creates different independent and specific views of an image. The transformations applied on the original images are learned by genetic algorithms which find the best set of transformations. Also, the final predictions are yielded by aggregating information that comes from independent views. To evaluate the suitability of the proposal, an extensive experimental study on fifteen public melanoma-image datasets was conducted. The impact of the parameters of the genetic algorithms on the proposed architecture were analyzed, and it was also demonstrated that the proposed architecture attained better results when using transformations derived from the genetic algorithm than using random transformations. Finally, the results showed the suitability of the proposed model, where four state-of-the-art data augmentation techniques were significantly outperformed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. https://www.isic-archive.com.

  2. https://www.fc.up.pt/addi/ph2%20database.html.

  3. http://www.cs.rug.nl/imaging/databases/melanoma_naevi/.

  4. http://derm.cs.sfu.ca.

  5. https://bit.ly/2TdZWQ6.

  6. https://mxnet.incubator.apache.org/.

References

  • Abbasi, N. R., et al. (2004). Early diagnosis of cutaneous melanoma: Revisiting the ABCD criteria. Journal of the American Medical Association, 292(22), 2771–2776.

    Article  Google Scholar 

  • Asif, U., et al. (2018). A multi-modal, discriminative and spatially invariant CNN for RGB-D object labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(9), 2051–2065.

    Article  Google Scholar 

  • Bäck, T. (1996). Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. Oxford University Press Inc.

  • Baur et al., C. (2018). MelanoGANs: High resolution skin lesion synthesis with GANs. arXiv:1804.04338

  • Boughorbel, S., Jarray, F., & El-Anbari, M. (2017). Optimal classifier for imbalanced data using Matthews correlation coefficient metric. PLoS ONE, 12(6), e0177678.

    Article  Google Scholar 

  • Cao, Y., et al. (2015). Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1), 54–66.

    Article  MathSciNet  Google Scholar 

  • Carneiro, G., et al. (2015). Unregistered multiview mammogram analysis with pre-trained deep learning models, Vol. 9351.

  • Chicco, D., & Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21(1), 6.

    Article  Google Scholar 

  • Chicco, D., Tötsch, N., & Jurman, G. (2021). The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData Mining, 14(1), 13.

    Article  Google Scholar 

  • Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. In Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR-2017) (pp. 1800–1807). Honolulu, HI, USA.

  • Ciresan, D. C., et al. (2010). Deep, big, simple neural nets for handwritten digit recognition. Neural Computation, 22(12), 3207–3220.

    Article  Google Scholar 

  • Codella, N. C. F., et al. (2018). Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC-2018). In Proceedings of the International Symposium on Biomedical Imaging, Vol. 2018-April (pp. 168–172). Washington, USA.

  • Combalia, M., et al. (2019). BCN20000: Dermoscopic Lesions in the Wild. arxiv:1908.02288

  • Cubuk, E. D., et al. (2019). Practical data augmentation with no separate search. arXiv:1909.13719

  • Deb, K. (1996). Genetic algorithms for function optimisation. Genetic Algorithms and Soft Computing, 8, 4–31.

    Google Scholar 

  • Defays, D. (1977). An efficient algorithm for a complete link method. The Computer Journal, 20(4), 364–366.

    Article  MathSciNet  MATH  Google Scholar 

  • Demyanov, S., et al. (2016). Classification of dermoscopy patterns using deep convolutional neural networks. In IEEE 13th International Symposium on Biomedical Imaging (ISBI) (pp. 364–368). Prague, Czech Republic.

  • Dietterich, T. (2000). Ensemble methods in machine learning, Vol. 1857 LNCS.

  • Dolata, P., et al. (2017). Double-stream convolutional neural networks for machine vision inspection of natural products. Applied Artificial Intelligence, 31(7–8), 643–659.

    Article  Google Scholar 

  • Drown, D., et al. (2007). Using evolutionary sampling to mine imbalanced data. In Sixth International Conference on Machine Learning and Applications (ICMLA 2007) (pp. 363–368). IEEE, Ohio, USA.

  • Drown, D. J., et al. (2009). Evolutionary sampling and software quality modeling of high-assurance systems. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 39(5), 1097–1107.

    Article  Google Scholar 

  • Ericsson. (2015). On the pulse of the networked society. Tech. rep. https://apo.org.au/node/59109.

  • Eshelman, L. J., & Schaffer, J. D. (1993). Real-coded genetic algorithms and interval-schemata. In Foundations of genetic algorithms (Vol. 2, pp. 187–202). Elsevier.

  • Esteva, A., et al. (2017). Dermatologist-level classification of skin cancer with deep neural networks. Nature, 542(7639), 115–118.

    Article  Google Scholar 

  • Felzenszwalb, P. F., et al. (2010). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9), 1627–1645.

    Article  Google Scholar 

  • Friedman, M. (1940). A comparison of alternative tests of significance for the problem of \(m\) rankings. The Annals of Mathematical Statistics, 11(1), 86–92.

    Article  MathSciNet  MATH  Google Scholar 

  • Geller, A. C., et al. (2007). Screening, early detection, and trends for melanoma: Current status (2000–2006) and future directions. Journal of the American Academy of Dermatology, 57(4), 555–572.

    Article  Google Scholar 

  • Gessert, N., Nielsen, M., Shaikh, M., Werner, R., & Schlaefer, A. (2019). Skin lesion classification using loss balancing and ensembles of multi-resolution efficientnets. línea], ISIC Challenge.

  • Giotis, I., et al. (2015). Med-node: A computer-assisted melanoma diagnosis system using non-dermoscopic images. Expert Systems with Applications, 42(19), 6578–6585.

    Article  Google Scholar 

  • Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (pp. 249–256). Sardinia, Italy.

  • Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimization and Machine Learning (1st ed.). Addison-Wesley Longman Publishing Co. Inc.

    MATH  Google Scholar 

  • Goodfellow , I., et al. (2016). Deep learning. MIT Press. http://www.deeplearningbook.org

  • Goodfellow, I., et al. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672–2680). Montreal, Canada.

  • Goodfellow, I., et al. (2016). Deep learning (Vol. 1). MIT Press.

    MATH  Google Scholar 

  • Gutman, D., et al. (2016). Skin lesion analysis toward melanoma detection: A challenge at the international symposium on biomedical imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC). arXiv:1605.01397

  • Haenssle, H., et al. (2018). Man against machine: Diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists. Annals of Oncology, 29(8), 1836–1842.

    Article  Google Scholar 

  • Harangi, B., et al. (2018). Classification of skin lesions using an ensemble of deep neural networks. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, (Vol. 2018-July, pp. 2575–2578). Honolulu, HI, USA.

  • Haritha, K., et al. (2017). Image fusion using evolutionary algorithms: A survey. In 4th International Conference on Advanced Computing and Communication Systems (ICACCS) (pp. 1–7). IEEE, Coimbatore, India.

  • He, K., et al. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 770–778). Las Vegas, NV, USA.

  • Hinton, G., et al. (2012). Rmsprop: Divide the gradient by a running average of its recent magnitude. Neural networks for machine learning, Coursera lecture 6e.

  • Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika, 75(2), 383–386.

    Article  MATH  Google Scholar 

  • Hossain, M. S., & Muhammad, G. (2019). Emotion recognition using deep learning approach from audio-visual emotional big data. Information Fusion, 49, 69–78.

    Article  Google Scholar 

  • Howard , A. G,. et al. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861

  • Hu, Z., et al. (2018). Deep learning for image-based cancer detection and diagnosis—A survey. Pattern Recognition, 83, 134–149.

    Article  Google Scholar 

  • Huang, G., et al. (2017). Densely connected convolutional networks. In Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. Honolulu, HI, USA.

  • Jin, L. et al. (2015). Hand-crafted features or machine learnt features? together they improve RGB-D object recognition. In Proceedings of the IEEE International Symposium on Multimedia (ISM-2014) (pp. 311–319). Taichung, Taiwan.

  • Kawahara, J., et al. (2019). Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE Journal of Biomedical and Health Informatics, 23(2), 538–546.

    Article  Google Scholar 

  • Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization (p. 15). arXiv:1412.6980

  • Krizhevsky, A., et al. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (Vol. 2, pp. 1097–1105). Harrahs and Harveys, Lake Tahoe, NV, USA.

  • Lee, C. Y., et al. (2015). Deeply-supervised nets. In Artificial intelligence and statistics (pp. 562–570). San Diego, California, USA.

  • Lee, H. D., et al. (2018). Dermoscopic assisted diagnosis in melanoma: Reviewing results, optimizing methodologies and quantifying empirical guidelines. Knowledge-Based Systems, 158, 9–24.

    Article  Google Scholar 

  • Lenc, K., & Vedaldi, A. (2019). Understanding image representations by measuring their equivariance and equivalence. International Journal of Computer Vision, 127(5), 456–476.

    Article  MathSciNet  MATH  Google Scholar 

  • Li, X., et al. (2018). Deeply supervised rotation equivariant network for lesion segmentation in dermoscopy images. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 11041 LNCS, 235–243.

  • Liu, X., et al. (2018). Proceedings of the interpretable deep convolutional neural networks via meta-learning. In Proceedings of the international joint conference on neural networks (Vol. 2018). Rio de Janeiro, Brazil.

  • Mahbod, A., et al. (2019). Fusing fine-tuned deep features for skin lesion classification. Computerized Medical Imaging and Graphics, 71, 19–29.

    Article  Google Scholar 

  • Matsunaga, K., Hamada, A., Minagawa, A., & Koga, H. (2017). Image classification of melanoma, nevus and seborrheic keratosis by deep neural network ensemble. arXiv:1703.03108

  • Mendonca, T., et al. (2013). Ph2 - a dermoscopic image database for research and benchmarking. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 5437–5440). Osaka, Japan.

  • Menegola, A., et al. (2017). RECOD Titans at ISIC Challenge 2017.

  • Miikkulainen, R., et al. (2019). Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing (pp. 293–312). Elsevier.

  • Mikolajczyk, K., & Schmid, C. (2005). A performance evaluation of local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(10), 1615–1630.

    Article  Google Scholar 

  • Miller, K. D., et al. (2019). Cancer treatment and survivorship statistics. CA Cancer Journal for Clinicians, 2019, 69(5), 363–385.

    Article  Google Scholar 

  • Nasr-Esfahani, E., et al. (2016). Melanoma detection by analysis of clinical images using convolutional neural network. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS (pp. 1373–1376). Florida, USA.

  • Nozdryn-Plotnicki, A., Yap, J., & Yolland, W. (2018). Ensembling convolutional neural networks for skin cancer classification. International Skin Imaging Collaboration (ISIC) Challenge on Skin Image Analysis for Melanoma Detection. MICCAI.

  • Perez, et al. (2018). Data augmentation for skin lesion analysis. In OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis (pp. 303–311). Springer.

  • Pérez et al., E. (2021). Convolutional neural networks for the automatic diagnosis of melanoma: An extensive experimental study. Medical Image Analysis, 67.

  • Perez, L., & Wang, J. (2017). The effectiveness of data augmentation in image classification using deep learning. arXiv:1712.04621

  • Pérez, E., Reyes, O., & Ventura, S. (2021). Convolutional neural networks for the automatic diagnosis of melanoma: An extensive experimental study. Medical Image Analysis, 67, 101858.

    Article  Google Scholar 

  • Reyes, O., et al. (2018). An ensemble-based method for the selection of instances in the multi-target regression problem. Integrated Computer-Aided Engineering, 25(4), 305–320.

    Article  Google Scholar 

  • Reyes, O., & Ventura, S. (2019). Performing multi-target regression via a parameter sharing-based deep network. International Journal of Neural Systems, 1950014(09), 1950014.

    Article  Google Scholar 

  • Rohlf, F. J. (1977) Computational efficiency of agglomerative clustering algorithms. IBM Research Report RC 6831.

  • Rokach, L. (2010). Ensemble-based classifiers. Artificial Intelligence Review, 33(1–2), 1–39.

    Article  Google Scholar 

  • Rothe, R., et al. (2018). Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision, 126(2–4), 144–157.

    Article  MathSciNet  Google Scholar 

  • Rousseeuw, P. J. (1987). Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20, 53–65.

    Article  MATH  Google Scholar 

  • Sabour, S., et al. (2017). Dynamic routing between capsules.

  • Sennrich, R., et al. (2015). Improving neural machine translation models with monolingual data. arXiv:1511.06709

  • Setio, A., et al. (2016). Pulmonary nodule detection in CT images: False positive reduction using multi-view convolutional networks. IEEE Transactions on Medical Imaging, 35(5), 1160–1169.

    Article  Google Scholar 

  • Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on Image Data Augmentation for Deep Learning. Journal of Big Data, 6(1).

  • Siegel, R. L., et al. (2019). Cancer statistics, 2019. CA Cancer Journal for Clinicians, 69, 7–34.

    Article  Google Scholar 

  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  • Sohn, K., & Lee, H. (2012). Learning invariant representations with local transformations. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012 (Vol. 2, pp. 1311–1318). Edinburgh, Scotland.

  • Sun, X., et al. (2016). A benchmark for automatic visual classification of clinical skin disease images. In European Conference on Computer Vision (pp. 206–222). Springer.

  • Szegedy, C., et al. (2015). Going deeper with convolutions. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Vol. 07-12-June-2015, pp. 1–9). Boston, Massachusetts, USA.

  • Szegedy, C., et al. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2818–2826). Las Vegas, NV, USA.

  • Tschandl, P., Rosendahl, C., & Kittler, H. (2018). The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data5.

  • Wang, J., & Perez, L. (2017). The effectiveness of data augmentation in image classification using deep learning. arXiv:1712.04621.

  • Wang, Z., Bovik, A., Sheikh, H., & Simoncelli, E. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612. https://doi.org/10.1109/TIP.2003.819861

  • Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics, 1(6), 80–83.

    Article  MathSciNet  Google Scholar 

  • Xie, Q., et al. (2019). Unsupervised Data Augmentation for Consistency Training. arXiv:1904.12848

  • Xie, S., & Tu, Z. (2017). Holistically-nested edge detection. International Journal of Computer Vision, 125(1–3), 3–18.

    Article  MathSciNet  Google Scholar 

  • Xu, Z., et al. (2016). Association discovery and diagnosis of Alzheimer’s disease with Bayesian multiview learning. Journal of Artificial Intelligence Research, 56, 247–268. https://doi.org/10.1613/jair.4956

    Article  MathSciNet  MATH  Google Scholar 

  • Yu, L., et al. (2017). Automated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks. IEEE Transactions on Medical Imaging, 36(4), 994–1004.

    Article  Google Scholar 

  • Zhang, R., Isola, P., Efros, A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. pp. 586–595. https://doi.org/10.1109/CVPR.2018.00068

  • Zhao, J., et al. (2017). Multi-view learning overview: Recent progress and new challenges. Information Fusion, 38, 43–54.

    Article  Google Scholar 

  • Zhong, Z., et al. (2017). Random erasing data augmentation. arXiv:1708.04896

  • Zoph, B.,et al.(2018). Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 8697–8710). Salt Lake City.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Óscar Reyes.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Communicated by Jan Kybic.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research was supported by the i-PFIS contract no. IFI17/00015 granted by Health Institute Carlos III of Spain. E. Pérez and Ó. Reyes have equally contributed to this work and should be considered joint first author.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pérez, E., Reyes, Ó. Performing Melanoma Diagnosis by an Effective Multi-view Convolutional Network Architecture. Int J Comput Vis 131, 3094–3117 (2023). https://doi.org/10.1007/s11263-023-01848-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-023-01848-0

Keywords

Navigation