Skip to main content
SpringerLink
Log in

GTADT: Gated tone-sensitive acne grading via augmented domain transfer

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Automatic selfie facial acne grading plays a crucial role in treatment for facial care customers, and it attracts increasing attention with the revolution of telemedicine and virtual beauty. Unfortunately, the limited quantity and quality of the selfie facial dataset greatly challenge the learning of acne grading models. In this paper, we propose a Gated Tone-sensitive Augmented Domain Transfer (GTADT) model to address the selfie facial acne grading problem. A “high-quality” clinical source domain and associated cross-domain data augmentation are introduced to generate sufficient data. Also, an aligned tone-sensitive model with multiple tone subnetworks is devised to bridge the domain gaps. In Addition, two gate networks are devised to capture the correlation between different tone subnetworks in both the label and feature spaces. We establish three selfie facial acne datasets which consist of people across different skin tones, ages, poses, etc. The experimental results on the newly established datasets demonstrate that: 1) both the cross-domain data augmentation, tone-sensitive module, and gate networks can enhance the performance; and 2) the proposed model performs favorably against state-of-the-art methods. We make both the code and datasets publicly available at https://github.com/WRLH/GTADT.git.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availibility Statement

We build three datasets for acne grading, i.e., Acnehdu, AcnehduP, and AcnePGP. The Acnehdu and AcnehduP are publicly available at https://github.com/WRLH/GTADT.git. AcnePGP is currently not publicly available due to the data agreement of Procter & Gamble, but will be available from the corresponding author on reasonable request.

Notes

  1. We have released “Acnehdu” and “AcnehduP” along with the source code, and will release the “AcnePGP” dataset in the future.

  2. The skin detection tool is available at https://github.com/topics/skin-detection.

  3. The source-only MK-MMD loss is applied on Acnehdu and AcnePGP, while shared alignment structure on both domains is used for AcnehduP.

  4. The optimal \(\gamma \) and \(\gamma '\) are selected from [0.1, 0.2, ..., 0.5].

  5. The optimal subnetwork number (\(N_t'\)) and expert number (\(N_e\)) are selected from [1, 2, ..., 5] and [1, 2, 3], respectively.

  6. Original DNN [39] takes three images captured for each patient as an input.

  7. NSH and A*STAR in Table 1 is NOT publicly available at present.

References

  1. Bernardis E, Shou H, Barbieri JS, McMahon PJ, Perman MJ, Rola LA, Streicher JL, Treat JR, Castelo-Soccio L, Yan AC (2020) Development and initial validation of a multidimensional acne global grading system integrating primary lesions and secondary changes. JAMA dermatology 156(3):296–302

    Article  PubMed  PubMed Central  Google Scholar 

  2. Chen YC, Lin YY, Yang MH, Huang JB (2019) Crdoco: Pixel-level domain transfer with cross-domain consistency. In: IEEE Conference on Computer Vision and Pattern Recognition 1791–1800

  3. Chen H, Guo T, Xu C, Li W, Xu C, Xu C, Wang Y (2021) Learning student networks in the wild. In: IEEE Conference on Computer Vision and Pattern Recognition 6424–6433

  4. Ganeshkumar M, Ravi V, Sowmya V, Gopalakrishnan E, Soman K, Chakraborty C (2022) Identification of intracranial haemorrhage (ich) using resnet with data augmentation using cyclegan and ich segmentation using segan. Multimedia Tools and Applications 81(25):36257–36273

    Article  Google Scholar 

  5. Gao Z, Zhang S, Huang K,Wang Q, Zhong C (2021) Gradient distribution alignment certificates better adversarial domain adaptation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision 8937–8946

  6. Harper JC (2020) Acne vulgaris: What’s new in our 40th year. J Am Acad Dermatol 82(2):526–527

    Article  PubMed  Google Scholar 

  7. Jain A, Rao ACS, Jain PK, Abraham A (2022) Multi-type skin diseases classification using op-dnn based feature extraction approach. Multimedia Tools and Applications 1–26

  8. Kang J, Gwak J (2022) Ensemble of multi-task deep convolutional neural networks using transfer learning for fruit freshness classification. Multimedia Tools and Applications 81(16):22355–22377

    Article  Google Scholar 

  9. Kang Q, Yao S, Zhou M, Zhang K, Abusorrah A (2021) Effective visual domain adaptation via generative adversarial distribution matching. IEEE Transactions on Neural Networks and Learning Systems 32(9):3919–3929. https://doi.org/10.1109/TNNLS.2020.3016180

    Article  MathSciNet  PubMed  Google Scholar 

  10. Kim T, Cha M, Kim H, Lee JK, Kim J (2017) Learning to discover crossdomain relations with generative adversarial networks. In: International Conference on Machine Learning 1857–1865. PMLR

  11. Krishnapriya K, King MC, Bowyer KW (2021) Analysis of manual and automated skin tone assignments for face recognition applications. arXiv:2104.14685

  12. Li P, Li R, Da Q, Zeng AX, Zhang L (2020) Improving multi-scenario learning to rank in e-commerce by exploiting task relationships in the label space. In: International Conference on Information & Knowledge Management 2605–2612

  13. Lim ZV, Akram F, Ngo CP, Winarto AA, Lee WQ, Liang K, Oon HH, Thng STG, Lee HK (2020) Automated grading of acne vulgaris by deep learning with convolutional neural networks. Skin Research and Technology 26(2):187–192

    Article  PubMed  Google Scholar 

  14. Lin Z, Li J, Yao Q, Shen H, Wan L (2022) Adversarial learning with data selection for cross-domain histopathological breast cancer segmentation. Multimedia Tools and Applications 81(4):5989–6008

    Article  Google Scholar 

  15. Lin Y, Jiang J, Ma Z, Chen D, Guan Y, You H, Cheng X, Liu B, Luo G (2022) Kieglfn: A unified acne grading framework on face images. Computer Methods and Programs in Biomedicine 221:106911

    Article  PubMed  Google Scholar 

  16. Lin Y, Guan Y, Ma Z, You H, Cheng X, Jiang J (2021) An acne grading framework on face images via skin attention and sfnet. International Conference on Bioinformatics and Biomedicine 2407–2414

  17. Liu Y, Jain A, Eng C, Way DH, Lee K, Bui P, Kanada K, de Oliveira Marinho G, Gallegos J, Gabriele S et al (2020) A deep learning system for differential diagnosis of skin diseases. Nature Medicine 26(6):900–908

    Article  CAS  PubMed  Google Scholar 

  18. Yu J, Tan M, Zhang H, Tao D, Rui Y (2019). Hierarchical deep click feature prediction for fine-grained image recognition[J]. IEEE Trans Pattern Anal Mach Intell 44(2): 563-578

  19. Li N, Yang XX, Yang Ry, Yi F (2021) Study of the characteristics of facial skin tone status in 1092 young chinese females according to the ita\(^\circ \). Journal of Cosmetic Dermatology 1–9

  20. Lu A, Li C, Yan Y, Tang J, Luo B (2021) Rgbt tracking via multiadapter network with hierarchical divergence loss. IEEE Transactions on Image Processing 30:5613–5625. https://doi.org/10.1109/TIP.2021.3087341

    Article  ADS  PubMed  Google Scholar 

  21. Lu N, Hu H, Yin T, Lei Y, Wang S (2021) Transfer relation network for fault diagnosis of rotating machinery with small data. IEEE Transactions on Cybernetics 1–15. https://doi.org/10.1109/TCYB.2021.3085476

  22. Lullo JJ, Lee D, Umar S (2020) 13938 a new acne keloidalis nuchae classification system to guide surgical and laser treatments. J Am Acad Dermatol 83(6):15

    Article  Google Scholar 

  23. Mijwil MM (2021) Skin cancer disease images classification using deep learning solutions. Multimedia Tools and Applications 80(17):26255–26271

    Article  Google Scholar 

  24. Min K, Lee GH, Lee SW (2021) Acnet: Mask-aware attention with dynamic context enhancement for robust acne detection. arXiv:2105.14891

  25. Mishra S, Imaizumi H, Yamasaki T (2019) Interpreting fine-grained dermatological classification by deep learning. In: IEEE Conference on Computer Vision and Pattern Recognition Workshop

  26. Morales A, Fierrez J, Vera-Rodriguez R, Tolosana R (2020) Sensitivenets: Learning agnostic representations with application to face images. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(6):2158–2164

    Article  Google Scholar 

  27. Nguyen A, Thai H, Le T (2021) Severity assessment of facial acne. In: International Conference on Computational Collective Intelligence, pp. 599–612. Springer

  28. Pons G, Masip D (2020) Multitask, multilabel, and multidomain learning with convolutional networks for emotion recognition. IEEE Transactions on Cybernetics 1–8. https://doi.org/10.1109/TCYB.2020.3036935

  29. Salma W, Eltrass AS (2022) Automated deep learning approach for classification of malignant melanoma and benign skin lesions. Multimedia Tools and Applications 81(22):32643–32660

    Article  Google Scholar 

  30. Samuels DV, Rosenthal R, Lin R, Chaudhari S, Natsuaki MN (2020) Acne vulgaris and risk of depression and anxiety: a meta-analytic review. J Am Acad Dermatol 83(2):532–541

    Article  PubMed  Google Scholar 

  31. Tan M, Jin T, Ye D, Xu K, Gu X, Yu J (2023). Electromagnetic imaging boosted visual object recognition under difficult visual conditions[J]. IEEE Transactions on Geoscience and Remote Sensing. https://doi.org/10.1109/TGRS.2023.3278349

  32. Shang J, Niu C, Zhou Z, Huang J, Yang Z, Li X (2021) Asymmetric alignment joint consistent regularization for multi-source domain adaptation. Multimedia Tools and Applications 80:6041–6064

    Article  Google Scholar 

  33. Tan M, Yu J, Zhang H, Rui Y, Tao D (2019) Image recognition by predicted user click feature with multidomain multitask transfer deep network. IEEE Transactions on Image Processing 28(12):6047–6062

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  34. Vandenhende S, Georgoulis S, Van Gansbeke W, Proesmans M, Dai D, Van Gool L (2021) Multi-task learning for dense prediction tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 1–1. https://doi.org/10.1109/TPAMI.2021.3054719

  35. Verma A, Subramanyam A, Wang Z, Satoh S, Shah RR (2021) Unsupervised domain adaptation for person re-identification via individualpreserving and environmental-switching cyclic generation. IEEE Transactions on Multimedia

  36. Wang Q, Luo L, Xie H, Rao Y, Lau RY, Zhang D (2022) A deep data augmentation framework based on generative adversarial networks. Multimedia Tools and Applications 1–17

  37. Wu X, Wen N, Liang J, Lai YK, She D, Cheng MM, Yang J (2019) Joint acne image grading and counting via label distribution learning. In: IEEE International Conference on Computer Vision, pp. 10642–10651

  38. Xu GX, Liu C, Liu J, Ding Z, Shi F, Guo M, Zhao W, Li X, Wei Y, Gao Y et al (2021) Cross-site severity assessment of covid-19 from ct images via domain adaptation. IEEE Transactions on Medical Imaging 41(1):88–102

    Article  CAS  PubMed  Google Scholar 

  39. Yang Y, Guo L, Wu Q, Zhang M, Zeng R, Ding H, Zheng H, Xie J, Li Y, Ge Y et al (2021) Construction and evaluation of a deep learning model for assessing acne vulgaris using clinical images. Dermatology and Therapy 11(04):1239–1248

    Article  PubMed  PubMed Central  Google Scholar 

  40. Yang J, Kiyavash N, He N (2020) Global convergence and variance reduction for a class of nonconvex-nonconcave minimax problems. In: Advances in Neural Information Processing Systems 1153–1165

  41. Yao S, Kang Q, Zhou M, Rawa MJ, Albeshri A (2022) Discriminative manifold distribution alignment for domain adaptation. IEEE Transactions on Systems, Man, and Cybernetics: Systems

  42. Yaqoob S, Saleem A, Jarullah FA, Asif A, Essar MY, Emad S (2021) Association of acne with face mask in healthcare workers amidst the covid- 19 outbreak in karachi, pakistan. Clinical, Cosmetic and Investigational Dermatology 14:1427–1433

    Article  PubMed  PubMed Central  Google Scholar 

  43. Ye S, Wu K, Zhou M, Yang Y, Tan SH, Xu K, Song J, Bao C, Ma K (2020) Light-weight calibrator: a separable component for unsupervised domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition 13733–13742

  44. Zhao T, Zhang H, Spoelstra J (2019) A computer vision application for assessing facial acne severity from selfie images. arXiv preprint arXiv:1907.07901

  45. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired Image-to-image Translation Using Cycle-consistent Adversarial Networks

  46. Zhu Y, Zhao D (2020) Online minimax q network learning for two-player zerosum markov games. IEEE Transactions on Neural Networks and Learning Systems 1–14. https://doi.org/10.1109/TNNLS.2020.3041469

Download references

Acknowledgements

This work was supported by Zhejiang Provincial Natural Science Foundation of China (No.LZ23F020007), and National Natural Science Foundation of China (No. 61972119). This work was carried out at the Rapid- Rich Object Search (ROSE) Lab, Nanyang Technological University (NTU), Singapore. The research is supported in part by the A*STAR under it’s A*STAR MBRC Strategic Positioning Fund (SPF) - ASTARP &G Collaboration (Award APG2013/113).

Author information

Authors and Affiliations

  1. Key Laboratory of Complex Systems Modeling and Simulation, School of Computer Science and Technology, Hangzhou Dianzi University, Xiasha Higher Education Zone, Hangzhou, 310018, Zhejiang, People’s Republic of China

    Min Tan, Ruirui Wang, Tao Jin & Jun Yu

  2. Procter & Gamble, 70 Biopolis Street Singapore, Singapore, 138547, Singapore

    Ankur Purwar

  3. School of Electrical and Electronic Engineering, Nanyang Technological University, 50 Nanyang Avenue, Singapore, 639798, Singapore

    Alex C Kot

Authors
  1. Min Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruirui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ankur Purwar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tao Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jun Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Alex C Kot
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Min Tan.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tan, M., Wang, R., Purwar, A. et al. GTADT: Gated tone-sensitive acne grading via augmented domain transfer. Multimed Tools Appl 83, 24875–24897 (2024). https://doi.org/10.1007/s11042-023-16444-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16444-0

Keywords

Search

Navigation