Skip to main content
SpringerLink
Log in

Explainable masked face recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The COVID-19 epidemic has made us all to understand that using face masks is one of the best ways to save ourselves from infections. Face recognition techniques often focus on the essential facial landmarks such as nose, mouth and eyes. Removing masks in airports or other public places for authentication will raise the danger of virus infection, and this will pose a challenge to the existing face recognition systems. This paper presents a novel approach to recognize masked faces that combines cropping the unmasked half of the face (the upper part of a face) image using a vision transformer model. Various scenarios are investigated in this paper. The model is trained using the images of the upper half of the face and tested on the same, full-face images with a mask and full face images without a mask. The experiments on the standard datasets, namely RMFRD, MLFW and masked CASIA-WebFace show that the proposed approach improves masked face recognition significantly compared to the existing methods. The explainability of the proposed model is demonstrated using a class activation map.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability

Raw data were available at the https://www.kaggle.com/datasets/ntl0601/casia-webface, https://www.kaggle.com/datasets/muhammeddalkran/lfw-simulated-masked-face-dataset and https://www.kaggle.com/datasets/muhammeddalkran/masked-facerecognition. Derived data supporting the findings of this study are available from the corresponding author on request. The created dataset for testing the model in real-time is having privacy issues and it can not be made public.

References

  1. Ali W, Tian W, Din SU, Iradukunda D, Khan AA (2021) Classical and modern face recognition approaches: a complete review. Multimed Tools Appl 80:4825–4880

    Article  Google Scholar 

  2. Turk M (1991) Pentland. eigenfaces for recognition. J Cogn Neurosci 4:72–86

    Google Scholar 

  3. Yang M, Zhang L, Shiu SCK, Zhang D (2012) Monogenic binary coding: An efficient local feature extraction approach to face recognition. IEEE Trans Inf Forensics Secur 7(6):1738–1751

    Article  ADS  Google Scholar 

  4. Cao Q, Shen L, Xie W, Parkhi OM, Zisserman A (2018) Vggface2: A dataset for recognising faces across pose and age. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) 67–74. IEEE

  5. Zhang G, Huang X, Li SZ, Wang Y, Wu X (2004) Boosting local binary pattern (lbp)-based face recognition. In: Chinese Conference on Biometric Recognition, pp. 179–186. Springer

  6. Sun Y, Chen Y, Wang X, Tang X (2014) Deep learning face representation by joint identification-verification. Advances in neural information processing systems 27

  7. Taigman Y, Yang M, Ranzato M, Wolf L (2014) Deepface: Closing the gap to human-level performance in face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 1701–1708

  8. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 815–823

  9. Wang H, Wang Y, Zhou Z, Ji X, Gong D, Zhou J, Li Z, Liu W (2018) Cosface: Large margin cosine loss for deep face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 5265–5274

  10. Deng J, Guo J, Xue N, Zafeiriou S (2019) Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 4690–4699

  11. Li Y, Guo K, Lu Y, Liu L (2021) Cropping and attention based approach for masked face recognition. Appl Intell 51(5):3012–3025

    Article  Google Scholar 

  12. Alzu’bi A, Albalas F, Al-Hadhrami T, Younis LB, Bashayreh A, (2021) Masked face recognition using deep learning: A review. Electronics 10(21):2666

  13. Liu W, Wen Y, Yu Z, Li M, Raj B, Song L (2017) Sphereface: Deep hypersphere embedding for face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 212–220

  14. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S et al (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929

  15. Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) End-to-end object detection with transformers. In: European Conference on Computer Vision, pp. 213–229. Springer

  16. Zhu Z, Huang G, Deng J, Ye Y, Huang J, Chen X, Zhu J, Yang T, Guo J, Lu J et al (2021) Masked face recognition challenge: The webface260m track report. arXiv:2108.07189

  17. Deng J, Guo J, An X, Zhu Z, Zafeiriou S (2021) Masked face recognition challenge: The insightface track report. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1437–1444

  18. Wang Z, Wang G, Huang B, Xiong Z, Hong Q, Wu H, Yi P, Jiang K, Wang N, Pei Y et al (2020) Masked face recognition dataset and application. arXiv:2003.09093

  19. Ge S, Li J, Ye Q, Luo Z (2017) Detecting masked faces in the wild with llecnns. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2682–2690

  20. Anwar A, Raychowdhury A (2020) Masked face recognition for secure authentication. arXiv:2008.11104

  21. Zhang Y, Wang X, Shakeel MS, Wan H, Kang W (2022) Learning upper patch attention using dual-branch training strategy for masked face recognition. Pattern Recognition 126:108522

    Article  Google Scholar 

  22. Ding F, Peng P, Huang Y, Geng M, Tian Y (2020) Masked face recognition with latent part detection. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2281–2289

  23. Hariri W (2022) Efficient masked face recognition method during the covid-19 pandemic. Signal, image and video processing 16(3):605–612

    Article  PubMed  Google Scholar 

  24. Du H, Shi H, Liu Y, Zeng D, Mei T (2021) Towards nir-vis masked face recognition. IEEE Signal Process Lett 28:768–772

    Article  ADS  Google Scholar 

  25. Golwalkar R, Mehendale N (2022) Masked-face recognition using deep metric learning and facemasknet-21. Applied Intelligence 1–12

  26. Geng M, Peng P, Huang Y, Tian Y (2020) Masked face recognition with generative data augmentation and domain constrained ranking. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2246–2254

  27. Kumar S, Yadav D, Gupta H, Kumar M, Verma OP (2022) Towards smart surveillance as an aftereffect of covid-19 outbreak for recognition of face masked individuals using yolov3 algorithm. Multimedia Tools and Applications 1–23

  28. Yi D, Lei Z, Liao S, Li SZ (2014) Learning face representation from scratch. arXiv:1411.7923

  29. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503

    Article  ADS  Google Scholar 

  30. Ku H, Dong W (2020) Face recognition based on mtcnn and convolutional neural network. Frontiers in Signal Processing 4(1):37–42

    Article  Google Scholar 

  31. Zhong Y, Deng W (2021) Face transformer for recognition. arXiv:2103.14803

  32. PyTorch: TORCHVISION.MODELS. https://pytorch.org/vision/0.8/models.html

  33. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929

  34. Yin B, Tran L, Li H, Shen X, Liu X (2019) Towards interpretable face recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9348–9357

  35. Song L, Gong D, Li Z, Liu C, Liu W (2019) Occlusion robust face recognition based on mask learning with pairwise differential siamese network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 773–782

  36. Shuai Z, Feng S (2021) Face recognition method of mask occlusion. In: 2021 IEEE International Conference on Smart Internet of Things (SmartIoT), pp. 82–88. IEEE

  37. Wu G (2021) Masked face recognition algorithm for a contactless distribution cabinet. Mathematical problems in engineering 2021

  38. Chen HQ, Xie K, Li MR, Wen C, He JB (2022) Face recognition with masks based on spatial fine-grained frequency domain broadening. Ieee Access 10:75536–75548

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Information and Technology Design and Manufacturing, Kancheepuram, Tamilnadu, 600127, India

    Anjali T & Masilamani V

Authors
  1. Anjali T
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Masilamani V
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anjali T.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

T, A., V, M. Explainable masked face recognition. Multimed Tools Appl 83, 31123–31138 (2024). https://doi.org/10.1007/s11042-023-16571-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16571-8

Keywords

Search

Navigation