Skip to main content
SpringerLink
Log in

Boosting facial recognition capability for faces wearing masks using attention augmented residual model with quadruplet loss

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Masked face recognition gained importance after the outbreak of COVID-19 when people started wearing facial masks to protect themselves against disease. The performance of existing face recognition systems is declined on the images of persons wearing face masks because the discriminative facial features are hidden under the mask. In addition, face recognition algorithms proposed for handling masked face recognition scenarios exhibit degraded performance in unmasked face recognition settings. Therefore, there is a need for a solution that can perform well in masked-face scenarios and maintains the same performance while recognizing the unmasked faces. In this paper, we propose a Masked Face Unveiling Model (MFUM), which can be added to the backbone of existing facial recognition systems and improves the performance of existing approaches in masked face settings without the need to retrain the current models. The MFUM utilizes the masked face embedding produced by the existing face recognition model backbone and uses attention augmented residual model for recognition. It enhances the similarity of masked face embedding with the unmasked face embedding of the same identity and diminishes the similarity with the unmasked facial embedding of the different identities. We experimented with different face recognition models’ backbone, MFUM alternative architectures, and attention mechanisms on the MFR2 dataset with real masked faces and the LFW dataset with synthetic masks. The results show that the proposed Masked Face Unveiling-Attention Augmented Dense Residual Unit trained with quadruplet loss outperformed not only other MFUM architectures and losses but also exhibited superior performance as compared to other state-of-the-art algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Ahmed, S.B., Ali, S.F., Ahmad, J., Adnan, M., Fraz, M.M.: On the frontiers of pose invariant face recognition: a review. Artif. Intell. Rev. 53, 2571–2634 (2020)

    Article  Google Scholar 

  2. Anwar, A., Raychowdhury, A.: Masked face recognition for secure authentication. arXiv preprint arXiv:2008.11104 (2020)

  3. Boutros, F., Damer, N., Kirchbuchner, F., Kuijper, A.: Self-restrained triplet loss for accurate masked face recognition. Pattern Recogn. 124, 108473 (2022)

    Article  Google Scholar 

  4. Carlos-Roca, L.R., Torres, I.H., Tena, C.F.: Facial recognition application for border control. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–7. IEEE (2018)

  5. Chen, S., Liu, Y., Gao, X., Han, Z.: MobileFaceNets: efficient CNNs for accurate real-time face verification on mobile devices. In: Chinese Conference on Biometric Recognition, pp. 428–438. Springer (2018)

  6. Dagnes, N., Vezzetti, E., Marcolin, F., Tornincasa, S.: Occlusion detection and restoration techniques for 3d face recognition: a literature review. Mach. Vis. Appl. 29, 789–813 (2018)

    Article  Google Scholar 

  7. Damer, N., Grebe, J.H., Chen, C., Boutros, F., Kirchbuchner, F., Kuijper, A.: The effect of wearing a mask on face recognition performance: an exploratory study. In: International Conference of the Biometrics Special Interest Group (BIOSIG), pp. 1–6. IEEE (2020)

  8. Deng, J., Guo, J., Xue, N., Zafeiriou, S.: ArcFace: additive angular margin loss for deep face recognition, In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4690–4699

  9. Déniz, O., Bueno, G., Salido, J., De la Torre, F.: Face recognition using histograms of oriented gradients. Pattern Recogn. Lett. 32(12), 1598–1603 (2011)

    Article  Google Scholar 

  10. Din, N.U., Javed, K., Bae, S., Yi, J.: A novel GAN-based network for unmasking of masked face. IEEE Access 8, 44276–44287 (2020)

    Article  Google Scholar 

  11. Hariri, W.: Efficient masked face recognition method during the COVID-19 pandemic. SIViP 16, 605–612 (2022)

    Article  Google Scholar 

  12. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  13. He, X., Yan, S., Hu, Y., Niyogi, P., Zhang, H.J.: Face recognition using Laplacianfaces. IEEE Trans. Pattern Anal. Mach. Intell. 27, 328–340 (2005)

    Article  Google Scholar 

  14. Huang, B., Wang, Z., Wang, G., Jiang, K., Zeng, K., Han, Z., Tian, X., Yang, Y.: When face recognition meets occlusion: a new benchmark. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4240–4244. IEEE (2021)

  15. Huang, G.B., Mattar, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database forstudying face recognition in unconstrained environments. In: Workshop on Faces in’Real-Life’Images: Detection, Alignment, and Recognition (2008)

  16. Huber, M., Boutros, F., Kirchbuchner, F., Damer, N.: Mask-invariant face recognition through template-level knowledge distillation. In: 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pp. 1–8. IEEE (2021)

  17. Koc, M.: A novel partition selection method for modular face recognition approaches on occlusion problem. Mach. Vis. Appl. 32, 1–11 (2021)

    Article  Google Scholar 

  18. Li, Y., Guo, K., Lu, Y., Liu, L.: Cropping and attention based approach for masked face recognition. Appl. Intell. 51, 3012–3025 (2021)

    Article  Google Scholar 

  19. Maharani, D.A., Machbub, C., Rusmin, P.H., Yulianti, L.: Improving the capability of real-time face masked recognition using cosine distance. In: 6th International Conference on Interactive Digital Media (ICIDM), pp. 1–6. IEEE (2020)

  20. Malakar, S., Chiracharit, W., Chamnongthai, K., Charoenpong, T.: Masked face recognition using principal component analysis and deep learning. In: 2021 18th Intl. Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), pp. 785–788. IEEE

  21. Muhi, O.A., Farhat, M., Frikha, M.: Transfer learning for robust masked face recognition. In: 2022 6th International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), pp. 1–5. https://doi.org/10.1109/ATSIP55956.2022.9805960(2022)

  22. Mundial, I.Q., Hassan, M.S.U., Tiwana, M.I., Qureshi, W.S., Alanazi, E.: Towards facial recognition problem in COVID-19 pandemic. In: 4rd International Conference on Electrical, Telecommunication and Computer Engineering (ELTICOM), pp. 210–214. IEEE (2020)

  23. Nawshad, M., Fraz, M.: Improving masked face recognition using dense residual unit aided with quadruplet loss. In: 2022 37th International Conference on Image and Vision Computing New Zealand (IVCNZ 2022) (2022)

  24. Nawshad, M., Zafar, Z., Fraz, M.: Recognition of faces wearing masks using skip connection based dense units augmented with self restrained triplet loss. In: 2022 24th International Multitopic Conference (INMIC), pp. 1–7. https://doi.org/10.1109/INMIC56986.2022.9972912 (2022)

  25. Ngan, M.L., Grother, P.J., Hanaoka, K.K.: Ongoing face recognition vendor test (FRVT) part 6a: face recognition accuracy with masks using pre-COVID-19 algorithms (2020)

  26. Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)

  27. Sharma, V., Gangaraju, S., Sharma, V.K.: Masked face recognition

  28. Singh, S., Prasad, S.: Techniques and challenges of face recognition: a critical review. Procedia Comput. Sci. 143, 536–543 (2018)

    Article  Google Scholar 

  29. Song, L., Gong, D., Li, Z., Liu, C., Liu, W.: Occlusion robust face recognition based on mask learning with pairwise differential Siamese network. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 773–782

  30. Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: DeepFace: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1708

  31. Wan, W., Chen, J.: Occlusion robust face recognition based on mask learning. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 3795–3799. IEEE (2017)

  32. Wang, M., Deng, W.: Deep face recognition: a survey. Neurocomputing 429, 215–244 (2021)

    Article  Google Scholar 

  33. Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: CBAM: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  34. Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Ma, Y.: Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31, 210–227 (2008)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National University of Sciences and Technology (NUST), Islamabad, Pakistan

    Muhammad Aasharib Nawshad, Ahsan Saadat & Muhammad Moazam Fraz

Authors
  1. Muhammad Aasharib Nawshad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahsan Saadat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Muhammad Moazam Fraz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Moazam Fraz.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nawshad, M.A., Saadat, A. & Fraz, M.M. Boosting facial recognition capability for faces wearing masks using attention augmented residual model with quadruplet loss. Machine Vision and Applications 34, 108 (2023). https://doi.org/10.1007/s00138-023-01461-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-023-01461-8

Keywords

Search

Navigation