Skip to main content
SpringerLink
Log in

An automated and efficient convolutional architecture for disguise-invariant face recognition using noise-based data augmentation and deep transfer learning

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Face recognition is diversely used in modern biometric and security applications. Most of the current face recognition techniques show good results in a constrained environment. However, these techniques face many problems in real-world scenarios such as low-quality images, temporal variations and facial disguises creating variations in facial features. The reason for these deteriorating results is the employment of handcrafted features having weak generalization capabilities and neglecting the complexities associated with domain adaption in case of deep learning models. In this paper, we have studied the efficacy of deep learning methods incorporating simple noise-based data augmentation for disguise invariant face recognition (DIFR). The proposed method detects face in an image using Viola Jones face detector and classifies it using a pre-trained Convolutional Neural Network (CNN) fine-tuned for DIFR. During transfer learning, a pre-trained CNN learns generalized disguise-invariant features from facial images of several subjects to correctly identify them under varying facial disguises. We have compared four different pre-trained 2D CNNs, each with different number of learning parameters, based on their classification accuracy and execution time for selecting a suitable model for DIFR. Comprehensive experiments and comparative analysis have been conducted on six challenging facial disguise datasets. Resnet-18 gives the best trade-off between accuracy and efficiency, by achieving an average accuracy of 98.19% with an average execution time of 0.32 seconds. The promising results achieved in these experiments reflect the efficiency of the proposed method and outperforms the existing methods in all aspects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. About Face ID advanced technology (2020). https://support.apple.com/en-us/HT208108

  2. Use Face ID on your iPhone or iPad Pro (2020). https://support.apple.com/en-us/HT208109

  3. Abdurrahim, S.H., Samad, S.A., Huddin, A.B.: Review on the effects of age, gender, and race demographics on automatic face recognition. Vis. Comput. 34(11), 1617–1630 (2018)

    Article  Google Scholar 

  4. Ahmad, H.M., Khan, M.J., Yousaf, A., Ghuffar, S., Khurshid, K.: Deep Learning: A breakthrough in Medical Imaging. Curr. Med. Imaging Formerly Curr. Med. Imaging Rev. 15(1), 1–14 (2020). https://doi.org/10.2174/1573405615666191219100824

    Article  Google Scholar 

  5. Ahonen, T., Hadid, A., Pietikäinen, M.: Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 2037–2041 (2006). https://doi.org/10.1109/TPAMI.2006.244

    Article  MATH  Google Scholar 

  6. Arsenovic, M., Sladojevic, S., Anderla, A., Stefanovic, D.: FaceTime - Deep learning based face recognition attendance system. In: Proceedings of the SISY 2017—IEEE 15th International Symposium on Intelligent Systems and Informatics, pp. 53–57. Institute of Electrical and Electronics Engineers Inc. (2017). https://doi.org/10.1109/SISY.2017.8080587

  7. Bansal, A., Ranjan, R., Castillo, C.D., Chellappa, R.: Deep features for recognizing disguised faces in the wild. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, vol. 2018, pp. 10–16. IEEE Computer Society (2018). https://doi.org/10.1109/CVPRW.2018.00009

  8. Cao, Z., Yin, Q., Tang, X., Sun, J.: Face recognition with learning-based descriptor. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2707–2714 (2010). https://doi.org/10.1109/CVPR.2010.5539992

  9. Chen, C., Dantcheva, A., Ross, A.: Automatic facial makeup detection with application in face recognition. In: Proceedings—2013 International Conference on Biometrics, ICB 2013. IEEE Computer Society (2013). https://doi.org/10.1109/ICB.2013.6612994

  10. Chen, C., Dantcheva, A., Ross, A.: An ensemble of patch-based subspaces for makeup-robust face recognition. Inf. fus. 32, 80–92 (2016)

    Article  Google Scholar 

  11. Chen, C., Dantcheva, A., Swearingen, T., Ross, A.: Spoofing faces using makeup: An investigative study. In: IEEE International Conference on Identity, Security and Behavior Analysis, ISBA 2017. Institute of Electrical and Electronics Engineers Inc. (2017). https://doi.org/10.1109/ISBA.2017.7947686

  12. Cheng, Y., Jiao, L., Cao, X., Li, Z.: Illumination-insensitive features for face recognition. Vis. Comput. 33(11), 1483–1493 (2017)

    Article  Google Scholar 

  13. Dantcheva, A., Chen, C., Ross, A.: Can facial cosmetics affect the matching accuracy of face recognition systems? In: 2012 IEEE 5th International Conference on Biometrics: Theory, Applications and Systems, BTAS 2012, pp. 391–398 (2012). https://doi.org/10.1109/BTAS.2012.6374605

  14. Deng, J., Guo, J., Xue, N., Zafeiriou, S.: ArcFace: Additive Angular Margin Loss for Deep Face Recognition. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2019-June, 4685–4694 (2018). arxiv: 1801.07698

  15. Deng, J., Guo, J., Zhou, Y., Yu, J., Kotsia, I., Zafeiriou, S.: RetinaFace: Single-stage Dense Face Localisation in the Wild (2019). arxiv: 1905.00641

  16. Deng, J., Zafeririou, S.: Arcface for disguised face recognition. In: Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019, pp. 485–493. Institute of Electrical and Electronics Engineers Inc. (2019). https://doi.org/10.1109/ICCVW.2019.00061

  17. Deng, W., Hu, J., Guo, J.: Compressive Binary Patterns: Designing a Robust Binary Face Descriptor with Random-Field Eigenfilters. IEEE Trans. Pattern Anal. Mach. Intell. 41(3), 758–767 (2019). https://doi.org/10.1109/TPAMI.2018.2800008

    Article  Google Scholar 

  18. Dhamecha, T.I., Nigam, A., Singh, R., Vatsa, M.: Disguise detection and face recognition in visible and thermal spectrums. In: Proceedings of 2013 International Conference on Biometrics, ICB 2013. IEEE Computer Society (2013). https://doi.org/10.1109/ICB.2013.6613019

  19. Dhamecha, T.I., Singh, R., Vatsa, M., Kumar, A.: Recognizing disguised faces: human and machine evaluation. PLoS One 9(7), 1 (2014). https://doi.org/10.1371/journal.pone.0099212

    Article  Google Scholar 

  20. Fredj, H.B., Bouguezzi, S., Souani, C.: Face recognition in unconstrained environment with CNN. The Visual Computer pp. 1–10 (2020)

  21. Gao, Y., Lee, H.J.: Pose-invariant features and personalized correspondence learning for face recognition. Neural Comput. Appl. 31(1), 607–616 (2019)

    Article  Google Scholar 

  22. Guo, G., Wen, L., Yan, S.: Face authentication with makeup changes. IEEE Trans. Circuits Syst. Video Technol. 24(5), 814–825 (2014). https://doi.org/10.1109/TCSVT.2013.2280076

    Article  MathSciNet  Google Scholar 

  23. Gupta, S., Thakur, K., Kumar, M.: 2d-human face recognition using sift and surf descriptors of face’s feature regions. Vis. Comput. 1, 1–10 (2020)

    Google Scholar 

  24. Huang, G.B., Mattar, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database forstudying face recognition in unconstrained environments (2008)

  25. Hung, K.M., Wu, J.A., Wen, C.H., Chen, L.M.: A system for disguised face recognition with convolution neural networks. In: ACM International Conference Proceeding Series, pp. 65–69. Association for Computing Machinery (2018). https://doi.org/10.1145/3299852.3299858

  26. Khan, M.J., Khan, H.S., Yousaf, A., Khurshid, K., Abbas, A.: Modern Trends in Hyperspectral Image Analysis: A Review. IEEE Access 6, 14118–14129 (2018). https://doi.org/10.1109/ACCESS.2018.2812999. https://ieeexplore.ieee.org/document/8314827/

  27. Khan, M.J., Khurshid, K., Shafait, F.: A Spatio-spectral hybrid convolutional architecture for hyperspectral document authentication. In: 2019 15th IAPR International Conference on Document Analysis and Recognition (ICDAR). IEEE (2019)

  28. Khan, M.J., Yousaf, A., Abbas, A., Khurshid, K.: Deep learning for automated forgery detection in hyperspectral document images. J. Electron. Imaging 27(05), 1 (2018). https://doi.org/10.1117/1.JEI.27.5.053001

    Article  Google Scholar 

  29. Khan, M.J., Yousaf, A., Javed, N., Nadeem, S., Khurshid, K.: Automatic Target Detection in Satellite Images using Deep Learning. J. Space Technol. 7(1), 44–49 (2017)

    Google Scholar 

  30. Khan, M.J., Yousaf, A., Khurshid, K., Abbas, A., Shafait, F.: Automated forgery detection in multispectral document images using fuzzy clustering. In: 13th IAPR International Workshop on Document Analysis Systems. IEEE, Vienna (2018). https://doi.org/10.1109/DAS.2018.26

  31. Kim, J., Sung, Y., Yoon, S.M., Park, B.G.: A new video surveillance system employing occluded face detection. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 3533 LNAI, pp. 65–68 (2005). https://doi.org/10.1007/11504894_10

  32. Kohli, N., Yadav, D., Noore, A.: Face verification with disguise variations via deep disguise recognizer. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, vol. 2018-June, pp. 17–24. IEEE Computer Society (2018). https://doi.org/10.1109/CVPRW.2018.00010

  33. Kose, N., Apvrille, L., Dugelay, J.L.: Facial makeup detection technique based on texture and shape analysis. In: 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, FG 2015. Institute of Electrical and Electronics Engineers Inc. (2015). https://doi.org/10.1109/FG.2015.7163104

  34. Kotwal, K., Mostaani, Z., Marcel, S.: Detection of age-induced makeup attacks on face recognition systems using multi-layer deep features. IEEE Trans. Biomet. Behav. Ident. Sci. 1, 1 (2019). https://doi.org/10.1109/tbiom.2019.2946175

    Article  Google Scholar 

  35. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks (2012)

  36. Kushwaha, V., Singh, M., Singh, R., Vatsa, M., Ratha, N., Chellappa, R.: Disguised faces in the wild. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, vol. 2018, pp. 1–9. IEEE Computer Society (2018). https://doi.org/10.1109/CVPRW.2018.00008

  37. Lei, Z., Pietikainen, M., Li, S.Z.: Learning discriminant face descriptor. IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 289–302 (2014). https://doi.org/10.1109/TPAMI.2013.112

    Article  Google Scholar 

  38. Li, Y., Song, L., Wu, X., He, R., Tan, T.: Anti-makeup: learning a bi-level adversarial network for makeup-invariant face verification (2017). arxiv: 1709.03654

  39. Liu, C., Wechsler, H.: Gabor feature based classification using the enhanced Fisher linear discriminant model for face recognition. IEEE Trans. Image Process. 11(4), 467–476 (2002). https://doi.org/10.1109/TIP.2002.999679

    Article  Google Scholar 

  40. Martínez, A.M.: Recognizing imprecisely localized, partially occluded, and expression variant faces from a single sample per class. IEEE Trans. Pattern Anal. Mach. Intell. 24(6), 748–763 (2002). https://doi.org/10.1109/TPAMI.2002.1008382

    Article  Google Scholar 

  41. Min, R., Hadid, A., Dugelay, J.L.: Improving the recognition of faces occluded by facial accessories. In: IEEE International conference on automatic face and gesture recognition and workshops, FG 2011, pp. 442–447 (2011). https://doi.org/10.1109/FG.2011.5771439

  42. Peri, S.V., Dhall, A.: DisguiseNet: A contrastive approach for disguised face verification in the wild. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, vol. 2018, pp. 25–31. IEEE Computer Society (2018). https://doi.org/10.1109/CVPRW.2018.00011

  43. Ramanathan, N., Chellappa, R., Roy Chowdhury, A.K.: Facial similarity across age, disguise, illumination and pose. Proc. Int. Conf. Image Process. ICIP 3, 1999–2002 (2004). https://doi.org/10.1109/ICIP.2004.1421474

    Article  Google Scholar 

  44. Ranjan, R., Sankaranarayanan, S., Castillo, C.D., Chellappa, R.: An all-in-one convolutional neural network for face analysis (2016). arxiv: 1611.00851

  45. Rasti, S., Yazdi, M., Masnadi-Shirazi, M.A.: Biologically inspired makeup detection system with application in face recognition. IET Biomet. 7(6), 530–535 (2018). https://doi.org/10.1049/iet-bmt.2018.5059

    Article  Google Scholar 

  46. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  47. Sajid, M., Ali, N., Dar, S.H., Iqbal Ratyal, N., Butt, A.R., Zafar, B., Shafique, T., Baig, M.J.A., Riaz, I., Baig, S.: Data augmentation-assisted makeup-invariant face recognition. Math. Problems Eng. 2018 (2018). https://doi.org/10.1155/2018/2850632

  48. Shu, X., Qi, G.J., Tang, J., Wang, J.: Weakly-shared deep transfer networks for heterogeneous-domain knowledge propagation. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 35–44 (2015)

  49. Singh, M., Singh, R., Vatsa, M., Ratha, N.K., Chellappa, R.: Recognizing disguised faces in the wild. IEEE Trans. Biometr. Behav. Ident. Sci. 1(2), 97–108 (2019). https://doi.org/10.1109/tbiom.2019.2903860

    Article  Google Scholar 

  50. Singh, R., Vatsa, M., Noore, A.: Face recognition with disguise and single gallery images. Image Vis. Comput. 27(3), 245–257 (2009). https://doi.org/10.1016/j.imavis.2007.06.010

    Article  Google Scholar 

  51. Smirnov, E., Ivanova, E., Melnikov, A., Kalinovskiy, I., Oleinik, A., Luckyanets, E.: Hard example mining with auxiliary embeddings. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, vol. 2018, pp. 37–46. IEEE Computer Society (2018). https://doi.org/10.1109/CVPRW.2018.00013

  52. Sun, Y., Ren, L., Wei, Z., Liu, B., Zhai, Y., Liu, S.: A weakly supervised method for makeup-invariant face verification. Pattern Recognit. 66, 153–159 (2017). https://doi.org/10.1016/j.patcog.2017.01.011

    Article  Google Scholar 

  53. Suri, S., Sankaran, A., Vatsa, M., Singh, R.: On matching faces with alterations due to plastic surgery and disguise. In: IEEE 9th International Conference on Biometrics Theory, Applications and Systems, BTAS 2018. Institute of Electrical and Electronics Engineers Inc. (2018). https://doi.org/10.1109/BTAS.2018.8698571

  54. Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: Closing the gap to human-level performance in face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1708 (2014)

  55. Tang, J., Shu, X., Li, Z., Qi, G.J., Wang, J.: Generalized deep transfer networks for knowledge propagation in heterogeneous domains. ACM Trans. Multimed. Comput. Commun. Appl. (TOMM) 12(4), 1–22 (2016)

    Google Scholar 

  56. Turk, M., Pentland, A.: Eigenfaces for recognition. J. Cognit. Neurosci. 3(1), 71–86 (1991). https://doi.org/10.1162/jocn.1991.3.1.71

    Article  Google Scholar 

  57. Viola, P., Jones, M.J.: Robust Real-Time Face Detection. Int. J. Comput. Vis. 57(2), 137–154 (2004). https://doi.org/10.1023/B:VISI.0000013087.49260.fb

    Article  Google Scholar 

  58. Wang, T.Y., Kumar, A.: Recognizing human faces under disguise and makeup. In: ISBA 2016—IEEE International Conference on Identity, Security and Behavior Analysis. Institute of Electrical and Electronics Engineers Inc. (2016). https://doi.org/10.1109/ISBA.2016.7477243

  59. Wang, Z., Miao, Z., Wu, Q.J., Wan, Y., Tang, Z.: Low-resolution face recognition: a review. Vis. Comput. 30(4), 359–386 (2014)

    Article  Google Scholar 

  60. Wu, F., Yan, S., Smith, J.S., Lu, W., Zhang, B.: Unsupervised domain adaptation for disguised face recognition. In: Proceedings of the 2019 IEEE International Conference on Multimedia and Expo Workshops, ICMEW 2019, pp. 537–542. Institute of Electrical and Electronics Engineers Inc. (2019). https://doi.org/10.1109/ICMEW.2019.00098

  61. Yang, M., Zhang, L.: Gabor feature based sparse representation for face recognition with Gabor occlusion dictionary. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 6316 LNCS, pp. 448–461 (2010). https://doi.org/10.1007/978-3-642-15567-3_33

  62. Yousaf, A., Khan, M.J., Khan, M.J., Javed, N., Ibrahim, H., Khurshid, K., Khurshid, K.: Size invariant handwritten character recognition using single layer feedforward backpropagation neural networks. In: 2nd International Conference on Computing, Mathematics and Engineering Technologies, iCoMET 2019 (2019). https://doi.org/10.1109/ICOMET.2019.8673459

  63. Yousaf, A., Khan, M.J., Khan, M.J., Siddiqui, A.M., Khurshid, K.: A robust and efficient convolutional deep learning framework for age-invariant face recognition. Expert Syst. e12503 (2019)

  64. Zhang, K., Chang, Y.L., Hsu, W.: Deep disguised faces recognition. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, vol. 2018, pp. 32–36. IEEE Computer Society (2018). https://doi.org/10.1109/CVPRW.2018.00012

  65. Zhao, J., Cheng, Y., Xu, Y., Xiong, L., Li, J., Zhao, F., Jayashree, K., Pranata, S., Shen, S., Xing, J., et al.: Towards pose invariant face recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2207–2216 (2018)

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Military College of Signals, National University of Sciences and Technology (NUST), Islamabad, 44000, Pakistan

    Muhammad Junaid Khan & Adil Masood Siddiqui

  2. Data Science Institute, National University of Ireland Galway, H91 TK33, Galway, Ireland

    Muhammad Jaleed Khan

  3. Artificial Intelligence and Computer Vision (iVision) Lab, Department of Electrical Engineering, Institute of Space Technology, Islamabad, 44000, Pakistan

    Khurram Khurshid

Authors
  1. Muhammad Junaid Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Muhammad Jaleed Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adil Masood Siddiqui
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Khurram Khurshid
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Junaid Khan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khan, M.J., Khan, M.J., Siddiqui, A.M. et al. An automated and efficient convolutional architecture for disguise-invariant face recognition using noise-based data augmentation and deep transfer learning. Vis Comput 38, 509–523 (2022). https://doi.org/10.1007/s00371-020-02031-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-02031-z

Keywords

Search

Navigation