Skip to main content
SpringerLink
Log in

Voxel-based 3D occlusion-invariant face recognition using game theory and simulated annealing

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

A novel voxel-based occlusion-invariant 3D face recognition framework (V3DOFR) based on game theory and simulated annealing is proposed. In V3DOFR approach, 3D meshes are converted to voxel form of sizes 43, 83, and 163. After that, locality preserving projection-based embeddings are computed for removing the sparseness of voxels and generating consistent linear embedding per mesh with size 64 × 3, 128 × 3, and 256 × 3, respectively. The generator of triplets provides the triplets of sizes 64x3x3, 128x3x3, and 256x3x3. The simulated annealing is used to check the threshold value of adversarial triplet loss generated after ensembling losses of different grid sizes. The proposed framework is compared with four well-known methods using three face datasets, namely, Bosphorus, UMBDB, and KinectFaceDB. The performance evaluation has been done using four different cases of experimentations, viz. voxel based face recognition, occlusion invariant face recognition, landmarks based 3D face recognition, and 3D mesh based face recognition. Seven evaluation metrics are used to compare the proposed technique with other methods. The proposed method provides better accuracy and computation time over the other existing techniques in the majority of cases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Abrevaya VF, Boukhayma A, Wuhrer S and Boyer E(2019) A decoupled 3D facial shape model by adversarial training. In proceedings of the IEEE international conference on computer vision (pp. 9419-9428).

  2. Alom, M.Z., Taha, T.M., Yakopcic, C., Westberg, S., Sidike, P., Nasrin, M.S., Van Esesn, B.C., Awwal, A.A.S. and Asari, V.K. (2018) The history began from AlexNet: a comprehensive survey on deep learning approaches. arXiv preprint arXiv:1803.01164

  3. Alom MZ, Taha TM, Yakopcic C, Westberg S, Sidike P, Nasrin MS, Hasan M, Van Essen BC, Awwal AA, Asari VK (2019) A state-of-the-art survey on deep learning theory and architectures. Electronics 8(3):292

    Article  Google Scholar 

  4. Antipov G, Baccouche M and Dugelay JL (2017) Face aging with conditional generative adversarial networks. In 2017 IEEE International Conference on Image Processing (ICIP). pp. 2089–2093. IEEE

  5. Bai S, Zhou Z, Wang J, Bai X, Jan Latecki L, Tian Q (2017) Ensemble diffusion for retrieval. In Proceedings of the IEEE International Conference on Computer Vision. pp. 774–783

  6. Bai S, Bai X, Tian Q, Latecki LJ (2017) Regularized diffusion process for visual retrieval. In Thirty-First AAAI Conference on Artificial Intelligence

  7. Bandyopadhyay S, Maulik U, Pakhira MK (2001) Clustering using simulated annealing with probabilistic redistribution. Int J Pattern Recognit Artif Intell 15(02):269–285

    Article  Google Scholar 

  8. Bandyopadhyay S, Saha S, Maulik U, Deb K (2008) A simulated annealing-based multiobjective optimization algorithm: AMOSA. IEEE Trans Evol Comput 12(3):269–283

    Article  Google Scholar 

  9. Belkin M, Niyogi P (2002) Laplacian eigenmaps and spectral techniques for embedding and clustering. In Advances in neural information processing systems. pp. 585–591

  10. Bi H, Li N, Guan H, Lu D and Yang L (2019, September) A multi-scale conditional generative adversarial network for face sketch synthesis. In 2019 IEEE international conference on image processing (ICIP) (pp. 3876-3880). IEEE.

  11. Bowyer KW, Chang K, Flynn P (2004) A survey of 3D and multi-modal 3D+ 2D face recognition.

    Google Scholar 

  12. CASIA-3D FaceV1, 3d face database

  13. Caves R, Quegan S, White R (1998) Quantitative comparison of the performance of SAR segmentation algorithms. IEEE Trans Image Process 7(11):1534–1546

    Article  Google Scholar 

  14. Chen Y, Garcia EK, Gupta MR, Rahimi A, Cazzanti L (2009) Similarity-based classification: concepts and algorithms. J Mach Learn Res 10:747–776

    MathSciNet  MATH  Google Scholar 

  15. Chen W, Chen X, Zhang J and Huang K (2017) Beyond triplet loss: a deep quadruplet network for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 403–412

  16. Cho M, Kim T, Kim IJ and Lee S (2020) Relational deep feature learning for heterogeneous face recognition. arXiv preprint arXiv:2003.00697.

  17. Colombo A, Cusano C, Schettini R (2011) UMB-DB: A database of partially occluded 3D faces. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops). pp. 2113–2119. IEEE

  18. Deng J, Guo J, Xue N, Zafeiriou S (2019) Arcface: additive angular margin loss for deep face recognition. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4690-4699).

  19. Ding C, Tao D (2016) A comprehensive survey on pose-invariant face recognition. ACM Transactions on intelligent systems and technology (TIST) 7(3):1–42

    Article  Google Scholar 

  20. Do TT, Tran T, Reid I, Kumar V, Hoang T, Carneiro G (2019) A theoretically sound upper bound on the triplet loss for improving the efficiency of deep distance metric learning. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 10404-10413).

  21. Dong Y, Zhang Z, Hong WC (2018) A hybrid seasonal mechanism with a chaotic cuckoo search algorithm with a support vector regression model for electric load forecasting. Energies 11(4):1009

    Article  Google Scholar 

  22. Dou P, Shah SK and Kakadiaris IA (2017) End-to-end 3D face reconstruction with deep neural networks. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5908-5917).

  23. El Sayed AR, El Chakik A, Alabboud H, Yassine A (2018) Efficient 3D point clouds classification for face detection using linear programming and data mining. The Imaging Science Journal 66(1):23–37

    Article  Google Scholar 

  24. El Sayed A, Kongar E, Mahmood A, Sobh T and Boult T (2018) Neural generative models for 3D faces with application in 3D texture free face recognition. arXiv preprint arXiv:1811.04358

  25. Faltemier TC, Bowyer KW and Flynn PJ (2007) Using a multi-instance enrollment representation to improve 3D face recognition. In 2007 First IEEE International Conference on Biometrics: Theory, Applications, and Systems. pp. 1–6. IEEE

  26. Fan DP, Cheng MM, Liu JJ, Gao SH, Hou Q and Borji A (2018) Salient objects in clutter: bringing salient object detection to the foreground. In proceedings of the European conference on computer vision (ECCV) (pp. 186-202).

  27. Fan DP, Zhang S, Wu YH, Liu Y, Cheng MM, Ren B, Rosin PL and Ji R (2019) Scoot: a perceptual metric for facial sketches. In proceedings of the IEEE international conference on computer vision (pp. 5612-5622).

  28. Gecer B, Ploumpis S, Kotsia I and Zafeiriou S (2019) GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction. arXiv preprint arXiv:1902.05978

  29. Goodfellow I (2016) NIPS 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160

  30. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In Advances in neural information processing systems:2672–2680

  31. Hassaballah M, Aly S (2015) Face recognition: challenges, achievements and future directions. IET Comput Vis 9(4):614–626

    Article  Google Scholar 

  32. He X (2005) Locality preserving projections. The University of Chicago, A dissertation submitted to the faculty of the division of the physical sciences in candidacy for the degree of doctor of philosophy Department of Computer Science

    Google Scholar 

  33. He X, Niyogi, P (2004) Locality preserving projections. In Advances in neural information processing systems pp. 153–160

  34. He Z, Zuo W, Kan M, Shan S, Chen X (2019) Attgan: facial attribute editing by only changing what you want. IEEE Trans Image Process 28:5464–5478

    Article  MathSciNet  MATH  Google Scholar 

  35. Hong WC, Dong Y, Lai CY, Chen LY, Wei SY (2011) SVR with hybrid chaotic immune algorithm for seasonal load demand forecasting. Energies 4(6):960–977

    Article  Google Scholar 

  36. Hong WC, Li MW, Geng J, Zhang Y (2019) Novel chaotic bat algorithm for forecasting complex motion of floating platforms. Appl Math Model 72:425–443

    Article  MathSciNet  MATH  Google Scholar 

  37. Hossin M, Sulaiman MN (2015) A review on evaluation metrics for data classification evaluations. International Journal of Data Mining & Knowledge Management Process 5(2):1

    Article  Google Scholar 

  38. Huang Y, Wang Y, Tai Y, Liu X, Shen P, Li S, Li J and Huang F (2020) CurricularFace: adaptive curriculum learning loss for deep face recognition. arXiv preprint arXiv:2004.00288.

  39. Jiao Y, Niu Y, Zhang Y, Li F, Zou C, Shi G (2019, December) Facial attention based convolutional neural network for 2D+ 3D facial expression recognition. In 2019 IEEE visual communications and image processing (VCIP) (pp. 1-4). IEEE.

  40. Kemelmacher-Shlizerman I, Seitz SM, Miller D and Brossard E (2016) The megaface benchmark: 1 million faces for recognition at scale. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4873-4882).

  41. Kim D, Hernandez M, Choi J and Medioni G (2017) Deep 3D face identification. In 2017 IEEE International Joint Conference on Biometrics (IJCB) pp. 133–142. IEEE

  42. Kim D, Hernandez M, Choi J, Medioni G (2017) Deep 3D face identification. In 2017 IEEE International Joint Conference on Biometrics (IJCB). pp. 133–142. IEEE

  43. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  44. Kirkpatrick S, Gelatt CD, Vecchi MP (1983) Optimization by simulated annealing. science, 220(4598), pp.671–680

  45. Korshunov P and Marcel S (2018) DeepFakes: a new threat to face recognition? Assessment and Detection. arXiv preprint arXiv:1812.08685

  46. Larsen, A.B.L., Sønderby, S.K., Larochelle, H. & Winther, O. (2016) Autoencoding beyond pixels using a learned similarity metric. Proceedings of The 33rd International Conference on Machine Learning, in PMLR 48:1558–1566

  47. Learned-Miller, E., Huang, G.B., RoyChowdhury, A., Li, H. and Hua, G., 2016. Labeled faces in the wild: a survey. In advances in face detection and facial image analysis (pp. 189-248). Springer, Cham.

  48. Lei Y, Guo Y, Hayat M, Bennamoun M, Zhou X (2016) A two-phase weighted collaborative representation for 3D partial face recognition with single sample. Pattern Recogn 52:218–237

    Article  Google Scholar 

  49. Li H, Huang D, Morvan JM, Chen L, Wang Y (2014) Expression-robust 3D face recognition via weighted sparse representation of multi-scale and multi-component local normal patterns. Neurocomputing 133:179–193

    Article  Google Scholar 

  50. Li H, Huang D, Morvan JM, Wang Y, Chen L (2015) Towards 3D face recognition in the real: a registration-free approach using fine-grained matching of 3D keypoint descriptors. Int J Comput Vis 113(2):128–142

    Article  MathSciNet  Google Scholar 

  51. Liu F, Zhao Q, Zeng D (2018) Joint face alignment and 3D face reconstruction with application to face recognition. IEEE Trans Pattern Anal Mach Intell

  52. Maulik U, Bandyopadhyay S, Trinder JC (2001) SAFE: an efficient feature extraction technique. Knowl Inf Syst 3(3):374–387

    Article  MATH  Google Scholar 

  53. Maze B, Adams J, Duncan JA, Kalka N, Miller T, Otto C, Jain AK, Niggel WT, Anderson J, Cheney J and Grother P, (2018, February) Iarpa janus benchmark-c: face dataset and protocol. In 2018 international conference on biometrics (ICB) (pp. 158-165). IEEE.

  54. Min R, Kose N, Dugelay JL (2014) Kinectfacedb: a kinect database for face recognition. IEEE Transactions on Systems, Man, and Cybernetics: Systems 44(11):1534–1548

    Article  Google Scholar 

  55. Moreno A (2004) GavabDB: a 3D face database. In Proc. 2nd COST275 workshop on biometrics on the internet, 2004 (pp. 75-80).

  56. Moschoglou S, Papaioannou A, Sagonas C, Deng J, Kotsia I, Zafeiriou S (2017) Agedb: the first manually collected, in-the-wild age database. In proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 51-59).

  57. ND-2006 Face Data Set. http://www.nd.edu/˜cvrl/. 2007.

  58. Ogáyar CJ, Rueda AJ, Segura RJ, Feito FR (2007) Fast and simple hardware accelerated voxelizations using simplicial coverings. Vis Comput 23(8):535–543

    Article  Google Scholar 

  59. Oh Song H, Xiang Y, Jegelka S, Savarese S (2016) Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4004–4012

  60. Pantaleoni J (2011) VoxelPipe: a programmable pipeline for 3D voxelization. In Proceedings of the ACM SIGGRAPH Symposium on High Performance Graphics. pp. 99–106. ACM

  61. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In British Machine Vision Conference (BMVC) 1(3):6

    Google Scholar 

  62. Patil H, Kothari A, Bhurchandi K (2015) 3-D face recognition: features, databases, algorithms and challenges. Artif Intell Rev 44(3):393–441

    Article  Google Scholar 

  63. Perarnau G, Van De Weijer J, Raducanu B and Álvarez JM (2016) Invertible conditional gans for image editing. arXiv preprint arXiv:1611.06355

  64. Pham HX, Chen C, Dao LN, Pavlovic V, Cai J and Cham TJ (2015) Robust performance-driven 3d face tracking in long range depth scenes. arXiv preprint arXiv:1507.02779

  65. Phillips PJ, Moon H, Rizvi SA, Rauss PJ (2000) The FERET evaluation methodology for face-recognition algorithms. IEEE Trans Pattern Anal Mach Intell 22(10):1090–1104

    Article  Google Scholar 

  66. Phillips PJ, Flynn PJ, Scruggs T, Bowyer KW, Chang J, Hoffman K, Marques J, Min J, Worek W (2005, June) Overview of the face recognition grand challenge. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05) (Vol. 1, pp. 947-954). IEEE.

  67. Ranjan A, Bolkart T, Sanyal S, Black MJ (2018) Generating 3D faces using convolutional mesh autoencoders. In proceedings of the European conference on computer vision (ECCV) (pp. 704-720).

  68. Rathgeb C, Dantcheva A, Busch C (2019) Impact and detection of facial beautification in face recognition: an overview. IEEE Access 7:152667–152678

    Article  Google Scholar 

  69. Salimans T, Goodfellow I, Zaremba W, Cheung V, Radford A and Chen X (2016) Improved techniques for training gans. In Advances in neural information processing systems. pp. 2234–2242

  70. Sanderson C (2002) The vidtimit database (No. REP_WORK). IDIAP

  71. Savran A, Alyüz N, Dibeklioğlu H, Çeliktutan O, Gökberk B, Sankur B and Akarun L (2008) Bosphorus database for 3D face analysis. In European Workshop on Biometrics and Identity Management. pp. 47–56. Springer, Berlin, Heidelberg

  72. Scherhag U, Rathgeb C, Merkle J, Breithaupt R, Busch C (2019) Face recognition systems under morphing attacks: a survey. IEEE Access 7:23012–23026

    Article  Google Scholar 

  73. Schroff F, Kalenichenko D and Philbin J (2015) Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 815–823

  74. Sengupta S, Chen JC, Castillo C, Patel VM, Chellappa R and Jacobs DW (2016, March) Frontal to profile face verification in the wild. In 2016 IEEE winter conference on applications of computer vision (WACV) (pp. 1-9). IEEE.

  75. Sharma S, Kumar V (2020) Voxel-based 3D face reconstruction and its application to face recognition using sequential deep learning. Multimedia tools and applications, pp.1-28.

  76. Spreeuwers L (2011) Fast and accurate 3d face recognition. Int J Comput Vis 93(3):389–414

    Article  MATH  Google Scholar 

  77. Tan Y, Lin H, Xiao Z, Ding S and Chao H (2018) Face recognition from sequential sparse 3D data via deep registration. arXiv preprint arXiv:1810.09658

  78. Vijayan V, Bowyer KW, Flynn PJ, Huang D, Chen L, Hansen M, Ocegueda O, Shah SK, Kakadiaris IA (2011, October) Twins 3D face recognition challenge. In 2011 international joint conference on biometrics (IJCB) (pp. 1-7). IEEE.

  79. Wang X, Tang X (2008) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31(11):1955–1967

    Article  Google Scholar 

  80. Whitelam C, Taborsky E, Blanton A, Maze B, Adams J, Miller T, Kalka N, Jain AK, Duncan JA, Allen K and Cheney J (2017) Iarpa janus benchmark-b face dataset. In proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 90-98).

  81. Wu Z, Song S, Khosla A, Tang X, Xiao J (2014) 3D Shapenets for 2.5D object recognition and next-best-view prediction. arXiv preprint arXiv:1406.5670, 2(4)

  82. Xu D, Hu P, Cao W, Li H (2008, June) SHREC’08 entry: 3D face recognition using moment invariants. In 2008 IEEE international conference on shape modeling and applications (pp. 261-262). IEEE.

  83. Yang XS (2010) Nature-inspired metaheuristic algorithms. Luniver press

  84. Yi D, Lei Z, Liao S, Li SZ (2014) Learning face representation from scratch. arXiv preprint arXiv:1411.7923.

  85. Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006, April) A 3D facial expression database for facial behavior research. In 7th international conference on automatic face and gesture recognition (FGR06) (pp. 211-216). IEEE.

  86. Yin L, Sun\ XCY, Worm T and Reale M (2008) A high-resolution 3d dynamic facial expression database. In IEEE International Conference on Automatic Face and Gesture Recognition, Amsterdam, The Netherlands. 126

  87. Zhang Z, Hong WC (2019) Electric load forecasting by complete ensemble empirical mode decomposition adaptive noise and support vector regression with quantum-based dragonfly algorithm. Nonlinear Dynamics 98(2):1107–1136

    Article  Google Scholar 

  88. Zhang W, Wang X, Tang X (2011, June) Coupled information-theoretic encoding for face photo-sketch recognition. In CVPR 2011 (pp. 513-520). IEEE.

  89. Zhang X, Yin L, Cohn JF, Canavan S, Reale M, Horowitz A and Liu P (2013, April) A high-resolution spontaneous 3d dynamic facial expression database. In 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG) (pp. 1-6). IEEE.

  90. Zhang X, Yin L, Cohn JF, Canavan S, Reale M, Horowitz A, Liu P, Girard JM (2014) Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database. Image Vis Comput 32(10):692–706

    Article  Google Scholar 

  91. Zhang Y, Zhang L, Neoh SC, Mistry K, Hossain MA (2015) Intelligent affect regression for bodily expressions using hybrid particle swarm optimization and adaptive ensembles. Expert Syst Appl 42(22):8678–8697

    Article  Google Scholar 

  92. Zhang Z, Hong WC, Li J (2020) Electric load forecasting by hybrid self-recurrent support vector regression model with variational mode decomposition and improved cuckoo search algorithm. IEEE Access 8:14642–14658

    Article  Google Scholar 

  93. Zhang J, Yu X, Li A, Song P, Liu B and Dai Y (2020) Weakly-supervised salient object detection via scribble annotations. arXiv preprint arXiv:2003.07685.

  94. Zhang J, Fan DP, Dai Y, Anwar S, Saleh FS, Zhang T and Barnes N (2020) UC-net: uncertainty inspired rgb-d saliency detection via conditional variational autoencoders. arXiv preprint arXiv:2004.05763.

  95. Zhao Y, Jin Z, Qi GJ, Lu H and Hua XS (2018) An adversarial approach to hard triplet generation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 501–517

  96. Zhao JX, Liu JJ, Fan DP, Cao Y, Yang J and Cheng MM (2019) EGNet: edge guidance network for salient object detection. In proceedings of the IEEE international conference on computer vision (pp. 8779-8788).

  97. Zheng T, Deng W, (2018) Cross-pose lfw: a database for studying cross-pose face recognition in unconstrained environments. Beijing University of Posts and Telecommunications, Tech. Rep, 5.

  98. Zheng T, Deng W, Hu J (2017) Cross-age lfw: a database for studying cross-age face recognition in unconstrained environments. arXiv preprint arXiv:1708.08197.

  99. Zhou Y, Deng J, Kotsia I and Zafeiriou S (2019) Dense 3D face decoding over 2500FPS: Joint Texture & Shape Convolutional Mesh Decoders. arXiv preprint arXiv:1904.03525

  100. Zhu W, Zeng N, Wang N (2010) Sensitivity, specificity, accuracy, associated confidence interval and ROC analysis with practical SAS implementations. NESUG proceedings: health care and life sciences, Baltimore, Maryland 19:67

    Google Scholar 

  101. Zulqarnain Gilani S, Mian A (2018) Learning from millions of 3d scans for large-scale 3d face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp 1896-1905

Download references

Author information

Authors and Affiliations

  1. Computer Science and Engineering Department, Thapar Institute of Engineering and Technology, Patiala, India

    Sahil Sharma

  2. Computer Science and Engineering Department, National Institute of Technology, Hamirpur, India

    Vijay Kumar

Authors
  1. Sahil Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vijay Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sahil Sharma.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, S., Kumar, V. Voxel-based 3D occlusion-invariant face recognition using game theory and simulated annealing. Multimed Tools Appl 79, 26517–26547 (2020). https://doi.org/10.1007/s11042-020-09331-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09331-5

Keywords

Search

Navigation