Abstract
A novel voxel-based occlusion-invariant 3D face recognition framework (V3DOFR) based on game theory and simulated annealing is proposed. In V3DOFR approach, 3D meshes are converted to voxel form of sizes 43, 83, and 163. After that, locality preserving projection-based embeddings are computed for removing the sparseness of voxels and generating consistent linear embedding per mesh with size 64 × 3, 128 × 3, and 256 × 3, respectively. The generator of triplets provides the triplets of sizes 64x3x3, 128x3x3, and 256x3x3. The simulated annealing is used to check the threshold value of adversarial triplet loss generated after ensembling losses of different grid sizes. The proposed framework is compared with four well-known methods using three face datasets, namely, Bosphorus, UMBDB, and KinectFaceDB. The performance evaluation has been done using four different cases of experimentations, viz. voxel based face recognition, occlusion invariant face recognition, landmarks based 3D face recognition, and 3D mesh based face recognition. Seven evaluation metrics are used to compare the proposed technique with other methods. The proposed method provides better accuracy and computation time over the other existing techniques in the majority of cases.
Similar content being viewed by others
References
Abrevaya VF, Boukhayma A, Wuhrer S and Boyer E(2019) A decoupled 3D facial shape model by adversarial training. In proceedings of the IEEE international conference on computer vision (pp. 9419-9428).
Alom, M.Z., Taha, T.M., Yakopcic, C., Westberg, S., Sidike, P., Nasrin, M.S., Van Esesn, B.C., Awwal, A.A.S. and Asari, V.K. (2018) The history began from AlexNet: a comprehensive survey on deep learning approaches. arXiv preprint arXiv:1803.01164
Alom MZ, Taha TM, Yakopcic C, Westberg S, Sidike P, Nasrin MS, Hasan M, Van Essen BC, Awwal AA, Asari VK (2019) A state-of-the-art survey on deep learning theory and architectures. Electronics 8(3):292
Antipov G, Baccouche M and Dugelay JL (2017) Face aging with conditional generative adversarial networks. In 2017 IEEE International Conference on Image Processing (ICIP). pp. 2089–2093. IEEE
Bai S, Zhou Z, Wang J, Bai X, Jan Latecki L, Tian Q (2017) Ensemble diffusion for retrieval. In Proceedings of the IEEE International Conference on Computer Vision. pp. 774–783
Bai S, Bai X, Tian Q, Latecki LJ (2017) Regularized diffusion process for visual retrieval. In Thirty-First AAAI Conference on Artificial Intelligence
Bandyopadhyay S, Maulik U, Pakhira MK (2001) Clustering using simulated annealing with probabilistic redistribution. Int J Pattern Recognit Artif Intell 15(02):269–285
Bandyopadhyay S, Saha S, Maulik U, Deb K (2008) A simulated annealing-based multiobjective optimization algorithm: AMOSA. IEEE Trans Evol Comput 12(3):269–283
Belkin M, Niyogi P (2002) Laplacian eigenmaps and spectral techniques for embedding and clustering. In Advances in neural information processing systems. pp. 585–591
Bi H, Li N, Guan H, Lu D and Yang L (2019, September) A multi-scale conditional generative adversarial network for face sketch synthesis. In 2019 IEEE international conference on image processing (ICIP) (pp. 3876-3880). IEEE.
Bowyer KW, Chang K, Flynn P (2004) A survey of 3D and multi-modal 3D+ 2D face recognition.
CASIA-3D FaceV1, 3d face database
Caves R, Quegan S, White R (1998) Quantitative comparison of the performance of SAR segmentation algorithms. IEEE Trans Image Process 7(11):1534–1546
Chen Y, Garcia EK, Gupta MR, Rahimi A, Cazzanti L (2009) Similarity-based classification: concepts and algorithms. J Mach Learn Res 10:747–776
Chen W, Chen X, Zhang J and Huang K (2017) Beyond triplet loss: a deep quadruplet network for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 403–412
Cho M, Kim T, Kim IJ and Lee S (2020) Relational deep feature learning for heterogeneous face recognition. arXiv preprint arXiv:2003.00697.
Colombo A, Cusano C, Schettini R (2011) UMB-DB: A database of partially occluded 3D faces. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops). pp. 2113–2119. IEEE
Deng J, Guo J, Xue N, Zafeiriou S (2019) Arcface: additive angular margin loss for deep face recognition. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4690-4699).
Ding C, Tao D (2016) A comprehensive survey on pose-invariant face recognition. ACM Transactions on intelligent systems and technology (TIST) 7(3):1–42
Do TT, Tran T, Reid I, Kumar V, Hoang T, Carneiro G (2019) A theoretically sound upper bound on the triplet loss for improving the efficiency of deep distance metric learning. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 10404-10413).
Dong Y, Zhang Z, Hong WC (2018) A hybrid seasonal mechanism with a chaotic cuckoo search algorithm with a support vector regression model for electric load forecasting. Energies 11(4):1009
Dou P, Shah SK and Kakadiaris IA (2017) End-to-end 3D face reconstruction with deep neural networks. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5908-5917).
El Sayed AR, El Chakik A, Alabboud H, Yassine A (2018) Efficient 3D point clouds classification for face detection using linear programming and data mining. The Imaging Science Journal 66(1):23–37
El Sayed A, Kongar E, Mahmood A, Sobh T and Boult T (2018) Neural generative models for 3D faces with application in 3D texture free face recognition. arXiv preprint arXiv:1811.04358
Faltemier TC, Bowyer KW and Flynn PJ (2007) Using a multi-instance enrollment representation to improve 3D face recognition. In 2007 First IEEE International Conference on Biometrics: Theory, Applications, and Systems. pp. 1–6. IEEE
Fan DP, Cheng MM, Liu JJ, Gao SH, Hou Q and Borji A (2018) Salient objects in clutter: bringing salient object detection to the foreground. In proceedings of the European conference on computer vision (ECCV) (pp. 186-202).
Fan DP, Zhang S, Wu YH, Liu Y, Cheng MM, Ren B, Rosin PL and Ji R (2019) Scoot: a perceptual metric for facial sketches. In proceedings of the IEEE international conference on computer vision (pp. 5612-5622).
Gecer B, Ploumpis S, Kotsia I and Zafeiriou S (2019) GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction. arXiv preprint arXiv:1902.05978
Goodfellow I (2016) NIPS 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In Advances in neural information processing systems:2672–2680
Hassaballah M, Aly S (2015) Face recognition: challenges, achievements and future directions. IET Comput Vis 9(4):614–626
He X (2005) Locality preserving projections. The University of Chicago, A dissertation submitted to the faculty of the division of the physical sciences in candidacy for the degree of doctor of philosophy Department of Computer Science
He X, Niyogi, P (2004) Locality preserving projections. In Advances in neural information processing systems pp. 153–160
He Z, Zuo W, Kan M, Shan S, Chen X (2019) Attgan: facial attribute editing by only changing what you want. IEEE Trans Image Process 28:5464–5478
Hong WC, Dong Y, Lai CY, Chen LY, Wei SY (2011) SVR with hybrid chaotic immune algorithm for seasonal load demand forecasting. Energies 4(6):960–977
Hong WC, Li MW, Geng J, Zhang Y (2019) Novel chaotic bat algorithm for forecasting complex motion of floating platforms. Appl Math Model 72:425–443
Hossin M, Sulaiman MN (2015) A review on evaluation metrics for data classification evaluations. International Journal of Data Mining & Knowledge Management Process 5(2):1
Huang Y, Wang Y, Tai Y, Liu X, Shen P, Li S, Li J and Huang F (2020) CurricularFace: adaptive curriculum learning loss for deep face recognition. arXiv preprint arXiv:2004.00288.
Jiao Y, Niu Y, Zhang Y, Li F, Zou C, Shi G (2019, December) Facial attention based convolutional neural network for 2D+ 3D facial expression recognition. In 2019 IEEE visual communications and image processing (VCIP) (pp. 1-4). IEEE.
Kemelmacher-Shlizerman I, Seitz SM, Miller D and Brossard E (2016) The megaface benchmark: 1 million faces for recognition at scale. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4873-4882).
Kim D, Hernandez M, Choi J and Medioni G (2017) Deep 3D face identification. In 2017 IEEE International Joint Conference on Biometrics (IJCB) pp. 133–142. IEEE
Kim D, Hernandez M, Choi J, Medioni G (2017) Deep 3D face identification. In 2017 IEEE International Joint Conference on Biometrics (IJCB). pp. 133–142. IEEE
Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
Kirkpatrick S, Gelatt CD, Vecchi MP (1983) Optimization by simulated annealing. science, 220(4598), pp.671–680
Korshunov P and Marcel S (2018) DeepFakes: a new threat to face recognition? Assessment and Detection. arXiv preprint arXiv:1812.08685
Larsen, A.B.L., Sønderby, S.K., Larochelle, H. & Winther, O. (2016) Autoencoding beyond pixels using a learned similarity metric. Proceedings of The 33rd International Conference on Machine Learning, in PMLR 48:1558–1566
Learned-Miller, E., Huang, G.B., RoyChowdhury, A., Li, H. and Hua, G., 2016. Labeled faces in the wild: a survey. In advances in face detection and facial image analysis (pp. 189-248). Springer, Cham.
Lei Y, Guo Y, Hayat M, Bennamoun M, Zhou X (2016) A two-phase weighted collaborative representation for 3D partial face recognition with single sample. Pattern Recogn 52:218–237
Li H, Huang D, Morvan JM, Chen L, Wang Y (2014) Expression-robust 3D face recognition via weighted sparse representation of multi-scale and multi-component local normal patterns. Neurocomputing 133:179–193
Li H, Huang D, Morvan JM, Wang Y, Chen L (2015) Towards 3D face recognition in the real: a registration-free approach using fine-grained matching of 3D keypoint descriptors. Int J Comput Vis 113(2):128–142
Liu F, Zhao Q, Zeng D (2018) Joint face alignment and 3D face reconstruction with application to face recognition. IEEE Trans Pattern Anal Mach Intell
Maulik U, Bandyopadhyay S, Trinder JC (2001) SAFE: an efficient feature extraction technique. Knowl Inf Syst 3(3):374–387
Maze B, Adams J, Duncan JA, Kalka N, Miller T, Otto C, Jain AK, Niggel WT, Anderson J, Cheney J and Grother P, (2018, February) Iarpa janus benchmark-c: face dataset and protocol. In 2018 international conference on biometrics (ICB) (pp. 158-165). IEEE.
Min R, Kose N, Dugelay JL (2014) Kinectfacedb: a kinect database for face recognition. IEEE Transactions on Systems, Man, and Cybernetics: Systems 44(11):1534–1548
Moreno A (2004) GavabDB: a 3D face database. In Proc. 2nd COST275 workshop on biometrics on the internet, 2004 (pp. 75-80).
Moschoglou S, Papaioannou A, Sagonas C, Deng J, Kotsia I, Zafeiriou S (2017) Agedb: the first manually collected, in-the-wild age database. In proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 51-59).
ND-2006 Face Data Set. http://www.nd.edu/˜cvrl/. 2007.
Ogáyar CJ, Rueda AJ, Segura RJ, Feito FR (2007) Fast and simple hardware accelerated voxelizations using simplicial coverings. Vis Comput 23(8):535–543
Oh Song H, Xiang Y, Jegelka S, Savarese S (2016) Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4004–4012
Pantaleoni J (2011) VoxelPipe: a programmable pipeline for 3D voxelization. In Proceedings of the ACM SIGGRAPH Symposium on High Performance Graphics. pp. 99–106. ACM
Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In British Machine Vision Conference (BMVC) 1(3):6
Patil H, Kothari A, Bhurchandi K (2015) 3-D face recognition: features, databases, algorithms and challenges. Artif Intell Rev 44(3):393–441
Perarnau G, Van De Weijer J, Raducanu B and Álvarez JM (2016) Invertible conditional gans for image editing. arXiv preprint arXiv:1611.06355
Pham HX, Chen C, Dao LN, Pavlovic V, Cai J and Cham TJ (2015) Robust performance-driven 3d face tracking in long range depth scenes. arXiv preprint arXiv:1507.02779
Phillips PJ, Moon H, Rizvi SA, Rauss PJ (2000) The FERET evaluation methodology for face-recognition algorithms. IEEE Trans Pattern Anal Mach Intell 22(10):1090–1104
Phillips PJ, Flynn PJ, Scruggs T, Bowyer KW, Chang J, Hoffman K, Marques J, Min J, Worek W (2005, June) Overview of the face recognition grand challenge. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05) (Vol. 1, pp. 947-954). IEEE.
Ranjan A, Bolkart T, Sanyal S, Black MJ (2018) Generating 3D faces using convolutional mesh autoencoders. In proceedings of the European conference on computer vision (ECCV) (pp. 704-720).
Rathgeb C, Dantcheva A, Busch C (2019) Impact and detection of facial beautification in face recognition: an overview. IEEE Access 7:152667–152678
Salimans T, Goodfellow I, Zaremba W, Cheung V, Radford A and Chen X (2016) Improved techniques for training gans. In Advances in neural information processing systems. pp. 2234–2242
Sanderson C (2002) The vidtimit database (No. REP_WORK). IDIAP
Savran A, Alyüz N, Dibeklioğlu H, Çeliktutan O, Gökberk B, Sankur B and Akarun L (2008) Bosphorus database for 3D face analysis. In European Workshop on Biometrics and Identity Management. pp. 47–56. Springer, Berlin, Heidelberg
Scherhag U, Rathgeb C, Merkle J, Breithaupt R, Busch C (2019) Face recognition systems under morphing attacks: a survey. IEEE Access 7:23012–23026
Schroff F, Kalenichenko D and Philbin J (2015) Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 815–823
Sengupta S, Chen JC, Castillo C, Patel VM, Chellappa R and Jacobs DW (2016, March) Frontal to profile face verification in the wild. In 2016 IEEE winter conference on applications of computer vision (WACV) (pp. 1-9). IEEE.
Sharma S, Kumar V (2020) Voxel-based 3D face reconstruction and its application to face recognition using sequential deep learning. Multimedia tools and applications, pp.1-28.
Spreeuwers L (2011) Fast and accurate 3d face recognition. Int J Comput Vis 93(3):389–414
Tan Y, Lin H, Xiao Z, Ding S and Chao H (2018) Face recognition from sequential sparse 3D data via deep registration. arXiv preprint arXiv:1810.09658
Vijayan V, Bowyer KW, Flynn PJ, Huang D, Chen L, Hansen M, Ocegueda O, Shah SK, Kakadiaris IA (2011, October) Twins 3D face recognition challenge. In 2011 international joint conference on biometrics (IJCB) (pp. 1-7). IEEE.
Wang X, Tang X (2008) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31(11):1955–1967
Whitelam C, Taborsky E, Blanton A, Maze B, Adams J, Miller T, Kalka N, Jain AK, Duncan JA, Allen K and Cheney J (2017) Iarpa janus benchmark-b face dataset. In proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 90-98).
Wu Z, Song S, Khosla A, Tang X, Xiao J (2014) 3D Shapenets for 2.5D object recognition and next-best-view prediction. arXiv preprint arXiv:1406.5670, 2(4)
Xu D, Hu P, Cao W, Li H (2008, June) SHREC’08 entry: 3D face recognition using moment invariants. In 2008 IEEE international conference on shape modeling and applications (pp. 261-262). IEEE.
Yang XS (2010) Nature-inspired metaheuristic algorithms. Luniver press
Yi D, Lei Z, Liao S, Li SZ (2014) Learning face representation from scratch. arXiv preprint arXiv:1411.7923.
Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006, April) A 3D facial expression database for facial behavior research. In 7th international conference on automatic face and gesture recognition (FGR06) (pp. 211-216). IEEE.
Yin L, Sun\ XCY, Worm T and Reale M (2008) A high-resolution 3d dynamic facial expression database. In IEEE International Conference on Automatic Face and Gesture Recognition, Amsterdam, The Netherlands. 126
Zhang Z, Hong WC (2019) Electric load forecasting by complete ensemble empirical mode decomposition adaptive noise and support vector regression with quantum-based dragonfly algorithm. Nonlinear Dynamics 98(2):1107–1136
Zhang W, Wang X, Tang X (2011, June) Coupled information-theoretic encoding for face photo-sketch recognition. In CVPR 2011 (pp. 513-520). IEEE.
Zhang X, Yin L, Cohn JF, Canavan S, Reale M, Horowitz A and Liu P (2013, April) A high-resolution spontaneous 3d dynamic facial expression database. In 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG) (pp. 1-6). IEEE.
Zhang X, Yin L, Cohn JF, Canavan S, Reale M, Horowitz A, Liu P, Girard JM (2014) Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database. Image Vis Comput 32(10):692–706
Zhang Y, Zhang L, Neoh SC, Mistry K, Hossain MA (2015) Intelligent affect regression for bodily expressions using hybrid particle swarm optimization and adaptive ensembles. Expert Syst Appl 42(22):8678–8697
Zhang Z, Hong WC, Li J (2020) Electric load forecasting by hybrid self-recurrent support vector regression model with variational mode decomposition and improved cuckoo search algorithm. IEEE Access 8:14642–14658
Zhang J, Yu X, Li A, Song P, Liu B and Dai Y (2020) Weakly-supervised salient object detection via scribble annotations. arXiv preprint arXiv:2003.07685.
Zhang J, Fan DP, Dai Y, Anwar S, Saleh FS, Zhang T and Barnes N (2020) UC-net: uncertainty inspired rgb-d saliency detection via conditional variational autoencoders. arXiv preprint arXiv:2004.05763.
Zhao Y, Jin Z, Qi GJ, Lu H and Hua XS (2018) An adversarial approach to hard triplet generation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 501–517
Zhao JX, Liu JJ, Fan DP, Cao Y, Yang J and Cheng MM (2019) EGNet: edge guidance network for salient object detection. In proceedings of the IEEE international conference on computer vision (pp. 8779-8788).
Zheng T, Deng W, (2018) Cross-pose lfw: a database for studying cross-pose face recognition in unconstrained environments. Beijing University of Posts and Telecommunications, Tech. Rep, 5.
Zheng T, Deng W, Hu J (2017) Cross-age lfw: a database for studying cross-age face recognition in unconstrained environments. arXiv preprint arXiv:1708.08197.
Zhou Y, Deng J, Kotsia I and Zafeiriou S (2019) Dense 3D face decoding over 2500FPS: Joint Texture & Shape Convolutional Mesh Decoders. arXiv preprint arXiv:1904.03525
Zhu W, Zeng N, Wang N (2010) Sensitivity, specificity, accuracy, associated confidence interval and ROC analysis with practical SAS implementations. NESUG proceedings: health care and life sciences, Baltimore, Maryland 19:67
Zulqarnain Gilani S, Mian A (2018) Learning from millions of 3d scans for large-scale 3d face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp 1896-1905
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sharma, S., Kumar, V. Voxel-based 3D occlusion-invariant face recognition using game theory and simulated annealing. Multimed Tools Appl 79, 26517–26547 (2020). https://doi.org/10.1007/s11042-020-09331-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-09331-5