Abstract
3D face recognition (FR) has been a popular field in recent years, which benefits from the advancement of 3D sensors and the application demands of video surveillance scenes. Existing 3D FR methods could show excellent performance when faces are complete. However, incomplete 3D faces, especially large poses and occluded, may prevent the model to learn effective and strong discriminative facial information adequately, resulting in unsatisfactory recognition results. To address this issue, we propose a cross-layer guidance-based multi-scale correlation fusion network (CG-MCFNet) for 3D FR. Firstly, we design a shallow feature enhancement extraction (SFE) module to obtain more effective facial detail information, and a deep feature enhancement extraction (DFE) module to learn more information with strong discrimination. Secondly, a novel multi-scale feature correlation fusion (MCF) module is proposed for fusing features from different layers, aiming to reduce the interference of redundant features and enhance the acquisition of discriminative features. Finally, the above three modules are integrated to form a new multi-scale local feature extraction (MLFE) module for capturing the face local information of rich and more strong discriminative. In addition, we introduce a global and local feature similarity weighted joint inference strategy, to further improve recognition accuracy. Extensive experiments on six challenging datasets, including three low-quality datasets (Lock3DFace, KinectFaceDB, and IIIT-D, where Lock3DFace is a video dataset), two high-quality datasets (UMB-DB, Bosphorus), and a cross-quality dataset synthesized by Bosphorus, prove that our CG-MCFNet achieves the best performance for incomplete 3D FR, which demonstrates the strong generalization ability of our model.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability
The data used in this paper are all from public datasets.
References
Alyuz N, Gokberk B, Akarun L (2013) 3-d face recognition under occlusion using masked projection. IEEE Trans Inf Forensics Secur 8(5):789–802
Alyüz N, Gökberk B, Dibeklioğlu H, Savran A, Salah AA, Akarun L, Sankur B (2008) 3d face recognition benchmarks on the bosphorus database with focus on facial expressions. In: European workshop on biometrics and identity management, pp 57–66. Springer
Cai Y, Lei Y, Yang M, You Z, Shan S (2019) A fast and robust 3d face recognition approach based on deeply learned face representation. Neurocomputing 363:375–397
Cao H, Wang Y, Chen J, Jiang D, Zhang X, Tian Q, Wang M (2022) Swin-unet: Unet-like pure transformer for medical image segmentation. In: European conference on computer vision, pp. 205–218. Springer
Cardia Neto JB (2020) 3d face recognition with descriptor images and shallow convolutional neural networks
Changwei L, Jun Y, Lingyun Y, Yali L, Shengjin W (2020) Overview of research progress on 3-d face recognition. J Tsinghua Univ Sci Technol 61(1):77–88
Chiu MT, Cheng HY, Wang CY, Lai SH (2021) High-accuracy rgb-d face recognition via segmentation-aware face depth estimation and mask-guided attention network. In: 2021 16th IEEE International conference on automatic face and gesture recognition (FG 2021), pp 1–8. IEEE
Colombo A, Cusano C, Schettini R (2011) Umb-db: A database of partially occluded 3d faces. In: 2011 IEEE international conference on computer vision workshops (ICCV workshops), pp 2113–2119. IEEE
Cui J, Zhang H, Han H, Shan S, Chen X (2018) Improving 2d face recognition via discriminative face depth estimation. In: 2018 International conference on biometrics, pp 140–147. IEEE
Dagnes N, Marcolin F, Nonis F, Tornincasa S, Vezzetti E (2019) 3d geometry-based face recognition in presence of eye and mouth occlusions. Int J Interact Des Manuf (IJIDeM) 13(4):1617–1635
Deng X, Da F, Shao H, Jiang Y (2020) A multi-scale three-dimensional face recognition approach with sparse representation-based classifier and fusion of local covariance descriptors. Comput Electr Eng 85:106700
Dutta K, Bhattacharjee D, Nasipuri M, Krejcar O (2021) Complement component face space for 3d face recognition from range images. Appl Intell 51(4):2500–2517
ElSayed A, Kongar E, Mahmood A, Sobh T, Boult T (2018) Neural generative models for 3d faces with application in 3d texture free face recognition. arXiv preprint arXiv:1811.04358
Fan Z, Wu X, Li C, Chen H, Liu W, Zheng Y, Chen J, Li X, Sun H, Jiang T et al (2023) Cam-vt: A weakly supervised cervical cancer nest image identification approach using conjugated attention mechanism and visual transformer. Comput Biol Med 162:107070
Feng Z, Zhao Q (2018) Robust face recognition with deeply normalized depth images. In: Chinese conference on biometric recognition, pp 418–427. Springer
Gilani SZ, Mian A (2018) Learning from millions of 3d scans for large-scale 3d face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1896–1905
Goswami G, Bharadwaj S, Vatsa M, Singh R (2013) On rgb-d face recognition using kinect. In: 2013 IEEE Sixth international conference on biometrics: theory, applications and systems, pp 1–6. IEEE
Goswami G, Vatsa M, Singh R (2014) Rgb-d face recognition with texture and attribute features. IEEE Trans Inf Forensics Secur 9(10):1629–1640
Grati N, Ben-Hamadou A, Hammami M (2020) Learning local representations for scalable rgb-d face recognition. Expert Syst Appl 150:113319
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
Hu Z, Zhao Q, Liu F (2019) Revisiting depth-based face recognition from a quality perspective. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 1–9
Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, pp 448–456. PMLR
Jiang C, Lin S, Chen W, Liu F, Shen L (2021) Pointface: Point set based feature learning for 3d face recognition. In: 2021 IEEE International joint conference on biometrics, pp 1–8. IEEE
Jiang C, Lin S, Chen W, Liu F, Shen L (2022) Pointface: Point cloud encoder based feature embedding for 3d face recognition. IEEE Transactions on Biometrics, Behavior, and Identity Science
Katz S, Tal A, Basri R (2007) Direct visibility of point sets. In: ACM SIGGRAPH 2007 papers, pp 24–es
Kim D, Hernandez M, Choi J, Medioni G (2017) Deep 3d face identification. In: 2017 IEEE international joint conference on biometrics (IJCB), pp 133–142. IEEE
Križaj J, Dobrišek S, Štruc V (2022) Making the most of single sensor information: A novel fusion approach for 3d face recognition using region covariance descriptors and gaussian mixture models. Sensors 22(6):2388
Lei Y, Guo Y, Hayat M, Bennamoun M, Zhou X (2016) A two-phase weighted collaborative representation for 3d partial face recognition with single sample. Pattern Recogn 52:218–237
Li H, Huang D, Morvan JM, Wang Y, Chen L (2015) Towards 3d face recognition in the real: a registration-free approach using fine-grained matching of 3d keypoint descriptors. Int J Comput Vis 113(2):128–142
Liang Y, Liao JC, Pan J (2020) Mesh-based scale-invariant feature transform-like method for three-dimensional face recognition under expressions and missing data. J Electron Imaging 29(5):053008
Lin S, Jiang C, Liu F, Shen L (2021) High quality facial data synthesis and fusion for 3d low-quality face recognition. In: 2021 IEEE International joint conference on biometrics, pp 1–8. IEEE
Liu F, Zhao Q, Liu X, Zeng D (2018) Joint face alignment and 3d face reconstruction with application to face recognition. IEEE Trans Pattern Anal Mach Intell 42(3):664–678
Liu W, Li C, Xu N, Jiang T, Rahaman MM, Sun H, Wu X, Hu W, Chen H, Sun C et al (2022) Cvm-cervix: A hybrid cervical pap-smear image classification framework using cnn, visual transformer and multilayer perceptron. Pattern Recogn 130:108829
Liu X, Zhu X, Li M, Wang L, Zhu E, Liu T, Kloft M, Shen D, Yin J, Gao W (2019) Multiple kernel \( k \) k-means with incomplete kernels. IEEE Trans Pattern Anal Mach Intell 42(5):1191–1204
Liu Z, Qiu Y, Peng Y, Pu J, Zhang X (2017) Quaternion based maximum margin criterion method for color face recognition. Neural Process Lett 45(3):913–923
Mahmood SA, Ghani RF, Kerim AA (2014) 3d face recognition using pose invariant nose region detector. In: 2014 6th Computer science and electronic engineering conference, pp 103–108. IEEE
Meden B, Rot P, Terhörst P, Damer N, Kuijper A, Scheirer WJ, Ross A, Peer P, Štruc V (2021) Privacy–enhancing face biometrics: A comprehensive survey. IEEE Transactions on Information Forensics and Security
Min R, Kose N, Dugelay JL (2014) Kinectfacedb: A kinect database for face recognition. IEEE Trans Syst Man Cybern Syst 44(11):1534–1548
Mu G, Huang D, Hu G, Sun J, Wang Y (2019) Led3d: A lightweight and efficient deep approach to recognizing low-quality 3d faces. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5773–5782
Mu G, Huang D, Li W, Hu G, Wang Y (2021) Refining single low-quality facial depth map by lightweight and efficient deep model. In: 2021 IEEE International joint conference on biometrics (IJCB), pp 1–8. IEEE
Neto JBC, Ferrari C, Marana AN, Berretti S, Del Bimbo A (2023) Learning streamed attention network from descriptor images for cross-resolution 3d face recognition. ACM Trans Multimedia Comput Commun Appl 19(1s):1–20
Neto JBC, Marana AN, Ferrari C, Berretti S, Del Bimbo A (2019) Depth-based face recognition by learning from 3d-lbp images. In: 3DOR@ Eurographics, pp 55–62
Niu W, Zhao Y, Yu Z, Liu Y, Gong Y (2023) Research on a face recognition algorithm based on 3d face data and 2d face image matching. J Vis Commun Image Represent 103757
Phillips PJ, Flynn PJ, Scruggs T, Bowyer KW, Chang J, Hoffman K, Marques J, Min J, Worek W (2005) Overview of the face recognition grand challenge. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol 1, pp 947–954. IEEE
Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520
Saoud A, Oumane A, Ouafi A, Taleb-Ahmed A (2020) Multimodal 2d+ 3d multi-descriptor tensor for face verification. Multimedia Tools Appl 79(31):23071–23092
Savran A, Sankur B, Bilge MT (2012) Comparative evaluation of 3d vs. 2d modality for automatic detection of facial action units. Pattern Recogn 45(2):767–782
Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp 618–626
Sharma S, Kumar V (2020) Voxel-based 3d face reconstruction and its application to face recognition using sequential deep learning. Multimedia Tools Appl 79(25):17303–17330
Sharma S, Kumar V (2020) Voxel-based 3d occlusion-invariant face recognition using game theory and simulated annealing. Multimedia Tools Appl 79(35):26517–26547
Sharma S, Kumar V (2021) 3d landmark-based face restoration for recognition using variational autoencoder and triplet loss. IET Biometrics 10(1):87–98
Soltanpour S, Wu QMJ (2019) Weighted extreme sparse classifier and local derivative pattern for 3d face recognition. IEEE Trans Image Process 28(6):3020–3033
Tan Y, Lin H, Xiao Z, Ding S, Chao H (2019) Face recognition from sequential sparse 3d data via deep registration. In: 2019 International conference on biometrics (ICB), pp 1–8. IEEE
Uppal H, Sepas-Moghaddam A, Greenspan M, Etemad A (2021) Depth as attention for face representation learning. IEEE Trans Inf Forensics Secur 16:2461–2476
Wang H, Wang Y, Zhou Z, Ji X, Gong D, Zhou J, Li Z, Liu W (2018) Cosface: Large margin cosine loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5265–5274
Wu W, Liu S, Xia Y, Zhang Y (2024) Dual residual attention network for image denoising. Pattern Recogn 149:110291
Xiao S, Li S, Zhao Q (2021) Low-quality 3d face recognition with soft thresholding. In: Chinese conference on biometric recognition, pp 419–427. Springer
Xiao X, Chen Y, Gong YJ, Zhou Y (2019) 2d quaternion sparse discriminant analysis. IEEE Trans Image Process 29:2271–2286
Yu C, Zhang Z, Li H, Sun J, Xu Z (2023) Meta-learning-based adversarial training for deep 3d face recognition on point clouds. Pattern Recogn 134:109065
Yu X, Liang X, Zhou Z, Zhang B, Xue H (2024) Deep soft threshold feature separation network for infrared handprint identity recognition and time estimation. Infrared Phys Technol 138:105223
Yu X, Lu Y, Gao Q (2021) Pipeline image diagnosis algorithm based on neural immune ensemble learning. Int J Press Vessel Pip 189:104249
Yu X, Ye X, Zhang S (2022) Floating pollutant image target extraction algorithm based on immune extremum region. Digital Signal Process 123:103442
Yu Y, Da F, Zhang Z (2022) Few-data guided learning upon end-to-end point cloud network for 3d face recognition. Multimedia Tools Appl 81(9):12795–12814
Zhang J, Huang D, Wang Y, Sun J (2016) Lock3dface: A large-scale database of low-cost kinect 3d faces. In: 2016 International conference on biometrics (ICB), pp 1–8. IEEE
Zhang J, Li C, Kosov S, Grzegorzek M, Shirahama K, Jiang T, Sun C, Li Z, Li H (2021) Lcu-net: A novel low-cost u-net for environmental microorganism image segmentation. Pattern Recogn 115:107885
Zhang Z, Da F, Yu Y (2022) Learning directly from synthetic point clouds for “in-the-wild” 3d face recognition. Pattern Recogn 123:108394
Zhang Z, Yu C, Xu S, Li H (2021) Learning flexibly distributional representation for low-quality 3d face recognition. Proc AAAI Conf Artif Intell 35:3465–3473
Zhao P, Ming Y, Hu N, Lyu B, Zhou J (2023) Dsnet: Dual-stream multi-scale fusion network for low-quality 3d face recognition. AIP Adv 13(8)
Zhao P, Ming Y, Meng X, Yu H (2022) Lmfnet: A lightweight multiscale fusion network with hierarchical structure for low-quality 3-d face recognition. IEEE Trans Human-Mach Syst 53(1):239–252
Zheng H, Wang W, Wen F, Liu P (2022) A complementary fusion strategy for rgb-d face recognition. In: International conference on multimedia modeling, pp 339–351. Springer
Zhu K, He X, Lv Z, Zhang X, Hao R, He X, Wang J, He J, Zhang L, Mu Z (2023) A 3d occlusion facial recognition network based on a multi-feature combination threshold. Appl Sci 13(10):5950
Zou C, Kou KI, Wang Y (2016) Quaternion collaborative and sparse representation with application to color face recognition. IEEE Trans Image Process 25(7):3287–3302
Acknowledgements
The work presented in this paper was partly supported by Natural Science Foundation of China (Grant No. 62076030), Beijing Natural Science Foundation (Grant No. L241011) and basic research fees of Beijing University of Posts and Telecommunications (Grant No. 2023ZCJH08).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing Interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhao, P., Ming, Y., Yu, H. et al. CG-MCFNet: cross-layer guidance-based multi-scale correlation fusion network for 3D face recognition. Appl Intell 55, 262 (2025). https://doi.org/10.1007/s10489-024-06221-3
Accepted:
Published:
DOI: https://doi.org/10.1007/s10489-024-06221-3