Skip to main content

Advertisement

CG-MCFNet: cross-layer guidance-based multi-scale correlation fusion network for 3D face recognition

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

3D face recognition (FR) has been a popular field in recent years, which benefits from the advancement of 3D sensors and the application demands of video surveillance scenes. Existing 3D FR methods could show excellent performance when faces are complete. However, incomplete 3D faces, especially large poses and occluded, may prevent the model to learn effective and strong discriminative facial information adequately, resulting in unsatisfactory recognition results. To address this issue, we propose a cross-layer guidance-based multi-scale correlation fusion network (CG-MCFNet) for 3D FR. Firstly, we design a shallow feature enhancement extraction (SFE) module to obtain more effective facial detail information, and a deep feature enhancement extraction (DFE) module to learn more information with strong discrimination. Secondly, a novel multi-scale feature correlation fusion (MCF) module is proposed for fusing features from different layers, aiming to reduce the interference of redundant features and enhance the acquisition of discriminative features. Finally, the above three modules are integrated to form a new multi-scale local feature extraction (MLFE) module for capturing the face local information of rich and more strong discriminative. In addition, we introduce a global and local feature similarity weighted joint inference strategy, to further improve recognition accuracy. Extensive experiments on six challenging datasets, including three low-quality datasets (Lock3DFace, KinectFaceDB, and IIIT-D, where Lock3DFace is a video dataset), two high-quality datasets (UMB-DB, Bosphorus), and a cross-quality dataset synthesized by Bosphorus, prove that our CG-MCFNet achieves the best performance for incomplete 3D FR, which demonstrates the strong generalization ability of our model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The data used in this paper are all from public datasets.

References

  1. Alyuz N, Gokberk B, Akarun L (2013) 3-d face recognition under occlusion using masked projection. IEEE Trans Inf Forensics Secur 8(5):789–802

    Article  MATH  Google Scholar 

  2. Alyüz N, Gökberk B, Dibeklioğlu H, Savran A, Salah AA, Akarun L, Sankur B (2008) 3d face recognition benchmarks on the bosphorus database with focus on facial expressions. In: European workshop on biometrics and identity management, pp 57–66. Springer

  3. Cai Y, Lei Y, Yang M, You Z, Shan S (2019) A fast and robust 3d face recognition approach based on deeply learned face representation. Neurocomputing 363:375–397

    Article  MATH  Google Scholar 

  4. Cao H, Wang Y, Chen J, Jiang D, Zhang X, Tian Q, Wang M (2022) Swin-unet: Unet-like pure transformer for medical image segmentation. In: European conference on computer vision, pp. 205–218. Springer

  5. Cardia Neto JB (2020) 3d face recognition with descriptor images and shallow convolutional neural networks

  6. Changwei L, Jun Y, Lingyun Y, Yali L, Shengjin W (2020) Overview of research progress on 3-d face recognition. J Tsinghua Univ Sci Technol 61(1):77–88

    MATH  Google Scholar 

  7. Chiu MT, Cheng HY, Wang CY, Lai SH (2021) High-accuracy rgb-d face recognition via segmentation-aware face depth estimation and mask-guided attention network. In: 2021 16th IEEE International conference on automatic face and gesture recognition (FG 2021), pp 1–8. IEEE

  8. Colombo A, Cusano C, Schettini R (2011) Umb-db: A database of partially occluded 3d faces. In: 2011 IEEE international conference on computer vision workshops (ICCV workshops), pp 2113–2119. IEEE

  9. Cui J, Zhang H, Han H, Shan S, Chen X (2018) Improving 2d face recognition via discriminative face depth estimation. In: 2018 International conference on biometrics, pp 140–147. IEEE

  10. Dagnes N, Marcolin F, Nonis F, Tornincasa S, Vezzetti E (2019) 3d geometry-based face recognition in presence of eye and mouth occlusions. Int J Interact Des Manuf (IJIDeM) 13(4):1617–1635

    Article  MATH  Google Scholar 

  11. Deng X, Da F, Shao H, Jiang Y (2020) A multi-scale three-dimensional face recognition approach with sparse representation-based classifier and fusion of local covariance descriptors. Comput Electr Eng 85:106700

    Article  Google Scholar 

  12. Dutta K, Bhattacharjee D, Nasipuri M, Krejcar O (2021) Complement component face space for 3d face recognition from range images. Appl Intell 51(4):2500–2517

    Article  Google Scholar 

  13. ElSayed A, Kongar E, Mahmood A, Sobh T, Boult T (2018) Neural generative models for 3d faces with application in 3d texture free face recognition. arXiv preprint arXiv:1811.04358

  14. Fan Z, Wu X, Li C, Chen H, Liu W, Zheng Y, Chen J, Li X, Sun H, Jiang T et al (2023) Cam-vt: A weakly supervised cervical cancer nest image identification approach using conjugated attention mechanism and visual transformer. Comput Biol Med 162:107070

  15. Feng Z, Zhao Q (2018) Robust face recognition with deeply normalized depth images. In: Chinese conference on biometric recognition, pp 418–427. Springer

  16. Gilani SZ, Mian A (2018) Learning from millions of 3d scans for large-scale 3d face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1896–1905

  17. Goswami G, Bharadwaj S, Vatsa M, Singh R (2013) On rgb-d face recognition using kinect. In: 2013 IEEE Sixth international conference on biometrics: theory, applications and systems, pp 1–6. IEEE

  18. Goswami G, Vatsa M, Singh R (2014) Rgb-d face recognition with texture and attribute features. IEEE Trans Inf Forensics Secur 9(10):1629–1640

    Article  MATH  Google Scholar 

  19. Grati N, Ben-Hamadou A, Hammami M (2020) Learning local representations for scalable rgb-d face recognition. Expert Syst Appl 150:113319

    Article  MATH  Google Scholar 

  20. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  21. Hu Z, Zhao Q, Liu F (2019) Revisiting depth-based face recognition from a quality perspective. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 1–9

  22. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, pp 448–456. PMLR

  23. Jiang C, Lin S, Chen W, Liu F, Shen L (2021) Pointface: Point set based feature learning for 3d face recognition. In: 2021 IEEE International joint conference on biometrics, pp 1–8. IEEE

  24. Jiang C, Lin S, Chen W, Liu F, Shen L (2022) Pointface: Point cloud encoder based feature embedding for 3d face recognition. IEEE Transactions on Biometrics, Behavior, and Identity Science

  25. Katz S, Tal A, Basri R (2007) Direct visibility of point sets. In: ACM SIGGRAPH 2007 papers, pp 24–es

  26. Kim D, Hernandez M, Choi J, Medioni G (2017) Deep 3d face identification. In: 2017 IEEE international joint conference on biometrics (IJCB), pp 133–142. IEEE

  27. Križaj J, Dobrišek S, Štruc V (2022) Making the most of single sensor information: A novel fusion approach for 3d face recognition using region covariance descriptors and gaussian mixture models. Sensors 22(6):2388

    Article  Google Scholar 

  28. Lei Y, Guo Y, Hayat M, Bennamoun M, Zhou X (2016) A two-phase weighted collaborative representation for 3d partial face recognition with single sample. Pattern Recogn 52:218–237

    Article  MATH  Google Scholar 

  29. Li H, Huang D, Morvan JM, Wang Y, Chen L (2015) Towards 3d face recognition in the real: a registration-free approach using fine-grained matching of 3d keypoint descriptors. Int J Comput Vis 113(2):128–142

    Article  MathSciNet  MATH  Google Scholar 

  30. Liang Y, Liao JC, Pan J (2020) Mesh-based scale-invariant feature transform-like method for three-dimensional face recognition under expressions and missing data. J Electron Imaging 29(5):053008

    MATH  Google Scholar 

  31. Lin S, Jiang C, Liu F, Shen L (2021) High quality facial data synthesis and fusion for 3d low-quality face recognition. In: 2021 IEEE International joint conference on biometrics, pp 1–8. IEEE

  32. Liu F, Zhao Q, Liu X, Zeng D (2018) Joint face alignment and 3d face reconstruction with application to face recognition. IEEE Trans Pattern Anal Mach Intell 42(3):664–678

    Article  MATH  Google Scholar 

  33. Liu W, Li C, Xu N, Jiang T, Rahaman MM, Sun H, Wu X, Hu W, Chen H, Sun C et al (2022) Cvm-cervix: A hybrid cervical pap-smear image classification framework using cnn, visual transformer and multilayer perceptron. Pattern Recogn 130:108829

    Article  Google Scholar 

  34. Liu X, Zhu X, Li M, Wang L, Zhu E, Liu T, Kloft M, Shen D, Yin J, Gao W (2019) Multiple kernel \( k \) k-means with incomplete kernels. IEEE Trans Pattern Anal Mach Intell 42(5):1191–1204

    MATH  Google Scholar 

  35. Liu Z, Qiu Y, Peng Y, Pu J, Zhang X (2017) Quaternion based maximum margin criterion method for color face recognition. Neural Process Lett 45(3):913–923

    Article  MATH  Google Scholar 

  36. Mahmood SA, Ghani RF, Kerim AA (2014) 3d face recognition using pose invariant nose region detector. In: 2014 6th Computer science and electronic engineering conference, pp 103–108. IEEE

  37. Meden B, Rot P, Terhörst P, Damer N, Kuijper A, Scheirer WJ, Ross A, Peer P, Štruc V (2021) Privacy–enhancing face biometrics: A comprehensive survey. IEEE Transactions on Information Forensics and Security

  38. Min R, Kose N, Dugelay JL (2014) Kinectfacedb: A kinect database for face recognition. IEEE Trans Syst Man Cybern Syst 44(11):1534–1548

    Article  MATH  Google Scholar 

  39. Mu G, Huang D, Hu G, Sun J, Wang Y (2019) Led3d: A lightweight and efficient deep approach to recognizing low-quality 3d faces. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5773–5782

  40. Mu G, Huang D, Li W, Hu G, Wang Y (2021) Refining single low-quality facial depth map by lightweight and efficient deep model. In: 2021 IEEE International joint conference on biometrics (IJCB), pp 1–8. IEEE

  41. Neto JBC, Ferrari C, Marana AN, Berretti S, Del Bimbo A (2023) Learning streamed attention network from descriptor images for cross-resolution 3d face recognition. ACM Trans Multimedia Comput Commun Appl 19(1s):1–20

    Article  MATH  Google Scholar 

  42. Neto JBC, Marana AN, Ferrari C, Berretti S, Del Bimbo A (2019) Depth-based face recognition by learning from 3d-lbp images. In: 3DOR@ Eurographics, pp 55–62

  43. Niu W, Zhao Y, Yu Z, Liu Y, Gong Y (2023) Research on a face recognition algorithm based on 3d face data and 2d face image matching. J Vis Commun Image Represent 103757

  44. Phillips PJ, Flynn PJ, Scruggs T, Bowyer KW, Chang J, Hoffman K, Marques J, Min J, Worek W (2005) Overview of the face recognition grand challenge. In: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol 1, pp 947–954. IEEE

  45. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  46. Saoud A, Oumane A, Ouafi A, Taleb-Ahmed A (2020) Multimodal 2d+ 3d multi-descriptor tensor for face verification. Multimedia Tools Appl 79(31):23071–23092

    Article  Google Scholar 

  47. Savran A, Sankur B, Bilge MT (2012) Comparative evaluation of 3d vs. 2d modality for automatic detection of facial action units. Pattern Recogn 45(2):767–782

  48. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp 618–626

  49. Sharma S, Kumar V (2020) Voxel-based 3d face reconstruction and its application to face recognition using sequential deep learning. Multimedia Tools Appl 79(25):17303–17330

    Article  MATH  Google Scholar 

  50. Sharma S, Kumar V (2020) Voxel-based 3d occlusion-invariant face recognition using game theory and simulated annealing. Multimedia Tools Appl 79(35):26517–26547

    Article  MATH  Google Scholar 

  51. Sharma S, Kumar V (2021) 3d landmark-based face restoration for recognition using variational autoencoder and triplet loss. IET Biometrics 10(1):87–98

    Article  MATH  Google Scholar 

  52. Soltanpour S, Wu QMJ (2019) Weighted extreme sparse classifier and local derivative pattern for 3d face recognition. IEEE Trans Image Process 28(6):3020–3033

    Article  MathSciNet  MATH  Google Scholar 

  53. Tan Y, Lin H, Xiao Z, Ding S, Chao H (2019) Face recognition from sequential sparse 3d data via deep registration. In: 2019 International conference on biometrics (ICB), pp 1–8. IEEE

  54. Uppal H, Sepas-Moghaddam A, Greenspan M, Etemad A (2021) Depth as attention for face representation learning. IEEE Trans Inf Forensics Secur 16:2461–2476

    Article  Google Scholar 

  55. Wang H, Wang Y, Zhou Z, Ji X, Gong D, Zhou J, Li Z, Liu W (2018) Cosface: Large margin cosine loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5265–5274

  56. Wu W, Liu S, Xia Y, Zhang Y (2024) Dual residual attention network for image denoising. Pattern Recogn 149:110291

    Article  MATH  Google Scholar 

  57. Xiao S, Li S, Zhao Q (2021) Low-quality 3d face recognition with soft thresholding. In: Chinese conference on biometric recognition, pp 419–427. Springer

  58. Xiao X, Chen Y, Gong YJ, Zhou Y (2019) 2d quaternion sparse discriminant analysis. IEEE Trans Image Process 29:2271–2286

    Article  MathSciNet  MATH  Google Scholar 

  59. Yu C, Zhang Z, Li H, Sun J, Xu Z (2023) Meta-learning-based adversarial training for deep 3d face recognition on point clouds. Pattern Recogn 134:109065

    Article  MATH  Google Scholar 

  60. Yu X, Liang X, Zhou Z, Zhang B, Xue H (2024) Deep soft threshold feature separation network for infrared handprint identity recognition and time estimation. Infrared Phys Technol 138:105223

    Article  Google Scholar 

  61. Yu X, Lu Y, Gao Q (2021) Pipeline image diagnosis algorithm based on neural immune ensemble learning. Int J Press Vessel Pip 189:104249

    Article  Google Scholar 

  62. Yu X, Ye X, Zhang S (2022) Floating pollutant image target extraction algorithm based on immune extremum region. Digital Signal Process 123:103442

    Article  Google Scholar 

  63. Yu Y, Da F, Zhang Z (2022) Few-data guided learning upon end-to-end point cloud network for 3d face recognition. Multimedia Tools Appl 81(9):12795–12814

    Article  MATH  Google Scholar 

  64. Zhang J, Huang D, Wang Y, Sun J (2016) Lock3dface: A large-scale database of low-cost kinect 3d faces. In: 2016 International conference on biometrics (ICB), pp 1–8. IEEE

  65. Zhang J, Li C, Kosov S, Grzegorzek M, Shirahama K, Jiang T, Sun C, Li Z, Li H (2021) Lcu-net: A novel low-cost u-net for environmental microorganism image segmentation. Pattern Recogn 115:107885

    Article  Google Scholar 

  66. Zhang Z, Da F, Yu Y (2022) Learning directly from synthetic point clouds for “in-the-wild” 3d face recognition. Pattern Recogn 123:108394

  67. Zhang Z, Yu C, Xu S, Li H (2021) Learning flexibly distributional representation for low-quality 3d face recognition. Proc AAAI Conf Artif Intell 35:3465–3473

    MATH  Google Scholar 

  68. Zhao P, Ming Y, Hu N, Lyu B, Zhou J (2023) Dsnet: Dual-stream multi-scale fusion network for low-quality 3d face recognition. AIP Adv 13(8)

  69. Zhao P, Ming Y, Meng X, Yu H (2022) Lmfnet: A lightweight multiscale fusion network with hierarchical structure for low-quality 3-d face recognition. IEEE Trans Human-Mach Syst 53(1):239–252

    Article  MATH  Google Scholar 

  70. Zheng H, Wang W, Wen F, Liu P (2022) A complementary fusion strategy for rgb-d face recognition. In: International conference on multimedia modeling, pp 339–351. Springer

  71. Zhu K, He X, Lv Z, Zhang X, Hao R, He X, Wang J, He J, Zhang L, Mu Z (2023) A 3d occlusion facial recognition network based on a multi-feature combination threshold. Appl Sci 13(10):5950

    Article  MATH  Google Scholar 

  72. Zou C, Kou KI, Wang Y (2016) Quaternion collaborative and sparse representation with application to color face recognition. IEEE Trans Image Process 25(7):3287–3302

Download references

Acknowledgements

The work presented in this paper was partly supported by Natural Science Foundation of China (Grant No. 62076030), Beijing Natural Science Foundation (Grant No. L241011) and basic research fees of Beijing University of Posts and Telecommunications (Grant No. 2023ZCJH08).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yue Ming.

Ethics declarations

Competing Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, P., Ming, Y., Yu, H. et al. CG-MCFNet: cross-layer guidance-based multi-scale correlation fusion network for 3D face recognition. Appl Intell 55, 262 (2025). https://doi.org/10.1007/s10489-024-06221-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-06221-3

Keywords