Skip to main content
Springer Nature Link
Log in

Towards 3D Face Recognition in the Real: A Registration-Free Approach Using Fine-Grained Matching of 3D Keypoint Descriptors

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Registration algorithms performed on point clouds or range images of face scans have been successfully used for automatic 3D face recognition under expression variations, but have rarely been investigated to solve pose changes and occlusions mainly since that the basic landmarks to initialize coarse alignment are not always available. Recently, local feature-based SIFT-like matching proves competent to handle all such variations without registration. In this paper, towards 3D face recognition for real-life biometric applications, we significantly extend the SIFT-like matching framework to mesh data and propose a novel approach using fine-grained matching of 3D keypoint descriptors. First, two principal curvature-based 3D keypoint detectors are provided, which can repeatedly identify complementary locations on a face scan where local curvatures are high. Then, a robust 3D local coordinate system is built at each keypoint, which allows extraction of pose-invariant features. Three keypoint descriptors, corresponding to three surface differential quantities, are designed, and their feature-level fusion is employed to comprehensively describe local shapes of detected keypoints. Finally, we propose a multi-task sparse representation based fine-grained matching algorithm, which accounts for the average reconstruction error of probe face descriptors sparsely represented by a large dictionary of gallery descriptors in identification. Our approach is evaluated on the Bosphorus database and achieves rank-one recognition rates of 96.56, 98.82, 91.14, and 99.21 % on the entire database, and the expression, pose, and occlusion subsets, respectively. To the best of our knowledge, these are the best results reported so far on this database. Additionally, good generalization ability is also exhibited by the experiments on the FRGC v2.0 database.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. http://www.csse.uwa.edu.au/~ajmal/code.html.

  2. https://perswww.kuleuven.be/~u0059456/meshSIFT.html.

References

  • Al-Osaimi, F., Bennamoun, M., & Mian, A. (2009). An expression deformation approach to non-rigid 3d face recognition. International Journal of Computer Vision, 81(3), 302–316.

    Article  Google Scholar 

  • Alyüz, N., Gökberk. B., & Akarun. L. (2008). A 3d face recognition system for expression and occlusion invariance. In: 2nd IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS) (pp 1–7).

  • Alyüz, N., Gökberk, B., & Akarun, L. (2010). Regional registration for expression resistant 3-d face recognition. IEEE Transactions on Information Forensics and Security, 5(3), 425–440.

    Article  Google Scholar 

  • Alyüz, N., Gokberk, B., & Akarun, L. (2013). 3-d face recognition under occlusion using masked projection. IEEE Transactions on Information Forensics and Security, 8(5), 789–802.

  • Berretti, S., Werghi, N., del Bimbo, A., & Pala, P. (2013). Matching 3d face scans using interest points and local histogram descriptors. Computers and Graphics, 37(5), 509–525.

    Article  Google Scholar 

  • Bowyer, K. W., Chang, K., & Flynn, P. (2006). A survey of approaches and challenges in 3d and multi-modal 3d+2d face recognition. Computer Vision and Image Understanding, 101, 1–15.

    Article  Google Scholar 

  • Colombo, A., Cusano, C., & Schettini, R. (2011). Three-dimensional occlusion detection and restoration of partially occluded faces. Journal of Mathematical Imaging and Vision, 40(1), 105–119.

    Article  MATH  MathSciNet  Google Scholar 

  • Drira, H., Ben Amor, B., Srivastava, A., Daoudi, M., & Slama, R. (2013). 3d face recognition under expressions, occlusions, and pose variations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(9), 2270–2283.

  • Faltemier, T. C., Bowyer, K. W., & Flynn, P. J. (2008). A region ensemble for 3d face recognition. IEEE Transactions on Information Forensics and Security, 3(1), 62–73.

    Article  Google Scholar 

  • Goldfeather, J., & Interrante, V. (2004). A novel cubic-order algorithm for approximating principal direction vectors. ACM Transactions Graphics, 23(1), 45–63.

    Article  Google Scholar 

  • Guo, Z., Zhang, Y., Xia, Y., Lin, Z., Fan, Y., & Feng, D. (2013). Multi-pose 3d face recognition based on 2d sparse representation. Journal of Visual Communication and Image Representation, 24(2), 1047–3203.

    Article  Google Scholar 

  • Huang, D., Ardabilian, M., Wang, Y., & Chen, L. (2012). 3-d face recognition using elbp-based facial description and local feature hybrid matching. IEEE Transactions on Information Forensics and Security, 7(5), 1551–1565.

    Article  Google Scholar 

  • Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cats visual cortex. The Journal of Physiology, 160, 106–154.

    Article  Google Scholar 

  • Johnson, A., & Hebert, M. (1999). Using spin images for efficient object recognition in cluttered 3d scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(5), 433–449.

    Article  Google Scholar 

  • Kakadiaris, I. A., Passalis, G., Toderici, G., Murtuza, M. N., Lu, Y., Karampatziakis, N., et al. (2007). Three-dimensional face recognition in the presence of facial expressions: An annotated deformable model approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(4), 640–649.

    Article  Google Scholar 

  • Li, H., Huang, D., Lemaire, P., Morvan, J., & Chen, L. (2011). Expression robust 3d face recognition via mesh-based histograms of multiple order surface differential quantities. In: Proceedings of IEEE International Conference on Image Processing (ICIP) (pp. 4019–4023).

  • Li, H., Huang, D., Morvan, J. M., Chen, L., & Wang, Y. (2014). Expression-robust 3d face recognition via weighted sparse representation of multi-scale and multi-component local normal patterns. Neurocomputing, 133, 179–193.

    Article  Google Scholar 

  • Li, S. Z., & Jain, A. K. (2005). Handbook of Face Recognition. Secaucus, NJ: Springer-Verlag New York Inc.

    MATH  Google Scholar 

  • Li, X., Jia, T., & Zhang, H. (2009). Expression-insensitive 3d face recognition using sparse representation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2575–2582).

  • Liao, S., Jain, A., & Li, S. (2013). Partial face recognition: Alignment-free approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(5), 1193–1205.

    Article  Google Scholar 

  • Lo, T., & Siebert, J. (2009). Local feature extraction and matching on range images: 2.5d sift. Computer Vision and Image Understanding, 113(12), 1235–1250.

    Article  Google Scholar 

  • Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.

    Article  Google Scholar 

  • Maes, C., Fabry, T., Keustermans, J., Smeets, D., Suetens, P., & Vandermeulen, D. (2010). Feature detection on 3d face surfaces for pose normalisation and recognition. In: Fourth IEEE International Conference on Biometrics: Theory Applications and Systems (BTAS), pp 1–6.

  • Masi, I., Lisanti, G., Bagdanov, A., Pala, P., & Del Bimbo, A. (2013). Using 3d models to recognize 2d faces in the wild. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 775–780).

  • Meyer, T., Eriksson, M., & Maggio, R. (2001). Gradient estimation from irregularly spaced data sets. Mathematical Geology, 33, 693–717.

    Article  MATH  Google Scholar 

  • Mian, A. S., Bennamoun, M., & Owens, R. A. (2007). An efficient multimodal 2d–3d hybrid approach to automatic face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(11), 1927–1943.

    Article  Google Scholar 

  • Mian, A. S., Bennamoun, M., & Owens, R. A. (2008). Keypoint detection and local feature matching for textured 3d face recognition. International Journal of Computer Vision, 79(1), 1–12.

    Article  Google Scholar 

  • Mohammadzade, H., & Hatzinakos, D. (2013). Iterative closest normal point for 3d face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(2), 381–397.

    Article  Google Scholar 

  • Ocegueda, O., Passalis, G., Theoharis, T., Shah, S., & Kakadiaris, I. (2011). Ur3d-c: Linear dimensionality reduction for efficient 3d face recognition. In: Proceedings of IEEE International Joint Conference on Biometrics (IJCB).

  • Passalis, G., Perakis, P., Theoharis, T., & Kakadiaris, I. (2011). Using facial symmetry to handle pose variations in real-world 3d face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(10), 1938–1951.

    Article  Google Scholar 

  • Pati, Y.C., Rezaiifar, R., & Krishnaprasad, P.S. (1993). Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In: Proceedings of 27th Asilomar Conference on Signals, Systems and Computers (ACSSC).

  • Phillips, P., Flynn, P. J., Scruggs, T., Bowyer, K., Chang, J., Hoffman, K., Marques, J., Min, J., & Worek, W. (2005). Overview of the face recognition grand challenge. In: Proeedings of of IEEE conference of Computer Vision and Pattern Recognition (CVPR).

  • Queirolo, C., Silva, L., Bellon, O., & Segundo, M. (2010). 3d face recognition using simulated annealing and the surface interpenetration measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(2), 206–219.

    Article  Google Scholar 

  • Savran, A., Alyüz, N., Dibeklioglu, H., Çeliktutan, O., Gökberk, B., Sankur, B., & Akarun, L. (2008). 3d face recognition benchmarks on the bosphorus database with focus on facial expressions. In: Proceedings of Workshop on Biometrics and Identity Management (WBIM).

  • Skellya, L. J., & Sclaroffb, S. (2007). Improved feature descriptors for 3-d surface matching. Proceedings of of SPIE Two- and Three-Dimensional Methods for Inspection and Metrology, 6762, 1–12.

    Google Scholar 

  • Smeets, D., Claes, P., Hermans, J., Vandermeulen, D., & Suetens, P. (2012). A comparative study of 3-d face recognition under expression variations. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, 42(5), 710–727.

  • Smeets, D., Keustermans, J., Vandermeulen, D., & Suetens, P. (2013). meshsift: Local surface features for 3d face recognition under expression variations and partial data. Computer Vision and Image Understanding, 117(2), 158–169.

    Google Scholar 

  • Spreeuwers, L. (2011). Fast and accurate 3d face recognition using registration to an intrinsic coordinate system and fusion of multiple region classifiers. International Journal of Computer Vision, 93(3), 389–414.

    Article  MATH  Google Scholar 

  • Szeptycki, P., Ardabilian, M., & Chen, L. (2009). A coarse-to-fine curvature analysis-based rotation invariant 3d face landmarking. In: International Conference on Biometrics: Theory, Applications and Systems (ICB) (pp. 3206–3211).

  • Tola, E., Lepetit, V., & Fua, P. (2010). Daisy: An efficient dense descriptor applied to wide-baseline stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(5), 815–830.

    Article  Google Scholar 

  • Tombari, F., Salti, S., & Di Stefano, L. (2013). Performance evaluation of 3d keypoint detectors. International Journal of Computer Vision, 102(1–3), 198–220.

    Article  Google Scholar 

  • Veltkamp, R.C., van Jole, S., Drira, H., Amor, B.B., Daoudi, M., Li, H., Chen, L., Claes, P., Smeets, D., Hermans, J., Vandermeulen, D., & Suetens, P. (2011). Shrec ’11 track: 3d face models retrieval. In: Euro-graphics Workshop on 3D Object Retrieval (3DOR) (pp. 89–95).

  • Wang, Y., Liu, J., & Tang, X. (2010). Robust 3d face recognition by local shape difference boosting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(10), 1858–1870.

    Article  Google Scholar 

  • Wright, J., Yang, A. Y., Ganesh, A., Sastry, S. S., & Ma, Y. (2009). Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(2), 210–227.

    Article  Google Scholar 

  • Yang, A.Y., Wright, J., Ma, Y., & Sastry, S.S. (2007). Feature selection in face recognition: A sparse representation perspective. Tech. Rep. UCB/EECS-2007-99, EECS Department, University of California, Berkeley. http://www.eecs.berkeley.edu/Pubs/TechRpts/2007/EECS-2007-99.html.

  • Zaharescu, A., Boyer, E., Varanasi, K., & Horaud, R. (2009). Surface feature detection and description with applications to mesh matching. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 373–380).

  • Zaharescu, A., Boyer, E., & Horaud, R. P. (2012). Keypoints and local descriptors of scalar functions on 2d manifolds. International Journal of Computer Vision, 100(1), 78–98.

    Article  MATH  Google Scholar 

  • Zhao, W., Chellappa, R., Phillips, P. J., & Rosenfeld, A. (2003). Face recognition: A literature survey. ACM Computing Surveys, 35, 399–458.

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 11401464, 61202237 and 61273263; the China Postdoctoral Science Foundation (No. 2014M560785); the Specialized Research Fund for the Doctoral Program of Higher Education (No. 20121102120016); the French research agency, Agence Nationale de la Recherche (ANR) under Grant ANR-07-SESU-004, ANR-2010-INTB-0301-01 and ANR-13-INSE-0004-02; the joint project by the LIA 2MCSI lab between the group of Ecoles Centrales and Beihang University; and the Fundamental Research Funds for the Central Universities. We would like to thank the Bosphorus (Savran et al. 2008) and the FRGC (Phillips et al. 2005) organizers for the face data, Peyré for the Toolbox Fast Marching.

Author information

Authors and Affiliations

  1. School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, 710049, People‘s Republic of China

    Huibin Li

  2. Beijing Center for Mathematics and Information Interdisciplinary Sciences (BCMIIS), Beijing, People‘s Republic of China

    Huibin Li

  3. Laboratory of Intelligent Recognition and Image Processing, School of Computer Science and Engineering, Beihang University, Beijing, 10091, People‘s Republic of China

    Di Huang & Yunhong Wang

  4. Département de Mathématiques, Université Claude Bernard Lyon 1, Lyon, 69622, France

    Jean-Marie Morvan

  5. Geometric Modeling and Scientific Visualization Center, King Abdullah University of Science and Technology, Makkah, Saudi Arabia

    Jean-Marie Morvan

  6. Département de Mathématiques et Informatique, UMR CNRS 5205, Ecole Centrale Lyon, Lyon, 69134, France

    Liming Chen

Authors
  1. Huibin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Di Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jean-Marie Morvan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yunhong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Liming Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Di Huang.

Additional information

Communicated by C. Schnörr.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Huang, D., Morvan, JM. et al. Towards 3D Face Recognition in the Real: A Registration-Free Approach Using Fine-Grained Matching of 3D Keypoint Descriptors. Int J Comput Vis 113, 128–142 (2015). https://doi.org/10.1007/s11263-014-0785-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-014-0785-6

Keywords