Skip to main content
SpringerLink
Log in

Towards Fine-Grained Optimal 3D Face Dense Registration: An Iterative Dividing and Diffusing Method

  • S.I. : Traditional Computer Vision in the Age of Deep Learning
  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Dense vertex-to-vertex correspondence (i.e. registration) between 3D faces is a fundamental and challenging issue for 3D &2D face analysis. While the sparse landmarks are definite with anatomically ground-truth correspondence, the dense vertex correspondences on most facial regions are unknown. In this view, the current methods commonly result in reasonable but diverse solutions, which deviate from the optimum to the dense registration problem. In this paper, we revisit dense registration by a dimension-degraded problem, i.e. proportional segmentation of a line, and employ an iterative dividing and diffusing method to reach an optimum solution that is robust to different initializations. We formulate a local registration problem for dividing and a linear least-square problem for diffusing, with constraints on fixed features on a 3D facial surface. We further propose a multi-resolution algorithm to accelerate the computational process. The proposed method is linked to a novel local scaling metric, where we illustrate the physical significance as smooth adaptions for local cells of 3D facial shapes. Extensive experiments on public datasets demonstrate the effectiveness of the proposed method in various aspects. Generally, the proposed method leads to not only significantly better representations of 3D facial data, but also coherent local deformations with elegant grid architecture for fine-grained registrations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. https://pardiso-project.org/.

  2. https://libigl.github.io/.

  3. The core code for the iterative dividing and diffusing method will be publicly available at https://github.com/NaughtyZZ/3D_face_dense_registration.

References

  • Allen, B., Curless, B., & Popović, Z. (2003). The space of human body shapes: Reconstruction and parameterization from range scans. ACM Transactions on Graphics, 22(3), 587–594.

    Article  Google Scholar 

  • Amberg, B., Romdhani, S., & Vetter, T. (2007). Optimal step nonrigid ICP algorithms for surface registration. In IEEE conference on computer vision and pattern recognition (pp. 1–8).

  • Bahri, M., O’Sullivan, E., Gong, S., Liu, F., Liu, X., Bronstein, M. M., & Zafeiriou, S. (2021). Shape my face: Registering 3D face scans by surface-to-surface translation. International Journal of Computer Vision, 129(9), 2680–2713.

    Article  Google Scholar 

  • Bentley, J. L. (1975). Multidimensional binary search trees used for associative searching. Communications of the ACM, 18(9), 509–517.

    Article  MATH  Google Scholar 

  • Bergen, Gvd. (1997). Efficient collision detection of complex deformable models using AABB trees. Journal of Graphics Tools, 2(4), 1–13.

    Article  MATH  Google Scholar 

  • Besl, P., & McKay, H. (1992). A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2), 239–256.

    Article  Google Scholar 

  • Blanz, V., & Vetter, T. (1999). A morphable model for the synthesis of 3D faces. In ACM annual conference on computer graphics and interactive techniques (pp. 187–194).

  • Bolkart, T., & Wuhrer, S. (2015). A groupwise multilinear correspondence optimization for 3D faces. In IEEE international conference on computer vision (pp. 3604–3612).

  • Booth, J., Roussos, A., Ponniah, A., Dunaway, D., & Zafeiriou, S. (2018). Large scale 3D morphable models. International Journal of Computer Vision, 126(2), 233–254.

    Article  MathSciNet  Google Scholar 

  • Bronstein, A. M., Bronstein, M. M., & Kimmel, R. (2005). Three-dimensional face recognition. International Journal of Computer Vision, 64(1), 5–30.

    Article  Google Scholar 

  • Brown, B. J., & Rusinkiewicz, S. M. (2007). Global non-rigid alignment of 3-D scans. ACM Transactions on Graphics, 26(3), 1276404.

    Article  Google Scholar 

  • Bulat, A., & Tzimiropoulos, G. (2017). How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In IEEE international conference on computer vision (pp. 1021–1030).

  • Cao, C., Weng, Y., Zhou, S., Tong, Y., & Zhou, K. (2013). FaceWarehouse: A 3D facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics, 20(3), 413–425.

    Google Scholar 

  • Cheng, S., Marras, I., Zafeiriou, S., & Pantic, M. (2015). Active nonrigid ICP algorithm. In IEEE international conference and workshops on automatic face and gesture recognition (Vol. 1, pp. 1–8).

  • Chen, Y., & Medioni, G. (1992). Object modelling by registration of multiple range images. Image and Vision Computing, 10(3), 145–155.

    Article  Google Scholar 

  • Corneanu, C. A., Simón, M. O., Cohn, J. F., & Guerrero, S. E. (2016). Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(8), 1548–1568.

    Article  Google Scholar 

  • Crane, K., Weischedel, C., & Wardetzky, M. (2013). Geodesics in heat: A new approach to computing distance based on heat flow. ACM Transactions on Graphics, 32(5), 1–11.

    Article  Google Scholar 

  • Creusot, C., Pears, N., & Austin, J. (2013). A machine-learning approach to keypoint detection and landmarking on 3D meshes. International Journal of Computer Vision, 102(1–3), 146–179.

    Article  Google Scholar 

  • Davies, R. H., Twining, C. J., Cootes, T. F., Waterton, J. C., & Taylor, C. J. (2002). A minimum description length approach to statistical shape modeling. IEEE Transactions on Medical Imaging, 21(5), 525–537.

    Article  MATH  Google Scholar 

  • Drira, H., Amor, B. B., Srivastava, A., Daoudi, M., & Slama, R. (2013). 3d face recognition under expressions, occlusions, and pose variations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(9), 2270–2283.

    Article  Google Scholar 

  • Egger, B., Smith, W. A., Tewari, A., Wuhrer, S., Zollhoefer, M., Beeler, T., Bernard, F., Bolkart, T., Kortylewski, A., Romdhani, S., et al. (2020). 3D morphable face models-past, present, and future. ACM Transactions on Graphics, 39(5), 1–38.

    Article  Google Scholar 

  • Eldar, Y., Lindenbaum, M., Porat, M., & Zeevi, Y. Y. (1997). The farthest point strategy for progressive image sampling. IEEE Transactions on Image Processing, 6(9), 1305–1315.

    Article  Google Scholar 

  • Fan, Z., Hu, X., Chen, C., & Peng, S. (2018). Dense semantic and topological correspondence of 3D faces without landmarks. In European conference on computer vision (pp. 523–539).

  • Fan, Z., Hu, X., Chen, C., & Peng, S. (2019). Boosting local shape matching for dense 3D face correspondence. In IEEE conference on computer vision and pattern recognition (pp. 10944–10954).

  • Feldmar, J., & Ayache, N. (1996). Rigid, affine and locally affine registration of free-form surfaces. International Journal of Computer Vision, 18(2), 99–119.

    Article  Google Scholar 

  • Ferrari, C., Berretti, S., Pala, P., & Del Bimbo, A. (2021). A sparse and locally coherent morphable face model for dense semantic correspondence across heterogeneous 3D faces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44, 6667–6682.

    Article  Google Scholar 

  • Ferrari, C., Lisanti, G., Berretti, S., & Del Bimbo, A. (2017). A dictionary learning-based 3D morphable shape model. IEEE Transactions on Multimedia, 19(12), 2666–2679.

    Article  Google Scholar 

  • Fujiwara, K., Nishino, K., Takamatsu, J., Zheng, B., & Ikeuchi, K. (2011). Locally rigid globally non-rigid surface registration. In International conference on computer vision (pp. 1527–1534).

  • Garrido, P., Valgaerts, L., Rehmsen, O., Thormahlen, T., Perez, P., & Theobalt, C. (2014). Automatic face reenactment. In IEEE conference on computer vision and pattern recognition (pp. 4217–4224).

  • Gerig, T., Morel-Forster, A., Blumer, C., Egger, B., Luthi, M., Schönborn, S., & Vetter, T. (2018). Morphable face models-an open framework. In IEEE international conference on automatic face and gesture recognition (pp. 75–82).

  • Gilani, S. Z., Mian, A., & Eastwood, P. (2017). Deep, dense and accurate 3D face correspondence for generating population specific deformable models. Pattern Recognition, 69, 238–250.

    Article  Google Scholar 

  • Gilani, S. Z., Mian, A., Shafait, F., & Reid, I. (2017). Dense 3D face correspondence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(7), 1584–1598.

    Article  Google Scholar 

  • Grewe, C. M., & Zachow, S. (2016). Fully automated and highly accurate dense correspondence for facial surfaces. In European conference on computer vision (pp. 552–568).

  • Gu, X., Wang, S., Kim, J., Zeng, Y., Wang, Y., Qin, H., & Samaras, D. (2007). Ricci flow for 3D shape analysis. In IEEE international conference on computer vision (pp. 1–8).

  • Li, H., Sumner, R. W., & Pauly, M. (2008). Global correspondence optimization for non-rigid registration of depth scans. In Computer graphics forum (Vol. 27, pp. 1421–1430).

  • Li, T., Bolkart, T., Black, M. J., Li, H., & Romero, J. (2017). Learning a model of facial shape and expression from 4D scans. ACM Transaction on Graphics, 36(6), 1–194.

    Google Scholar 

  • Liu, F., Tran, L., & Liu, X. (2019). 3D face modeling from diverse raw scan data. In IEEE international conference on computer vision (pp. 9408–9418).

  • Liu, F., Zeng, D., Zhao, Q., & Liu, X. (2016). Joint face alignment and 3D face reconstruction. In European conference on computer vision (pp. 545–560).

  • Maiseli, B., Gu, Y., & Gao, H. (2017). Recent developments and trends in point set registration methods. Journal of Visual Communication and Image Representation, 46, 95–106.

    Article  Google Scholar 

  • Ma, J., Zhao, J., Tian, J., Yuille, A. L., & Tu, Z. (2014). Robust point matching via vector field consensus. IEEE Transactions on Image Processing, 23(4), 1706–1721.

    Article  MathSciNet  MATH  Google Scholar 

  • Mohammadzade, H., & Hatzinakos, D. (2012). Iterative closest normal point for 3D face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(2), 381–397.

    Article  Google Scholar 

  • Myronenko, A., & Song, X. (2010). Point set registration: Coherent point drift. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(12), 2262–2275.

    Article  Google Scholar 

  • Pan, G., Zhang, X., Wang, Y., Hu, Z., Zheng, X., & Wu, Z. (2013). Establishing point correspondence of 3D faces via sparse facial deformable model. IEEE Transactions on Image Processing, 22(11), 4170–4181.

    Article  MathSciNet  MATH  Google Scholar 

  • Patel, A., & Smith, W. A. (2009). 3D morphable face models revisited. In IEEE conference on computer vision and pattern recognition (pp. 1327–1334).

  • Phillips, P. J., Flynn, P. J., Scruggs, T., Bowyer, K. W., Chang, J., Hoffman, K., Marques, J., Min, J., & Worek, W. (2005). Overview of the face recognition grand challenge. In IEEE conference on computer vision and pattern recognition (Vol. 1, pp. 947–954).

  • Ploumpis, S., Ververas, E., O’Sullivan, E., Moschoglou, S., Wang, H., Pears, N., Smith, W., Gecer, B., & Zafeiriou, S. P. (2020). Towards a complete 3D morphable model of the human head. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43, 4142–4160.

    Article  Google Scholar 

  • Rissanen, J. (1978). Modeling by shortest data description. Automatica, 14(5), 465–471.

    Article  MATH  Google Scholar 

  • Salazar, A., Wuhrer, S., Shu, C., & Prieto, F. (2014). Fully automatic expression-invariant face correspondence. Machine Vision and Applications, 25(4), 859–879.

    Article  Google Scholar 

  • Savran, A., Alyüz, N., Dibeklioğlu, H., Çeliktutan, O., Gökberk, B., Sankur, B., & Akarun, L. (2008). Bosphorus database for 3D face analysis. In European workshop on biometrics and identity management (pp. 47–56).

  • Segundo, M. P. P., Silva, L., Bellon, O. R. P., & Queirolo, C. C. (2010). Automatic face segmentation and facial landmark detection in range images. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 40(5), 1319–1330.

    Article  Google Scholar 

  • Sorkine, O., & Alexa, M. (2007) As-rigid-as-possible surface modeling. In Symposium on geometry processing (Vol. 4, pp. 109–116).

  • Sorkine-Hornung, O., & Rabinovich, M. (2017). Least-squares rigid motion using SVD. Computing, 1(1), 1–5.

    Google Scholar 

  • Sun, Y., & Abidi, M. A. (2001). Surface matching by 3D point’s fingerprint. In IEEE international conference on computer vision (Vol. 2, pp. 263–269).

  • Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017). Synthesizing obama: learning lip sync from audio. ACM Transactions on Graphics, 36(4), 1–13.

    Article  Google Scholar 

  • Tam, G. K., Cheng, Z.-Q., Lai, Y.-K., Langbein, F. C., Liu, Y., Marshall, D., Martin, R. R., Sun, X.-F., & Rosin, P. L. (2012). Registration of 3D point clouds and meshes: A survey from rigid to nonrigid. IEEE Transactions on Visualization and Computer Graphics, 19(7), 1199–1217.

    Article  Google Scholar 

  • Terzopoulos, D., Platt, J., Barr, A., & Fleischer, K. (1987). Elastically deformable models. In ACM annual conference on computer graphics and interactive techniques (pp. 205–214).

  • Vlasic, D., Brand, M., Pfister, H., & Popovic, J. (2005). Face transfer with multilinear models. ACM Transactions on Graphics, 24(3), 426–433.

    Article  Google Scholar 

  • Wang, Y., Liu, J., & Tang, X. (2010). Robust 3D face recognition by local shape difference boosting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(10), 1858–1870.

    Article  Google Scholar 

  • Yang, H., Zhu, H., Wang, Y., Huang, M., Shen, Q., Yang, R., & Cao, X. (2020). Facescape: A large-scale high quality 3D face dataset and detailed riggable 3D face prediction. In IEEE conference on computer vision and pattern recognition (pp. 601–610).

  • Yang, J., Li, H., Campbell, D., & Jia, Y. (2015). Go-ICP: A globally optimal solution to 3D ICP point-set registration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(11), 2241–2254.

  • Yin, L., Wei, X., Sun, Y., Wang, J., & Rosato, M. J. (2006). A 3D facial expression database for facial behavior research. In International conference on automatic face and gesture recognition (pp. 211–216).

  • Zeng, W., Samaras, D., & Gu, D. (2010). Ricci flow for 3D shape analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(4), 662–677.

    Article  Google Scholar 

  • Zhang, C., Smith, W. A., Dessein, A., Pears, N., & Dai, H. (2016). Functional faces: Groupwise dense correspondence using functional maps. In IEEE conference on computer vision and pattern recognition (pp. 5033–5041).

  • Zhang, Z. (1994). Iterative point matching for registration of free-form curves and surfaces. International Journal of Computer Vision, 13(2), 119–152.

    Article  Google Scholar 

  • Zhu, X., Liu, X., Lei, Z., & Li, S. Z. (2017). Face alignment in full pose range: A 3D total solution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(1), 78–92.

    Article  Google Scholar 

  • Zollhöfer, M., Thies, J., Garrido, P., Bradley, D., Beeler, T., Pérez, P., Stamminger, M., Nießner, M., & Theobalt, C. (2018). State of the art on monocular 3D face reconstruction, tracking, and applications. In Computer graphics forum (Vol. 37, pp. 523–550).

  • Zulqarnain Gilani, S., Shafait, F., & Mian, A. (2015). Shape-based automatic detection of a large number of 3D facial landmarks. In IEEE conference on computer vision and pattern recognition (pp. 4639–4648).

Download references

Acknowledgements

We would like to thank the anonymous reviewers for their insightful comments. This work is supported in part by the National Key Research and Development Program of China (No. 2022YFF0902302), the National Science Foundation of China (No. 62106250), and China Postdoctoral Science Foundation (No. 2021M703272).

Author information

Authors and Affiliations

  1. Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China

    Zhenfeng Fan & Shihong Xia

  2. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Silong Peng

  3. University of Chinese Academy of Sciences, Beijing, China

    Zhenfeng Fan, Silong Peng & Shihong Xia

Authors
  1. Zhenfeng Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Silong Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shihong Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shihong Xia.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, Z., Peng, S. & Xia, S. Towards Fine-Grained Optimal 3D Face Dense Registration: An Iterative Dividing and Diffusing Method. Int J Comput Vis 131, 2356–2376 (2023). https://doi.org/10.1007/s11263-023-01825-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-023-01825-7

Keywords

Search

Navigation