Skip to main content
Log in

3D face reconstruction via landmark depth estimation and shape deformation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Since a human face could be represented by a few landmarks with less redundant information, and calculated by a linear combination of a small number of prototypical faces, we propose a two-step 3D face reconstruction approach including landmark depth estimation and shape deformation. The proposed approach allows us to reconstruct a realistic 3D face from a 2D frontal face image. First, we apply a coupled dictionary learning method based on sparse representation to explore the underlying mappings between pair of 2D and 3D training landmarks. In the method, a weighted l 1 norm sparsity function is introduced to better pursuit the l 0 norm sparsity. Then, the depth of the landmarks could be estimated. Second, we propose a novel shape deformation method to reconstruct the 3D face by combining a small number of most relevant deformed faces which are obtained by the estimated landmarks. The sparsity regulation is also introduced to find the relevant faces in the second step. The proposed approach could explore the distributions of 2D and 3D faces and the underlying mappings between them well, because human faces are represented by low-dimensional landmarks, and their distributions are described by sparse representations. Moreover, it is much more flexible since we can make any change in any step. Extensive experiments are conducted on BJUT_3D database, and the results validate the effectiveness of the proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Aharon M, Elad M, Bruckstein A (2005) K-SVD: design of dictionaries for sparse representation. Proc SPARS 5:9–12

    Google Scholar 

  2. Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th annual conference on computer graphics and interactive techniques, pp 187–194

  3. Blake A, Isard M (1998) Active shape models. Springer

  4. Cao C, Weng Y, Zhou S, Tong Y, Zhou K (2014) Facewarehouse: A 3d facial expression database for visual computing. IEEE Trans Vis Comput Graph 20(3):413–425

    Article  Google Scholar 

  5. Cao X, Wei Y, Wen F, Sun J (2014) Face alignment by explicit shape regression. Int J Comput Vis 107(2):177–190

    Article  MathSciNet  Google Scholar 

  6. Castelán M, Van Horebeek J (2008) 3D face shape approximation from intensities using partial least squares. In: IEEE computer society conference on proceedings of computer vision and pattern recognition workshops, 2008. CVPRW’08, pp 1–8

  7. Catmull E, Clark J (1978) Recursively generated b-spline surfaces on arbitrary topological meshes. Comput Aided Des 10(6):350–355

    Article  Google Scholar 

  8. Dou P, Wu Y, Shah S, Kakadiaris I (2014) Robust 3d face shape reconstruction from single images via two-fold coupled structure learning and off-the-shelf landmark detectors. In: Proceedings of the British machine vision conference. BMVA Press

  9. Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32(2):407–499

    Article  MathSciNet  MATH  Google Scholar 

  10. Horn BK (1989) Obtaining shape from shading information. MIT press

  11. Hu Y, Jiang D, Yan S, Zhang L, Zhang H (2004) Automatic 3D reconstruction for face recognition. In: Proceedings of automatic face and gesture recognition, 2004. Sixth IEEE International Conference on Proceedings, pp 843–848

  12. Jenatton JMFBJPGSR, Obozinski G (2008) Sparse modeling software. http://spams-devel.gforge.inria.fr/

  13. Jones MJ, Poggio T (1998) Multidimensional morphable models: a framework for representing and matching object classes 29(2):107–131

  14. Lei Z, Bai Q, He R, Li S (2008) Face shape recovery from a single image using cca mapping between tensor spaces. In: IEEE conference on computer vision and pattern recognition, 2008. CVPR 2008, pp 1–7

  15. Pighin F, Hecker J, Lischinski D, Szeliski R, Salesin DH (2006) Synthesizing realistic facial expressions from photographs. In: Proceedings of ACM SIGGRAPH 2006 courses, p 19

  16. Platt SM, Badler NI (1981) Animating facial expressions. SIGGRAPH Comput Graph 15(3):245–252

    Article  Google Scholar 

  17. Prados E, Faugeras O (2005) Shape from shading: a well-posed problem?. In: IEEE Computer Society Conference on proceedings of computer vision and pattern recognition, 2005. CVPR 2005, pp 870– 877

  18. Reiter M, Dormer R, Langs G, Bischof H (2006) 3D and infrared face reconstruction from rgb data using canonical correlation analysis. In: 18th International conference on proceedings of pattern recognition, 2006. ICPR 2006, pp 425–428

  19. Ren S, Cao X, Wei Y, Sun J (2014) Face alignment at 3000 fps via regressing local binary features. In: 2014 IEEE conference on computer vision and pattern recognition (CVPR), pp 1685–1692

  20. Sanchez-Escobedo D, Castelan M (2012) Face synthesis from a frontal pose image using partial least squares and b-splines. In: 2012 19th IEEE international conference on image processing (ICIP), pp 1801–1804

  21. Sanchez-Escobedo D, Castelan M (2013) 3d face shape prediction from a frontal image using cylindrical coordinates and partial least squares. Pattern Recogn Lett 34(4):389–399. advances in Pattern Recognition Methodology and Applications

    Article  Google Scholar 

  22. Sederberg TW, PSR (1986) Free-form deformation of solid geometric models. In: Proceedings of the 13th annual conference on computer graphics and interactive techniques. ser. SIGGRAPH ’86. ACM, pp 151–160

  23. Song M, Tao D, Huang X, Chen C, Bu J (2012) Three-dimensional face reconstruction from a single image by a coupled RBF network 21(5):2887–2897

  24. Tech M (2005) The BJUT-3D large-scale chinese face database. Graphics Lab, Technical Report, Beijing University of Technology, Tech. Rep.

  25. Terzopoulos D, Waters K (1990) Physically-based facial modelling, analysis, and animation. J Vis Comput Animat 1(2):73–80

    Article  Google Scholar 

  26. Vetter T, Poggio T (1997) Linear object classes and image synthesis from a single example image 19(7):733–742

  27. Wang S, Zhang L, Liang Y, Pan Q (2012) Synthesizing semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In: 2012 IEEE conference on Proceedings computer vision and pattern recognition (CVPR), pp 2216–2223

  28. Xiao Q, Han L, Liu P (2014) 3d face reconstruction via feature point depth estimation and shape deformation. In: Proceedings of the 2014 22Nd international conference on pattern recognition, ser. ICPR ’14. IEEE Computer Society, Washington, pp 2257–2262. [Online]. Available. doi:10.1109/ICPR.2014.392

  29. Xiaowei Zhou XH, Leonardos S, Daniilidis K (2014) 3d shape reconstruction from 2d landmarks: a convex formulation. CoRR. arXiv:1411.2942

  30. Xu L, Zheng S, Jia J (2013) Unnatural L0 sparse representation for natural image deblurring. In: 2013 IEEE conference on proceedings of computer vision and pattern recognition (CVPR), pp 1107– 1114

Download references

Acknowledgments

This work was supported by Cloud Computing Platform for Internet of Things-Fujian Scientific Research Platform for Innovation by the Foundation of Quanzhou City under No. 2014Z103 and No.2014Z113. This work was supported in part by the Fundamental Research Funds for the Central Universities JB-ZR1202, and by new IT platform construction project in Fujian Province 2013H2002.The authors would like to thank the reviewers for their valuable suggestions and comments

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Peizhong Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, P., Hong, M., Wang, M. et al. 3D face reconstruction via landmark depth estimation and shape deformation. Multimed Tools Appl 76, 2749–2767 (2017). https://doi.org/10.1007/s11042-016-3259-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3259-8

Keywords

Navigation