Abstract
Since a human face could be represented by a few landmarks with less redundant information, and calculated by a linear combination of a small number of prototypical faces, we propose a two-step 3D face reconstruction approach including landmark depth estimation and shape deformation. The proposed approach allows us to reconstruct a realistic 3D face from a 2D frontal face image. First, we apply a coupled dictionary learning method based on sparse representation to explore the underlying mappings between pair of 2D and 3D training landmarks. In the method, a weighted l 1 norm sparsity function is introduced to better pursuit the l 0 norm sparsity. Then, the depth of the landmarks could be estimated. Second, we propose a novel shape deformation method to reconstruct the 3D face by combining a small number of most relevant deformed faces which are obtained by the estimated landmarks. The sparsity regulation is also introduced to find the relevant faces in the second step. The proposed approach could explore the distributions of 2D and 3D faces and the underlying mappings between them well, because human faces are represented by low-dimensional landmarks, and their distributions are described by sparse representations. Moreover, it is much more flexible since we can make any change in any step. Extensive experiments are conducted on BJUT_3D database, and the results validate the effectiveness of the proposed approach.
Similar content being viewed by others
References
Aharon M, Elad M, Bruckstein A (2005) K-SVD: design of dictionaries for sparse representation. Proc SPARS 5:9–12
Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th annual conference on computer graphics and interactive techniques, pp 187–194
Blake A, Isard M (1998) Active shape models. Springer
Cao C, Weng Y, Zhou S, Tong Y, Zhou K (2014) Facewarehouse: A 3d facial expression database for visual computing. IEEE Trans Vis Comput Graph 20(3):413–425
Cao X, Wei Y, Wen F, Sun J (2014) Face alignment by explicit shape regression. Int J Comput Vis 107(2):177–190
Castelán M, Van Horebeek J (2008) 3D face shape approximation from intensities using partial least squares. In: IEEE computer society conference on proceedings of computer vision and pattern recognition workshops, 2008. CVPRW’08, pp 1–8
Catmull E, Clark J (1978) Recursively generated b-spline surfaces on arbitrary topological meshes. Comput Aided Des 10(6):350–355
Dou P, Wu Y, Shah S, Kakadiaris I (2014) Robust 3d face shape reconstruction from single images via two-fold coupled structure learning and off-the-shelf landmark detectors. In: Proceedings of the British machine vision conference. BMVA Press
Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32(2):407–499
Horn BK (1989) Obtaining shape from shading information. MIT press
Hu Y, Jiang D, Yan S, Zhang L, Zhang H (2004) Automatic 3D reconstruction for face recognition. In: Proceedings of automatic face and gesture recognition, 2004. Sixth IEEE International Conference on Proceedings, pp 843–848
Jenatton JMFBJPGSR, Obozinski G (2008) Sparse modeling software. http://spams-devel.gforge.inria.fr/
Jones MJ, Poggio T (1998) Multidimensional morphable models: a framework for representing and matching object classes 29(2):107–131
Lei Z, Bai Q, He R, Li S (2008) Face shape recovery from a single image using cca mapping between tensor spaces. In: IEEE conference on computer vision and pattern recognition, 2008. CVPR 2008, pp 1–7
Pighin F, Hecker J, Lischinski D, Szeliski R, Salesin DH (2006) Synthesizing realistic facial expressions from photographs. In: Proceedings of ACM SIGGRAPH 2006 courses, p 19
Platt SM, Badler NI (1981) Animating facial expressions. SIGGRAPH Comput Graph 15(3):245–252
Prados E, Faugeras O (2005) Shape from shading: a well-posed problem?. In: IEEE Computer Society Conference on proceedings of computer vision and pattern recognition, 2005. CVPR 2005, pp 870– 877
Reiter M, Dormer R, Langs G, Bischof H (2006) 3D and infrared face reconstruction from rgb data using canonical correlation analysis. In: 18th International conference on proceedings of pattern recognition, 2006. ICPR 2006, pp 425–428
Ren S, Cao X, Wei Y, Sun J (2014) Face alignment at 3000 fps via regressing local binary features. In: 2014 IEEE conference on computer vision and pattern recognition (CVPR), pp 1685–1692
Sanchez-Escobedo D, Castelan M (2012) Face synthesis from a frontal pose image using partial least squares and b-splines. In: 2012 19th IEEE international conference on image processing (ICIP), pp 1801–1804
Sanchez-Escobedo D, Castelan M (2013) 3d face shape prediction from a frontal image using cylindrical coordinates and partial least squares. Pattern Recogn Lett 34(4):389–399. advances in Pattern Recognition Methodology and Applications
Sederberg TW, PSR (1986) Free-form deformation of solid geometric models. In: Proceedings of the 13th annual conference on computer graphics and interactive techniques. ser. SIGGRAPH ’86. ACM, pp 151–160
Song M, Tao D, Huang X, Chen C, Bu J (2012) Three-dimensional face reconstruction from a single image by a coupled RBF network 21(5):2887–2897
Tech M (2005) The BJUT-3D large-scale chinese face database. Graphics Lab, Technical Report, Beijing University of Technology, Tech. Rep.
Terzopoulos D, Waters K (1990) Physically-based facial modelling, analysis, and animation. J Vis Comput Animat 1(2):73–80
Vetter T, Poggio T (1997) Linear object classes and image synthesis from a single example image 19(7):733–742
Wang S, Zhang L, Liang Y, Pan Q (2012) Synthesizing semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In: 2012 IEEE conference on Proceedings computer vision and pattern recognition (CVPR), pp 2216–2223
Xiao Q, Han L, Liu P (2014) 3d face reconstruction via feature point depth estimation and shape deformation. In: Proceedings of the 2014 22Nd international conference on pattern recognition, ser. ICPR ’14. IEEE Computer Society, Washington, pp 2257–2262. [Online]. Available. doi:10.1109/ICPR.2014.392
Xiaowei Zhou XH, Leonardos S, Daniilidis K (2014) 3d shape reconstruction from 2d landmarks: a convex formulation. CoRR. arXiv:1411.2942
Xu L, Zheng S, Jia J (2013) Unnatural L0 sparse representation for natural image deblurring. In: 2013 IEEE conference on proceedings of computer vision and pattern recognition (CVPR), pp 1107– 1114
Acknowledgments
This work was supported by Cloud Computing Platform for Internet of Things-Fujian Scientific Research Platform for Innovation by the Foundation of Quanzhou City under No. 2014Z103 and No.2014Z113. This work was supported in part by the Fundamental Research Funds for the Central Universities JB-ZR1202, and by new IT platform construction project in Fujian Province 2013H2002.The authors would like to thank the reviewers for their valuable suggestions and comments
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Liu, P., Hong, M., Wang, M. et al. 3D face reconstruction via landmark depth estimation and shape deformation. Multimed Tools Appl 76, 2749–2767 (2017). https://doi.org/10.1007/s11042-016-3259-8
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-016-3259-8