End-to-End 3D Facial Shape Reconstruction From an Unconstrained Image
Authors: 
Yanghong Han, Xing Qiao, Yan Wu, Zili ZhangAuthors Info & Claims
Yanghong Han, Xing Qiao, Yan Wu, Zili ZhangAuthors Info & ClaimsPages 325 - 332
Abstract
Three-dimensional face generation is of high importance for computer graphics and computer vision applications. The reconstruction task of human faces is challenging and face varies extensively when considering pose, expression, occlusion and illumination. Conventional face models are learned from a set of 3D face scans and represented by Principal Component Analysis(PCA), which is always constrained in linear space. Nevertheless, the shape of face and variations in images cannot be represented faithfully by these traditional models, which are very crucial for face synthesis. To address this problem, a novel architecture is proposed to learn face models in nonlinear spaces. Explicitly, our framework is a Convolutional Neural Networks(CNN) based End-to-End 3D Facial Shape Reconstruction(3DFSR-E2E) from a single-view 2D image scheme. And the variations in facial images are also coped with the model. Firstly, our Encoder is composed of new Improved Residual Blocks(IR-B) utilized in face recognition task, and Decoder consists of Fractionally-Strided Convolutions. Secondly, a UV map is estimated by the trainable Encoder-Decoder network, which is a representation of face geometry and face alignment information. Furthermore, a novel loss function is combined by a weighted Vertex-wise and a Laplacian regularization loss. The measure is employed to improve the results of reconstructed face model. Finally, the experimental results demonstrate the benefits and effectiveness of our overall method with qualitative and quantitative comparisons on public challenging face datasets.
References
[1]
Insaf Adjabi, Abdeldjalil Ouahabi, Amir Benzaoui, and Abdelmalik Taleb-Ahmed. 2020. Past, present, and future of face recognition: A review. Electronics 9, 8 (2020), 1188.
[2]
Ziqian Bai, Zhaopeng Cui, Jamal Ahmed Rahim, Xiaoming Liu, and Ping Tan. 2020. Deep facial non-rigid multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5850–5860.
[3]
Stephen W Bailey, Dalton Omens, Paul Dilorenzo, and James F O’Brien. 2020. Fast and deep facial deformations. ACM Transactions on Graphics (TOG) 39, 4 (2020), 94–1.
[4]
Anil Bas, Patrik Huber, William AP Smith, Muhammad Awais, and Josef Kittler. 2017. 3D morphable models as spatial transformer networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 904–912.
[5]
Volker Blanz and Thomas Vetter. 1999. A morphable model for the synthesis of 3D faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. 187–194.
[6]
James Booth, Anastasios Roussos, Allan Ponniah, David Dunaway, and Stefanos Zafeiriou. 2018. Large scale 3d morphable models. International Journal of Computer Vision 126, 2 (2018), 233–254.
[7]
Adrian Bulat and Georgios Tzimiropoulos. 2017. How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks). In Proceedings of the IEEE International Conference on Computer Vision. 1021–1030.
[8]
Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2013. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics 20, 3(2013), 413–425.
[9]
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4690–4699.
[10]
Mingtao Feng, Syed Zulqarnain Gilani, Yaonan Wang, and Ajmal Mian. 2018. 3D face reconstruction from light field images: A model-free approach. In Proceedings of the European Conference on Computer Vision (ECCV). 501–518.
[11]
Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, and Xi Zhou. 2018. Joint 3d face reconstruction and dense alignment with position map regression network. In Proceedings of the European Conference on Computer Vision (ECCV). 534–551.
[12]
Pablo Garrido, Michael Zollhöfer, Dan Casas, Levi Valgaerts, Kiran Varanasi, Patrick Pérez, and Christian Theobalt. 2016. Reconstruction of Personalized 3D Face Rigs from Monocular Video. ACM Transactions on Graphics 35, 3 (2016).
[13]
Syed Zulqarnain Gilani and Ajmal Mian. 2018. Learning from millions of 3D scans for large-scale 3D face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1896–1905.
[14]
Tal Hassner. 2013. Viewing real-world faces in 3D. In Proceedings of the IEEE International Conference on Computer Vision. 3607–3614.
[15]
Tal Hassner and Ronen Basri. 2013. Single view depth estimation from examples. arXiv preprint arXiv:1304.3915(2013).
[16]
Aaron S Jackson, Adrian Bulat, Vasileios Argyriou, and Georgios Tzimiropoulos. 2017. Large pose 3D face reconstruction from a single image via direct volumetric CNN regression. In Proceedings of the IEEE International Conference on Computer Vision. 1031–1039.
[17]
Luo Jiang, Juyong Zhang, Bailin Deng, Hao Li, and Ligang Liu. 2018. 3D face reconstruction with geometry details from a single image. IEEE Transactions on Image Processing 27, 10 (2018), 4756–4770.
[18]
Ira Kemelmacher-Shlizerman and Ronen Basri. 2010. 3D face reconstruction from a single image using a single reference face shape. IEEE transactions on pattern analysis and machine intelligence 33, 2(2010), 394–405.
[19]
Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR (Poster).
[20]
Yaojie Liu, Amin Jourabloo, William Ren, and Xiaoming Liu. 2017. Dense face alignment. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 1619–1628.
[21]
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. 2009. A 3D face model for pose and illumination invariant face recognition. In 2009 sixth IEEE international conference on advanced video and signal based surveillance. Ieee, 296–301.
[22]
Elad Richardson, Matan Sela, and Ron Kimmel. 2016. 3D face reconstruction by learning from synthetic data. In 2016 fourth international conference on 3D vision (3DV). IEEE, 460–469.
[23]
Elad Richardson, Matan Sela, Roy Or-El, and Ron Kimmel. 2017. Learning detailed face reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1259–1268.
[24]
Sami Romdhani and Thomas Vetter. 2005. Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 2. IEEE, 986–993.
[25]
Joseph Roth, Yiying Tong, and Xiaoming Liu. 2016. Adaptive 3D face reconstruction from unconstrained photo collections. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4197–4206.
[26]
Matan Sela, Elad Richardson, and Ron Kimmel. 2017. Unrestricted facial geometry reconstruction using image-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision. 1576–1585.
[27]
Soumyadip Sengupta, Angjoo Kanazawa, Carlos D Castillo, and David W Jacobs. 2018. Sfsnet: Learning shape, reflectance and illuminance of facesin the wild’. In Proceedings of the IEEE conference on computer vision and pattern recognition. 6296–6305.
[28]
Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2016. Face2face: Real-time face capture and reenactment of rgb videos. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2387–2395.
[29]
Luan Tran and Xiaoming Liu. 2019. On learning 3d face morphable model from in-the-wild images. IEEE transactions on pattern analysis and machine intelligence 43, 1(2019), 157–171.
[30]
Anh Tuan Tran, Tal Hassner, Iacopo Masi, and Gérard Medioni. 2017. Regressing robust and discriminative 3D morphable models with a very deep neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5163–5172.
[31]
Thomas Vetter and Volker Blanz. 1998. Estimating coloured 3D face models from single images: An example based approach. In European conference on computer vision. Springer, 499–513.
[32]
Fanzi Wu, Linchao Bao, Yajing Chen, Yonggen Ling, Yibing Song, Songnan Li, King Ngi Ngan, and Wei Liu. 2019. Mvf-net: Multi-view 3d face morphable model regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 959–968.
[33]
Dan Ye and Chiou-Shann Fuh. 2020. 3D Morphable Face Model for Face Animation. International Journal of Image and Graphics 20, 01 (2020), 2050003.
[34]
Xianfang Zeng, Yusu Pan, Mengmeng Wang, Jiangning Zhang, and Yong Liu. 2020. Realistic face reenactment via self-supervised disentangling of identity and pose. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 12757–12764.
[35]
Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z Li. 2016. Face alignment across large poses: A 3d solution. In Proceedings of the IEEE conference on computer vision and pattern recognition. 146–155.
Recommendations
Efficient 3D reconstruction for face recognition
Face recognition with variant pose, illumination and expression (PIE) is a challenging problem. In this paper, we propose an analysis-by-synthesis framework for face recognition with variant PIE. First, an efficient two-dimensional (2D)-to-three-...
The Elements of End-to-end Deep Face Recognition: A Survey of Recent Advances
Face recognition (FR) is one of the most popular and long-standing topics in computer vision. With the recent development of deep learning techniques and large-scale datasets, deep face recognition has made remarkable progress and has been widely used in ...
2D face fitting-assisted 3D face reconstruction for pose-robust face recognition
Special issue on Digital Information ForensicsRecent face recognition algorithm can achieve high accuracy when the tested face samples are frontal. However, when the face pose changes largely, the performance of existing methods drop drastically. Efforts on pose-robust face recognition are highly ...
Comments
Information & Contributors
Information
Published In
October 2021
366 pages
ISBN:9781450390675
DOI:10.1145/3494885
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 20 December 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
CSSE 2021
CSSE 2021: 2021 4th International Conference on Computer Science and Software Engineering
October 22 - 24, 2021
Singapore, Singapore
Acceptance Rates
Overall Acceptance Rate 33 of 74 submissions, 45%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 80Total Downloads
- Downloads (Last 12 months)5
- Downloads (Last 6 weeks)1
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format