Skip to main content
SpringerLink
Log in

Coarse-to-fine multiview 3d face reconstruction using multiple geometrical features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

3D face reconstruction from multi-view video sequences has become a hotspot in computer vision for the last decades. Structure from Motion (SfM) methods, which have been widely used for multi-view 3D face reconstruction, have two main limitations. First, self-occlusion causes certain facial feature points (FFPs) to be invisible in the images, which will lead to missing data. The existing SfM methods could recover the missing data through iterative calculation, however, with high computational costs and long processing time. Second, the SfM methods cannot reconstruct the accurate 3D facial shapes of cheeks because there are no FFPs in this area. This paper proposes a novel “coarse-to-fine” multi-view 3D face reconstruction method by taking the advantage of the complementarity between FFPs and occluding contours, i.e., the boundary lines depicted between the facial region and the background. In this method, a block SfM algorithm is firstly proposed to reconstruct a “coarse” 3D facial shape by utilizing sparse FFPs. The block SfM algorithm does not estimate the true locations of the self-occluded FFPs iteratively. Thus, the computational cost is significantly reduced. Then, a kernel partial least squares (KPLS) algorithm is introduced to refine the “coarse” 3D facial shape. The KPLS method applies occluding contours to remedy the limitation of sparse FFPs correspondence-based SfM method. The proposed method is evaluated on the synthetic sequences generated from the BJUT-3D face database and the real-world multi-view video sequences obtained in a controlled indoor environment. The results show improvements in both accuracy and efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Amberg B, Blake A, Fitzgibbon A, et al (2007) Reconstructing high quality face-surfaces using model based stereo. IEEE 11th International Conference on Computer Vision (ICCV) 2007:1–8

    Google Scholar 

  2. Aissaoui A, Martinet J, Djeraba C (2014) Rapid and accurate face depth estimation in passive stereo systems. Multimedia Tools and Applications 72(3):2413–2438

    Article  Google Scholar 

  3. Angelopoulou M, Petrou M (2014) Uncalibrated flatfielding and illumination vector estimationfor photometric stereo face reconstruction. Mach Vis Appl 25(5):1317–1332

    Article  Google Scholar 

  4. Bookstein FL (1989) Principal warps: Thin-plate splines and the decomposition of deformations. IEEE Trans Pattern Anal Mach Intell 6:567–585

    Article  MATH  Google Scholar 

  5. Bookstein Fred L (1997) Morphometric tools for landmark data: geometry and biology. Cambridge University Press

  6. Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. Proceedings of the 26th annual conference on Computer graphics and interactive techniques 1999:187–194

    Google Scholar 

  7. Chowdhury AKR, Chellappa R (2003) Face reconstruction from monocular video using uncertainty analysis and a generic model. Comput Vis Image Underst 91(1):188–213

    Article  Google Scholar 

  8. Castrilln M, Dniz O, Guerra C, et al (2007) ENCARA2: Real-time detection of multiple faces at different resolutions in video streams. J Vis Commun Image Represent 18(2):130–140

    Article  Google Scholar 

  9. Chouvatut V, Madarasmi S, Tuceryan M (2013) 3D face and motion estimation from sparse points using adaptive bracketed minimization. Multimedia tools and applications 63(2):569–589

    Article  Google Scholar 

  10. Ding L, Ding X, Fang C (2014) 3D face sparse reconstruction based on local linear fitting. Vis Comput 30(2):189–200

    Article  Google Scholar 

  11. Dai W, Milenkovic O (2010) SET: an algorithm for consistent matrix completion. IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP) 2010:3646–3649

    Google Scholar 

  12. Da F, Sui Y (2011) 3D reconstruction of human face based on an improved seeds-growing algorithm. Mach Vis Appl 22(5):879–887

    Article  Google Scholar 

  13. Emrith K, Broadbent L, Smith LN, et al (2013) Real-time recovery of moving 3D faces for emerging applications. Comput Ind 64(9):1390–1398

    Article  Google Scholar 

  14. Furukawa Y, Ponce J (2010) Accurate, dense, and robust multiview stereopsis. IEEE Trans Pattern Anal Mach Intell 32(8):1362–1376

    Article  Google Scholar 

  15. Gonzalez-Mora J, De la Torre F, Guil N, et al (2010) Learning a generic 3D face model from 2D image databases using incremental structure-from-motion. Image Vis Comput 28(7):1117–1129

    Article  Google Scholar 

  16. Hansen MF, Atkinson GA, Smith LN et al (2010) 3D face reconstructions from photometric stereo using near infrared and visible light. Comput Vis Image Underst 114(8):942–951

    Article  Google Scholar 

  17. Herold C, Despiegel V, Gentric S, et al (2014) Recursive head reconstruction from multi-view video sequences. Comput Vis Image Underst 122:182–201

    Article  Google Scholar 

  18. Jo J, Choi H, Kim IJ, et al (2015) Single-view-based 3D facial reconstruction method robust against pose variations. Pattern Recognit 48(1):73–85

    Article  Google Scholar 

  19. Jones A, Lang M, Fyffe G, et al (2009) Achieving eye contact in a one-to-many 3D video teleconferencing system. ACM Trans Graph 28(3):64

    Article  Google Scholar 

  20. Jain AK, Ross A, Prabhakar S (2004) An introduction to biometric recognition. IEEE Trans Circuits Syst Video Technol 14(1):4–20

    Article  Google Scholar 

  21. Kemelmacher-Shlizerman I, Basri R (2011) 3d face reconstruction from a single image using a single reference face shape. IEEE Trans Pattern Anal Mach Intell 33(2):394–405

    Article  Google Scholar 

  22. Kurtek S, Drira H (2015) A comprehensive statistical framework for elastic shape analysis of 3D faces. Comput Graph 51:52–59

    Article  Google Scholar 

  23. Koo HS, Lam KM (2008) Recovering the 3d shape and poses of face images based on the similarity transform. Pattern Recogn Lett 29(6):712–723

    Article  Google Scholar 

  24. Lin WY, Chen MY (2014) A novel framework for automatic 3D face recognition using quality assessment. Multimedia tools and applications 68(3):877–893

    Article  Google Scholar 

  25. Lee SJ, Park KR, Kim J (2011) A SfM-based 3D face reconstruction method robust to self-occlusion by using a shape conversion matrix. Pattern Recogn 44(7):1470–1486

    Article  Google Scholar 

  26. Lin Y, Medioni G, Choi J (2010) Accurate 3D face reconstruction from weakly calibrated wide baseline images with profile contours. IEEE Conf Comput Vis Pattern Recognit (CVPR) 2010:1490–1497

    Google Scholar 

  27. Lhuillier M, Quan L (2005) A quasi-dense approach to surface reconstruction from uncalibrated images. IEEE Trans Pattern Anal Mach Intell 27(3):418–433

    Article  Google Scholar 

  28. Lin K, Wang X, Tan Y (2015) Self-adaptive morphable model based collaborative multi-view 3d face reconstruction in visual sensor network. Multimedia Tools and Applications 2015:1–23

    Google Scholar 

  29. Levine MD, Yu YC (2009) State-of-the-art of 3D facial reconstruction methods for face recognition based on a single 2D training image per person. Pattern Recogn Lett 30(10):908–913

    Article  Google Scholar 

  30. Marques M, Costeira J (2009) Estimating 3D shape from degenerate sequences with missing data. Comput Vis Image Underst 113(2):261–272

    Article  Google Scholar 

  31. Medioni G, Choi J, Kuo CH, et al (2009) Identifying noncooperative subjects at a distance using face images and inferred three-dimensional face models. IEEE Trans Syst Man Cybern Syst Hum 39(1):12–24

    Article  Google Scholar 

  32. Maejima A, Wemler S, Machida T, et al (2008) Instant casting movie theater: the future cast system. IEICE Trans Inf Syst 91(4):1135–1148

    Article  Google Scholar 

  33. Ohue K, Yamada Y, Uozumi S, et al (2006) Development of a new pre-crash safety system. SAE Technical Paper:2006

  34. Peng W, Xu C, Feng Z (2016) 3D face modeling based on structure optimization and surface reconstruction with B-Spline. Neurocomputing 179:228–237

    Article  Google Scholar 

  35. Rannar S, Lindgren F, Geladi P, et al (1994) A PLS kernel algorithm for data sets with many variables and fewer objects. Part 1: Theory and algorithm. J Chemom 8(2):111–125

    Article  Google Scholar 

  36. Snchez-Escobedo D, Casteln M (2013) 3D face shape prediction from a frontal image using cylindrical coordinates and partial least squares. Pattern Recogn Lett 34(4):389–399

    Article  Google Scholar 

  37. Snchez-Escobedo D, Casteln M, Smith WAP (2016) Statistical 3D face shape estimation from occluding contours. Comput Vis Image Underst 142:111–124

    Article  Google Scholar 

  38. Sun Y, Dong J, Jian M, et al (2015) Fast 3D face reconstruction based on uncalibrated photometric stereo. Multimedia Tools and Applications 74(11):3635–3650

    Article  Google Scholar 

  39. Schendel SA, Jacobson R, Khalessi S (2013) 3-dimensional facial simulation in orthognathic surgery: is it accurate?. J Oral Maxillofac Surg 71(8):1406–1414

    Article  Google Scholar 

  40. Sun ZL, Lam KM, Gao QW (2013) Depth estimation of face images using the nonlinear least-squares model. IEEE Trans Image Process 22(1):17–30

    Article  MathSciNet  MATH  Google Scholar 

  41. Stylianou G, Lanitis A (2009) Image based 3D face reconstruction: a survey. International Journal of Image and Graphics 9(02):217–250

    Article  Google Scholar 

  42. Song M, Tao D, Huang X, et al (2012) Three-dimensional face reconstruction from a single image by a coupled RBF network. IEEE Trans Image Process 21(5):2887–2897

    Article  MathSciNet  MATH  Google Scholar 

  43. Strecha C, von Hansen W, Gool LV, et al (2008) On benchmarking camera calibration and multi-view stereo for high resolution imagery. IEEE Conf Comput Vis Pattern Recognit (CVPR) 2008:1–8

    Google Scholar 

  44. Tomasi C, Kanade T (1992) Shape and motion from image streams under orthography: a factorization method. Int J Comput Vis 9(2):137–154

    Article  Google Scholar 

  45. Tech M (2005) The BJUT-3D large-scale Chinese face database. Graphics Lab, Technical Report, Beijing University of Technology, 2005

  46. Vandereycken B (2013) Low-rank matrix completion by Riemannian optimization. SIAM J Optim 23(2):1214–1236

    Article  MathSciNet  MATH  Google Scholar 

  47. Wang N, Gao X, Tao D, et al (2014) Facial feature point detection: A comprehensive survey. arXiv:1410.1037

  48. Zeng D, Zhao Q, Long S, et al (2016) Examplar coherent 3D face reconstruction from forensic mugshot database. Image Vis Comput:2016

Download references

Acknowledgments

This paper is supported by National Natural Science Foundation of China under Grant #61472216, and by PhD Programs Foundation of Ministry of Education of China under Grant #20120002110067.

Author information

Authors and Affiliations

  1. State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, Beijng, 100084, China

    Peng Dai, Xue Wang & Weihang Zhang

Authors
  1. Peng Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xue Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Weihang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xue Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dai, P., Wang, X. & Zhang, W. Coarse-to-fine multiview 3d face reconstruction using multiple geometrical features. Multimed Tools Appl 77, 939–966 (2018). https://doi.org/10.1007/s11042-016-4325-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-4325-y

Keywords

Search

Navigation