Skip to main content
SpringerLink
Log in

“Edutainment 2017” a visual and semantic representation of 3D face model for reshaping face in images

  • Regular Paper
  • Published:
Journal of Visualization Aims and scope Submit manuscript

Abstract

Reshaping faces is an interesting thing and a common demand in today’s media industry and cosmetic industry. Especially, the reshaped face images contribute a lot to special effects industry and plastic-beauty industry. In this paper, we provide a novel parametric representation of 3D face model that describes the face shape as a linear combination of semantic and non-semantic bases. The semantic (local) bases correspond to each individual face part and they are used to semantically edit each face part, while the non-semantic bases explain other shape variations to span the subspace of the face shape. First, we build a sparse and spatially localized parametric face model from a dataset of 3D face models by sparse principal component analysis. Second, to define the semantic bases, we train a regression model to correlate semantically significant values like nose height and mouth width. Finally, the bases of the resulting parametric face model is orthogonalized to all defined semantic bases by the Gram–Schmidt algorithm for generating the novel parametric representation of 3D face model. The novel representation can be applied in reshaping face in the images. The experiment results demonstrate that our representation not only retains the accuracy for 3D face reconstruction but also provides users a user-friendly tool to edit facial parts for desired facial shapes.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Alvarez FJA, Parra EBB, Tubio FM (2017) Improving graphic expression training with 3D models. J Vis 20(4):889–904

    Article  Google Scholar 

  • Bach F, Jenatton R, Mairal J, Obozinski G (2012) Optimization with sparsity-inducing penalties. Found Trends Mach Learn 4(1):1–106

    Article  MATH  Google Scholar 

  • Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th annual conference on computer graphics and interactive techniques. ACM, pp 187–194

  • Blanz V, Basso C, Poggio T, Vetter T (2003) Reanimating faces in images and video. Comput Gr Forum (Blackwell Publishing, Inc) 22(3):641–650

    Article  Google Scholar 

  • Blanz V, Scherbaum K, Vetter T, Seidel H-P (2004) Exchanging faces in images. Comput Gr Forum (Blackwell Publishing, Inc) 23(3):669–676

    Article  Google Scholar 

  • Cao C, Weng Y, Zhou S, Tong Y, Zhou K (2014) Facewarehouse: a 3d facial expression database for visual computing. Vis Comput Gr IEEE Trans 20(3):413–425

    Article  Google Scholar 

  • Chalas I, Urbanova P, Jurik V, Ferkova Z, Jandova M, Sochor J, Kozlikova B (2017) Generating various composite human faces from real 3D facial images. V Comput 33(4):443–458

    Google Scholar 

  • Chou J-K, Yang C-K, Gong S-D (2012) Face-off: automatic alteration of facial features. Multimed Tools Appl 56(3):569–596

    Article  Google Scholar 

  • Fan X, Chai Z, Feng Y, Wang Y, Wang S, Lou Z (2016) An efficient mesh-based face beautifier on mobile devices. Neurocomputing. 172:134–142

    Article  Google Scholar 

  • Golub GH, Vanloan CF (1996) Matrix computations (Johns Hopkins studies in mathematical sciences). The Johns Hopkins University Press, Baltimore

    Google Scholar 

  • Guo Y, Zhang J, Cai J, Jiang B, Zheng J (2017) 3DFaceNet: real-time dense face reconstruction via synthesizing photo-realistic face images. arXiv:1708.00980

  • Hasler N, Stoll C, Sunkel M, Rosenhahn B, Seidel H-P (2009) A statistical model of human pose and body shape. Comput Gr Forum 28(2):337–346

    Article  Google Scholar 

  • Kang D, Tian F, Sahandi R (2017) Interactive illustration of collage for children with folktale e-book. J Vis 20(3):639–650

    Article  Google Scholar 

  • Kavan L, Sloan PP, O’Sullivan C (2010) Fast and efficient skinning of animated meshes. Comput Gr Forum (Blackwell Publishing Ltd) 29(2):327–336

    Article  Google Scholar 

  • Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1867–1874

  • Kemelmacher-Shlizerman I (2016) Transfiguring portraits. ACM Trans Gr (TOG) 35(4):94 (1–94:8)

    Google Scholar 

  • Klum S, Han H, Jain AK, Klara B (2013) Sketch based face recognition: forensic vs. composite sketches. In: Proceedings of the 6th international conference on biometrics. IEEE, pp 1–8

  • Lewis JP, Anjyo K, Rhee T, Zhang M, Pighin FH, Deng Z (2014) Practice and theory of blendshape facial models. Eurographics (State Art Rep) 1(8)

  • Leyvand T, Cohen-Or D, Dror G, Lischinski D (2008) Data-driven enhancement of facial attractiveness. ACM Trans Gr (TOG) 27(3):38

    Google Scholar 

  • Li H, Yu J, Ye Y, Bregler C (2013) Realtime facial animation with on-the-fly correctives. ACM Trans Gr (TOG) 32(4):42 (1–42:10)

    MATH  Google Scholar 

  • Liao Q, Jin X, Zeng W (2012) Enhancing the symmetry and proportion of 3D face geometry. Vis Comput Gr IEEE Trans 18(10):1704–1716

    Article  Google Scholar 

  • Lu J, Sunkavalli K, Carr N, Hadap S, Forsyth D (2016) A visual representation for editing face images. arXiv:1612.00522

  • Lu Z, Yang J, Liu Q (2017) Face image retrieval based on shape and texture feature fusion. Comput Visual Media 3(4):359–368

    Article  Google Scholar 

  • Ma DS, Correll J, Wittenbrink B (2015) The Chicago face database: a free stimulus set of faces and norming data. Behav Res Methods 47(4):1122–1135

    Article  Google Scholar 

  • Mansouri S, Ebrahimnezhad H (2016) Segmentation-based semi-regular remeshing of 3D models using curvature-adapted subdivision surface fitting. J Vis 19(1):141–155

    Article  Google Scholar 

  • Neumann T, Varanasi K, Wenger S, Wacker M, Magnor M, Theobalt C (2013) Sparse localized deformation components. ACM Trans Gr (TOG) 32(6):179

    Google Scholar 

  • Nirkin Y, Masi I, Tran AT, Hassner T, Medioni G (2017) On face segmentation, face swapping, and face perception. arXiv:1704.06729

  • Paysan P, Knothe R, Amberg B (2009) A 3D face model for pose and illumination invariant face recognition. In: 6th IEEE international conference on advanced video and signal based surveillance (AVSS’09), IEEE, Genova, Italy, pp 296–301

  • Ramamoorthi R, Hanrahan P (2001) An efficient representation for irradiance environment maps. In: Proceedings of the 28th annual conference on computer graphics and interactive techniques. ACM, pp 497–500

  • Shamir A, Sorkine O (2009) Visual media retargeting. ACM SIGGRAPH ASIA 2009 courses. ACM, p 11

  • Streuber S, Quiros-Ramirez MA, Hill MQ, Hahn CA, Zuffi S, O’Toole A, Black MJ (2016) Body talk: crowdshaping realistic 3D avatars with words. ACM Trans Gr (TOG) 35(4):54

    Google Scholar 

  • Suryanarayana GK, Dubey R (2017) Image analyses of supersonic air-intake buzz and control by natural ventilation. J Vis 1–17

  • Thies J, Zollhofer M, Stamminger M, Theobalt C, Niebner M (2016) Face2face: real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 2387–2395

  • Tran AT, Hassner T, Masi I, Medioni G (2016) Regressing robust and discriminative 3D morphable models with a very deep neural network. arXiv:1612.04904

  • Vlasic D, Brand M, Pfister H, Popovic J (2005) Face transfer with multilinear models. ACM Trans Gr (TOG) 24(3):426–433

    Article  Google Scholar 

  • Zhang Y, Lin W, Zhou B, Chen Z, Sheng B, Wu J (2014) Facial expression cloning with elastic and muscle models. J Vis Commun Image Represent 25(5):916–927

    Article  Google Scholar 

  • Zhou S, Fu H, Liu L, Cohen-Or D, Han X (2010) Parametric reshaping of human bodies in images. ACM Trans Gr (TOG) 29(4):126

    Google Scholar 

Download references

Acknowledgements

The research is supported in part by NSFC (61572424) and the People Programme (Marie Curie Actions) of the European Union’s Seventh Framework Programme FP7 (2007-2013) under REA Grant Agreement no. 612627-”AniNex”. Min Tang is supported in part by NSFC (61572423) and Zhejiang Provincial NSFC (LZ16F020003).

Author information

Authors and Affiliations

  1. State Key Lab of CAD&CG, Zhejiang University, Hangzhou, 310027, China

    Jiang Du, Dan Song, Yanlong Tang, Ruofeng Tong & Min Tang

Authors
  1. Jiang Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dan Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yanlong Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ruofeng Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Min Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ruofeng Tong.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Du, J., Song, D., Tang, Y. et al. “Edutainment 2017” a visual and semantic representation of 3D face model for reshaping face in images. J Vis 21, 649–660 (2018). https://doi.org/10.1007/s12650-018-0476-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12650-018-0476-4

Keywords

Search

Navigation