Skip to main content
SpringerLink
Log in

Facial expression synthesis with direction field preservation based mesh deformation and lighting fitting based wrinkle mapping

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Nowadays, facial expression synthesis is widely used in expression simulation, recognition and animation. While a variety of works available in literature are 3D based, these algorithms usually require face matching and trimming, are thus time consuming. In this work, an automatic algorithm for expression synthesis in 2D space is proposed, which mainly consists of three stages.The optimum matching of three sets of feature points on the faces of source neutral (F s n ), source expression (F s e ) and target neutral (F t n ) are obtained in the first stage. Different components on the target face are deformed by learning from not only the displacements but also the geometry shape of face F s e in the second stage. In the last stage, the details of the source expression are mapped onto the corresponding positions on the target face by fitting of the lighting differences. Experimental results on expression synthesis with large geometry deformation and lighting difference show that the proposed algorithm is able to accurately preserve the geometry deformation, and the synthesized expressions are visually realistic.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

References

  1. Al-Osaimi F, Bennamoun M, Mian A (2009) An expression deformation approach to non-rigid 3d face recognition. Int J Comput Vis 81(3):302–316

    Article  Google Scholar 

  2. Bando Y, Kuratate T, Nishita T (2002) A simple method for modeling wrinkles on human skin. In: the 10th Pacific Conference on Computer Graphics and Applications, IEEE Computer Society, p 166

  3. Beier T, Neely S (1992) Feature-based image metamorphosis. In: ACM SIGGRAPH Computer Graphics, ACM, pp 35–42

  4. Beymer D, Shashua A, Poggio T (1993) Example based image analysis and synthesis. PhD thesis, Artificial Intelligence Laboratory, Massachusetts Institute of Technology

  5. Blanz V, Basso C, Poggio T, Vetter T (2003) Reanimating faces in images and video. Comput Graph Forum 22(3):641–650

    Article  Google Scholar 

  6. Debevec P (1998) Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: SIGGRAPH, ACM, pp 189–198

  7. Gibbons J, Chakraborti S (2003) Nonparametric Statistical Inference. Marcel Dekker

  8. Huang D, De la Torre F (2010) Bilinear kernel reduced rank regression for facial expression synthesis. In: the 11th European Conference on Computer Vision. Springer-Verlag, Verlag, pp 364–377

  9. Kazemi V, Josephine S (2014) One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp 1867–1874

  10. Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16(1):172–187

    Article  MathSciNet  Google Scholar 

  11. Larboulette C, Cani M (2004) Real-time dynamic wrinkles. In: the Computer Graphics International, IEEE Computer Society, pp 522–525

  12. Lee Y, Terzopoulos D, Waters K (1995) Realistic modeling for facial animation. In: the 22nd Annual Conference on Computer Graphics and Interactive Techniques, ACM, pp 55–62

  13. Li K, Dai Q, Wang R, Liu Y, Xu F, Wang J (2014) A data-driven approach for facial expression retargeting in video. IEEE Trans Multimedia 16(2):299–310

    Article  Google Scholar 

  14. Liao J, Lima R, Nehab D, Hoppe H, Sander P, Yu J (2014) Automating image morphing using structural similarity on a halfway domain. ACM Trans Graph 33(5):1–12

    Article  Google Scholar 

  15. Liu Z, Shan Y, Zhang Z (2001) Expressive expression mapping with ratio images. In: the 28th Annual Conference on Computer Graphics and Interactive Techniques, ACM, pp 271–276

  16. Lu X, Jain A (2008) Deformation modeling for robust 3d face matching. IEEE Trans Pattern Anal Mach Intell 30(8):1346–1356

    Article  Google Scholar 

  17. Lucey P, Cohn J, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPRW, IEEE, pp 94– 101

  18. Matthews I, Xiao J, Baker S (2007) 2d vs. 3d deformable face models: representational power, construction, and real-time fitting. Int J Comput Vis 75(1):93–113

    Article  Google Scholar 

  19. Pan G, Han S, Wu Z, Zhang Y (2010) Removal of 3d facial expressions: a learning-based approach. In: Computer Vision and Pattern Recognition, CVPR, IEEE, pp 2614–2621

  20. Qian K, Wang B, Chen H (2014) Automatic flexible face replacement with no auxiliary data. Comput & Graph 45(0):64–74

    Article  Google Scholar 

  21. Raouzaiou A, Tsapatsoulis N, Karpouzis K, Kollias S (2002) Parameterized facial expression synthesis based on mpeg-4. EURASIP J Appl Signal Processing 2002(1):1021–1038

    MATH  Google Scholar 

  22. Ren S, Cao X, Wei Y, Sun J (2014) Face alignment at 3000 fps via regressing local binary features. In: IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp 1685–1692

  23. Sanchez A, Ruiz J, Moreno A, Montemayor A, Hernandez J, Pantrigo J (2011) Differential optical flow applied to automatic facial expression recognition. Neurocomputing 74(8):1272–1282

    Article  Google Scholar 

  24. Seitz S, Dyer C (1996) View morphing. In: the 23rd Annual Conference on Computer Graphics and Interactive Techniques, ACM, pp 21–30

  25. Song M, Dong Z, Theobalt C, Wang H, Liu Z, Seidel H (2007) A generic framework for efficient 2-d and 3-d facial expression analogy. IEEE Trans Multimedia 9(7):1384–1395

    Article  Google Scholar 

  26. Sumner R, Popovic J (2004) Deformation transfer for triangle meshes. ACM Trans Graph 23(3):399–405

    Article  Google Scholar 

  27. Tzimiropoulos G, Pantic M (2013) Optimization problems for fast aam fitting in-the-wild. In: IEEE International Conference on Computer Vision, ICCV, IEEE, pp 593–600

  28. Vlasic D, Brand M, Pfister H, Popovic J (2005) Face transfer with multilinear models. ACM Trans Graph 24(3):426–433

    Article  Google Scholar 

  29. Weise T, Bouaziz S, Li H, Pauly M (2011) Realtime performance-based facial animation. ACM Trans Graph 30(4):76–79

    Article  Google Scholar 

  30. Weng Y, Xu W, Wu Y, Zhou K, Guo B (2006) 2d shape deformation using nonlinear least squares optimization. Visual Comput 22(9–11):653–660

    Article  Google Scholar 

  31. Xie W, Shen L, Yang M, Hou Q (2015) Lighting difference based wrinkle mapping for expression synthesis. In: 2015 8th International Congress on Image and Signal Processing, CISP, IEEE, pp 636– 641

  32. Yu H, Garrod O, Jack R, Schyns P (2015) A framework for automatic and perceptually valid facial expression generation. Multimed Tools Appl 74(21):9427–9447

    Article  Google Scholar 

  33. Zhang Q, Liu Z, Guo B, Terzopoulos D, Shum H (2006) Geometry-driven photorealistic facial expression synthesis. IEEE Trans Vis Comput Graph 12(1):48–60

    Article  Google Scholar 

  34. Zhang Y, Lin W, Zhou B, Chen Z, Sheng B, Wu J (2014) Facial expression cloning with elastic and muscle models. J Vis Commun Image Represent 25(5):916–927

    Article  Google Scholar 

Download references

Acknowledgments

The authors thank the anonymous reviewers for their helpful comments and suggestions. The work was supported by Natural Science Foundation of China under grand no. 61672357, 61602315, 61402289 and 61272050, China Postdoctoral Science Foundation under grant no. 2015M572363, the Science Foundation of Guangdong Province under grant no. 2014A030313556 and 2014A030313558.

Author information

Authors and Affiliations

  1. Computer Vision Institute, School of Computer Science & Software Engineering, Shenzhen University, Shenzhen, China

    Weicheng Xie, Linlin Shen & Meng Yang

  2. Research Institute for Future Media Computing, School of Computer Science & Software Engineering, Shenzhen University, Shenzhen, China

    Jianmin Jiang

Authors
  1. Weicheng Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Linlin Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianmin Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Linlin Shen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xie, W., Shen, L., Yang, M. et al. Facial expression synthesis with direction field preservation based mesh deformation and lighting fitting based wrinkle mapping. Multimed Tools Appl 77, 7565–7593 (2018). https://doi.org/10.1007/s11042-017-4661-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4661-6

Keywords

Search

Navigation