Skip to main content
SpringerLink
Log in

From 2D to 3D real-time expression transfer for facial animation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we present a three-stage approach, which creates realistic facial animations by tracking expressions of a human face in 2D and transferring them to a human-like 3D model in real-time. Our calibration-free method, which is based on an average human face, does not require training. The tracking is performed using a single camera to enable several practical applications, for example, using tablets and mobile devices, and the expressions are transferred with a joint-based system to improve the quality and persuasiveness of animations. In the first step of the method, a joint-based facial rig providing mobility to pseudo-muscles is attached to the 3D model. The second stage covers the tracking of 2D positions of the facial landmarks from a single camera view and transfer of 3D relative movement data to move the respective joints on the model. The last step includes the recording of animation using a partially automated key-framing technique. Experiments on the extended Cohn-Kanade dataset using peak frames in frontal-view videos have shown that the presented method produces visually satisfying facial animations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Video: (https://vimeo.com/153692017)

References

  1. Al-Nuaimi T (2006) Face recognition and computer graphics for modelling expressive faces in 3D. PhD thesis, Massachusetts Institute of Technology

  2. Autodesk (2015) Makehuman. https://www.autodesk.com/education/free-software/maya

  3. Black MJ, Yacoob Y (1995) Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In: Proceedings of IEEE international conference on computer vision, pp 374–381

  4. Cao C, Hou Q, Zhou K (2014) Displaced dynamic expression regression for real-time facial tracking and animation. ACM Trans Graph 33(4):1–10

    Google Scholar 

  5. Cao C, Wu H, Weng Y, Shao T, Zhou K (2016) Real-time facial animation with image-based dynamic avatars. ACM Trans Graph 35(4):1–12

    Article  Google Scholar 

  6. Chai J, Xiao J, Hodgins J (2003) Vision-based control of 3d facial animation. In: Proceedings of the 2003 ACM SIGGRAPH / Eurographics symposium on computer animation, Eurographics Association, Aire-la-Ville, Switzerland, pp 193-206

  7. Cohen M M, Massaro D W (1990) Synthesis of visible speech. Behav Res Methods Instrum Comput 22(2):260–263

    Article  Google Scholar 

  8. Cohen M M, Massaro D W, et al. (1993) Modeling coarticulation in synthetic visual speech. In: Thalmann NM, Thalmann D (eds) Models and techniques in computer animation. Springer Japan, Tokyo, pp 139–156

    Chapter  Google Scholar 

  9. Cootes T F, Edwards G J, Taylor C J (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23(6):681–685

    Article  Google Scholar 

  10. Covell M (1996) Eigen-points: Control-point location using principal component analyses. In: Proceedings of the 2nd international conference on automatic face and gesture recognition, pp 122–127

  11. DeCarlo D, Metaxas D (2000) Optical flow constraints on deformable models with applications to face tracking. Int J Comput Vis 38(2):99–127

    Article  Google Scholar 

  12. Deng Z, Neumann U (2008) Data-driven 3D facial animation. Springer, London

    Google Scholar 

  13. Ekmen B, Ekenel HK (2016) Real time animated facial expression transfer. In: 2016 IEEE 24th signal processing and communication application conference (SIU), pp 1193–1196

  14. Essa I, Basu S, Darrell T, Pentland A (1996) Modeling, tracking and interactive animation of faces and heads//using input from video. In: Computer animation ’96. Proceedings, pp 68–79

  15. Faceshift (2017) Faceshift. http://faceshift.com/studio/2015.2/index.html

  16. Gambaretto E, Piña C (2014) Real-time animation of cartoon character faces. In: ACM SIGGRAPH 2014 Computer Animation Festival, ACM, New York, NY, USA, SIGGRAPH ’14, p 1

  17. Gokturk S B, Bouguet J Y, Grzeszczuk R (2001) A data-driven model for monocular face tracking. In: Proceedings 8th IEEE international conference on computer vision. ICCV 2001, vol 2, pp 701–708

  18. Ichim A E, Bouaziz S, Pauly M (2015) Dynamic 3D avatar creation from hand-held video input. ACM Trans Graph 34(4):1–14

    Article  Google Scholar 

  19. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings 4th IEEE international conference on automatic face and gesture recognition (Cat. No. PR00580), pp 46–53

  20. Lanitis A, Taylor C J, Cootes T F (1997) Automatic interpretation and coding of face images using flexible models. IEEE Trans Pattern Anal Mach Intell 19 (7):743–756

    Article  Google Scholar 

  21. Laursen M H, Pedersen K S (2012) Partially automated system for synthesising human facial expressions in interactive media, Master’s thesis, Aalborg University

  22. Lewis J P, Parke F I (1986) Automated lip-synch and speech synthesis for character animation. SIGCHI Bull 17(SI):143–147

    Article  Google Scholar 

  23. Lewis JP, Pighin F (2006) Retargeting: Algorithms for performance-driven animation. In: ACM SIGGRAPH 2006 Courses, ACM, New York, NY, USA, SIGGRAPH ’06

  24. Li L, Liu Y, Zhang H (2012) A survey of computer facial animation techniques. In: 2012 International conference on computer science and electronics engineering, vol 3, pp 434–438

  25. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society conference on computer vision and pattern recognition - workshops, pp 94–101

  26. MakeHuman (2015) Makehuman. http://www.makehuman.org

  27. Noh J, Neumann U (1998) A survey of facial modeling and animation techniques. Tech. rep

  28. Parke FI (1974) A parametric model for human faces. Tech. rep

  29. Parke FI, Waters K (2008) Computer facial animation. CRC Press, New York

    Book  Google Scholar 

  30. Pelachaud C, Badler N I, Steedman M (1996) Generating facial expressions for speech. Cogn Sci 20(1):1–46

    Article  Google Scholar 

  31. Pighin F, Szeliski R, Salesin D H (1999) Resynthesizing facial animation through 3d model-based tracking. In: Proceedings of the 7th IEEE international conference on computer vision, vol 1, pp 143–150

  32. Platt S M, Badler N I (1981) Animating facial expressions. SIGGRAPH Comput Graph 15(3):245–252

    Article  Google Scholar 

  33. Ruhland K, Prasad M, McDonnell R (2017) Data-driven approach to synthesizing facial animation using motion capture. IEEE Comput Graph Appl 37 (4):30–41

    Article  Google Scholar 

  34. Singular Inversions (2015) Facegen modeller. https://facegen.com

  35. Sreeja P S, Mahalakshmi G S (2017) Emotion models: a review. Int J Control Theory Appl 10:651–657

    Google Scholar 

  36. Terzopoulos D, Waters K (1993) Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Trans Pattern Anal Mach Intell 15 (6):569–579

    Article  Google Scholar 

  37. Thies J, Zollhöfer M, Stamminger M, Theobalt C, Nießner M (2016) Face2Face: Real-time face capture and reenactment of RGB videos. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 2387–2395

  38. Villagrasa S, Susín Sánchez A (2009) Face! 3D facial animation system based on facs. In: IV Iberoamerican symposium in computer graphics, pp 203–209

  39. Waters K (1987) A muscle model for animation three-dimensional facial expression. SIGGRAPH Comput Graph 21(4):17–24

    Article  Google Scholar 

  40. Williams L (1990) Performance-driven facial animation. SIGGRAPH Comput Graph 24(4):235–242

    Article  Google Scholar 

  41. Xiong X, la Torre FD (2013) Supervised descent method and its applications to face alignment. In: 2013 IEEE conference on computer vision and pattern recognition, pp 532–539

  42. Yehia H, Rubin P, Vatikiotis-Bateson E (1998) Quantitative association of vocal-tract and facial behavior. Speech Comm 26(1):23–43

    Article  Google Scholar 

  43. Zhao H, Tai C L (2007) Subtle facial animation transfer from 2D videos to 3D faces with laplacian deformation. Proceedings of computer animation and social agents, Hasselt, Belgium

Download references

Acknowledgements

This work was supported by the TÜBİTAK project 113E067 and the EU Seventh Framework Programme Marie Curie FP7 integration project.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Istanbul Technical University, Maslak, Istanbul, 34469, Turkey

    Beste Ekmen & Hazım Kemal Ekenel

Authors
  1. Beste Ekmen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hazım Kemal Ekenel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Beste Ekmen.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ekmen, B., Ekenel, H.K. From 2D to 3D real-time expression transfer for facial animation. Multimed Tools Appl 78, 12519–12535 (2019). https://doi.org/10.1007/s11042-018-6785-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6785-8

Keywords

Search

Navigation