skip to main content
research-article

Displaced dynamic expression regression for real-time facial tracking and animation

Published: 27 July 2014 Publication History

Abstract

We present a fully automatic approach to real-time facial tracking and animation with a single video camera. Our approach does not need any calibration for each individual user. It learns a generic regressor from public image datasets, which can be applied to any user and arbitrary video cameras to infer accurate 2D facial landmarks as well as the 3D facial shape from 2D video frames. The inferred 2D landmarks are then used to adapt the camera matrix and the user identity to better match the facial expressions of the current user. The regression and adaptation are performed in an alternating manner. With more and more facial expressions observed in the video, the whole process converges quickly with accurate facial tracking and animation. In experiments, our approach demonstrates a level of robustness and accuracy on par with state-of-the-art techniques that require a time-consuming calibration step for each individual user, while running at 28 fps on average. We consider our approach to be an attractive solution for wide deployment in consumer-level applications.

Supplementary Material

ZIP File (a43-cao.zip)
Supplemental material.
MP4 File (a43-sidebyside.mp4)

References

[1]
Asthana, A., Zafeiriou, S., Cheng, S., and Pantic, M. 2013. Robust discriminative response map fitting with constrained local models. In IEEE CVPR, 3444--3451.
[2]
Baltrušaitis, T., Robinson, P., and Morency, L.-P. 2012. 3D constrained local model for rigid and non-rigid facial tracking. In Proceedings of IEEE CVPR, 2610--2617.
[3]
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R. W., and Gross, M. 2011. High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. 30, 4, 75:1--75:10.
[4]
Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proceedings of SIGGRAPH, 187--194.
[5]
Bouaziz, S., Wang, Y., and Pauly, M. 2013. Online modeling for realtime facial animation. ACM Trans. Graph. 32, 4 (July), 40:1--40:10.
[6]
Bradley, D., Heidrich, W., Popa, T., and Sheffer, A. 2010. High resolution passive facial performance capture. ACM Trans. Graph. 29, 4, 41:1--41:10.
[7]
Burgos-Artizzu, X. P., Perona, P., and Dollár, P. 2013. Robust face landmark estimation under occlusion. In Proceedings of ICCV, 117--124.
[8]
Byrd, R. H., Lu, P., Nocedal, J., and Zhu, C. 1995. A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comput. 16, 5 (Sept.), 1190--1208.
[9]
Cao, X., Wei, Y., Wen, F., and Sun, J. 2012. Face alignment by explicit shape regression. Proceedings of IEEE CVPR, 2887--2894.
[10]
Cao, C., Weng, Y., Lin, S., and Zhou, K. 2013. 3d shape regression for real-time facial animation. ACM Trans. Graph. 32, 4 (July), 41:1--41:10.
[11]
Cao, C., Weng, Y., Zhou, S., Tong, Y., and Zhou, K. 2013. Facewarehouse: a 3D facial expression database for visual computing. IEEE TVCG, PrePrints.
[12]
Chai, J.-X., Xiao, J., and Hodgins, J. 2003. Vision-based control of 3d facial animation. In Symp. Comp. Anim., 193--206.
[13]
Cootes, T. F., Taylor, C. J., Cooper, D. H., and Graham, J. 1995. Active shape models - their training and application. Computer Vision and Image Understanding 61, 38--59.
[14]
Cootes, T. F., Edwards, G. J., and Taylor, C. J. 1998. Active appearance models. In Proceedings of ECCV, 484--498.
[15]
DeCarlo, D., and Metaxas, D. 2000. Optical flow constraints on deformable models with applications to face tracking. Int. Journal of Computer Vision 38, 2, 99--127.
[16]
Dollar, P., Welinder, P., and Perona, P. 2010. Cascaded pose regression. In Proceedings of IEEE CVPR, 1078--1085.
[17]
Ekman, P., and Friesen, W. 1978. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press.
[18]
Essa, I., Basu, S., Darrell, T., and Pentland, A. 1996. Modeling, tracking and interactive animation of faces and heads: using input from video. In Computer Animation, 68--79.
[19]
Garrido, P., Valgaert, L., Wu, C., and Theobalt, C. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graph. 32, 6 (Nov.), 158:1--158:10.
[20]
Huang, G. B., Ramesh, M., Berg, T., and Learned-Miller, E. 2007. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Tech. Rep. 07-49, University of Massachusetts, Amherst, October.
[21]
Huang, H., Chai, J., Tong, X., and Wu, H.-T. 2011. Leveraging motion capture and 3d scanning for high-fidelity facial performance acquisition. ACM Trans. Graph. 30, 4, 74:1--74:10.
[22]
Lewis, J. P., and Anjyo, K. 2010. Direct manipulation blendshapes. IEEE CG&A 30, 4, 42--50.
[23]
Li, H., Yu, J., Ye, Y., and Bregler, C. 2013. Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. 32, 4 (July), 42:1--42:10.
[24]
Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. H. 1998. Synthesizing realistic facial expressions from photographs. In Proceedings of SIGGRAPH, 75--84.
[25]
Pighin, F., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3d model-based tracking. In Int. Conf. Computer Vision, 143--150.
[26]
Saragih, J. M., Lucey, S., and Cohn, J. F. 2011. Real-time avatar animation from a single image. In IEEE International Conference on Automatic Face Gesture Recognition and Workshops, 117--124.
[27]
Saragih, J., Lucey, S., and Cohn, J. 2011. Deformable model fitting by regularized landmark mean-shift. International Journal of Computer Vision 91, 2, 200--215.
[28]
Tarres, F., and Rama, A. GTAV Face Database. A http://gps-tsc.upc.es/GTAV/ResearchAreas/UPCFaceDatabase/GTAVFaceDatabase.htm.
[29]
Viola, P., and Jones, M. 2004. Robust real-time face detection. International Journal of Computer Vision 57, 2, 137--154.
[30]
Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Trans. Graph. 24, 3, 426--433.
[31]
Weise, T., Li, H., Gool, L. V., and Pauly, M. 2009. Face/off: Live facial puppetry. In Symp. Computer Animation, 7--16.
[32]
Weise, T., Bouaziz, S., Li, H., and Pauly, M. 2011. Realtime performance-based facial animation. ACM Trans. Graph. 30, 4 (July), 77:1--77:10.
[33]
Weng, Y., Cao, C., Hou, Q., and Zhou, K. 2013. Real-time facial animation on mobile devices. Graphical Models, PrePrints.
[34]
Williams, L. 1990. Performance-driven facial animation. In Proceedings of SIGGRAPH, 235--242.
[35]
Xiao, J., Baker, S., Matthews, I., and Kanade, T. 2004. Real-time combined 2d+3d active appearance models. In Proceedings of IEEE CVPR, 535--542.
[36]
Xiong, X., and De La Torre, F. 2013. Supervised descent method and its applications to face alignment. In Proceedings of IEEE CVPR, 532--539.
[37]
Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Trans. Graph. 23, 3, 548--558.

Cited By

View all
  • (2024)The Rising Threat of Deepfake Technology and Frightening Advancements of Social EngineeringEffective Strategies for Combatting Social Engineering in Cybersecurity10.4018/979-8-3693-6665-3.ch014(307-330)Online publication date: 27-Sep-2024
  • (2024)3D Gaussian Blendshapes for Head Avatar AnimationACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657462(1-10)Online publication date: 13-Jul-2024
  • (2024)LIA: Latent Image AnimatorIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.344907546:12(10829-10844)Online publication date: 1-Dec-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 33, Issue 4
July 2014
1366 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/2601097
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2014
Published in TOG Volume 33, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. blendshape models
  2. face animation
  3. face tracking
  4. performance capture

Qualifiers

  • Research-article

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)94
  • Downloads (Last 6 weeks)7
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)The Rising Threat of Deepfake Technology and Frightening Advancements of Social EngineeringEffective Strategies for Combatting Social Engineering in Cybersecurity10.4018/979-8-3693-6665-3.ch014(307-330)Online publication date: 27-Sep-2024
  • (2024)3D Gaussian Blendshapes for Head Avatar AnimationACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657462(1-10)Online publication date: 13-Jul-2024
  • (2024)LIA: Latent Image AnimatorIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.344907546:12(10829-10844)Online publication date: 1-Dec-2024
  • (2024)Generating Multiple 4D Expression Transitions by Learning Face Landmark TrajectoriesIEEE Transactions on Affective Computing10.1109/TAFFC.2023.328067115:2(566-578)Online publication date: Apr-2024
  • (2024)MorpheuS: Neural Dynamic $360^{\circ}$ Surface Reconstruction from Monocular RGB-D Video2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01981(20965-20976)Online publication date: 16-Jun-2024
  • (2024)3D Facial Expressions through Analysis-by-Neural-Synthesis2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00241(2490-2501)Online publication date: 16-Jun-2024
  • (2024)MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00147(1481-1490)Online publication date: 16-Jun-2024
  • (2024)3D Face Reconstruction Based on a Single Image: A ReviewIEEE Access10.1109/ACCESS.2024.338197512(59450-59473)Online publication date: 2024
  • (2024)DynamicSurf: Dynamic Neural RGB-D Surface Reconstruction With an Optimizable Feature Grid2024 International Conference on 3D Vision (3DV)10.1109/3DV62453.2024.00046(820-830)Online publication date: 18-Mar-2024
  • (2024)Robust facial marker tracking based on a synthetic analysis of optical flows and the YOLO networkThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-023-02931-w40:4(2471-2489)Online publication date: 1-Apr-2024
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media