skip to main content
10.1145/3450623.3464681acmconferencesArticle/Chapter ViewAbstractPublication PagessiggraphConference Proceedingsconference-collections
invited-talk

Fast Facial Animation from Video

Published: 06 August 2021 Publication History

Abstract

Real time facial animation for virtual 3D characters has important applications such as AR/VR, interactive 3D entertainment, pre-visualization and video conferencing. Yet despite important research breakthroughs in facial tracking and performance capture, there are very few commercial examples of real-time facial animation applications in the consumer market. Mass adoption requires realtime performance on commodity hardware and visually pleasing animation that is robust to real world conditions, without requiring manual calibration. We present an end-to-end deep learning framework for regressing facial animation weights from video that addresses most of these challenges. Our formulation is fast (3.2 ms), utilizes images of real human faces along with millions of synthetic rendered frames to train the network on real-world scenarios, and produces jitter-free visually pleasing animations.

References

[1]
Apple. 2021. ARKit Developer Documentation. https://developer.apple.com/documentation/arkit/arfaceanchor
[2]
Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael Black. 2019. Capture, Learning, and Synthesis of 3D Speaking Styles. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 10101–10111. http://voca.is.tue.mpg.de/
[3]
Ivan Grishchenko, Artsiom Ablavatski, Yury Kartynnik, Karthik Raveendran, and Matthias Grundmann. 2020. Attention Mesh: High-fidelity Face Mesh Prediction in Real-time. arxiv:2006.10962 [cs.CV]
[4]
Xiaojie Guo, Siyuan Li, Jiawan Zhang, Jiayi Ma, Lin Ma, Wei Liu, and Haibin Ling. 2019. PFLD: A Practical Facial Landmark Detector. CoRR abs/1902.10859(2019). arxiv:1902.10859http://arxiv.org/abs/1902.10859
[5]
Sina Honari, Pavlo Molchanov, Stephen Tyree, Pascal Vincent, Christopher Pal, and Jan Kautz. 2018. Improving Landmark Localization with Semi-Supervised Learning. arxiv:1709.01591 [cs.CV]

Cited By

View all
  • (2023)A survey on the pipeline evolution of facial capture and tracking for digital humansMultimedia Systems10.1007/s00530-023-01081-229:4(1917-1940)Online publication date: 1-Apr-2023
  • (2023)A Digital Human System with Realistic Facial Expressions for Friendly Human-Machine InteractionAdvanced Intelligent Computing Technology and Applications10.1007/978-981-99-4755-3_68(787-798)Online publication date: 10-Aug-2023
  1. Fast Facial Animation from Video

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGGRAPH '21: ACM SIGGRAPH 2021 Talks
    July 2021
    116 pages
    ISBN:9781450383738
    DOI:10.1145/3450623
    Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 06 August 2021

    Check for updates

    Qualifiers

    • Invited-talk
    • Research
    • Refereed limited

    Conference

    SIGGRAPH '21
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)16
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)A survey on the pipeline evolution of facial capture and tracking for digital humansMultimedia Systems10.1007/s00530-023-01081-229:4(1917-1940)Online publication date: 1-Apr-2023
    • (2023)A Digital Human System with Realistic Facial Expressions for Friendly Human-Machine InteractionAdvanced Intelligent Computing Technology and Applications10.1007/978-981-99-4755-3_68(787-798)Online publication date: 10-Aug-2023

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media