abstract

LIPSYNC.AI: A.I. Driven Lips and Tongue Animations Using Articulatory Phonetic Descriptors and FACS Blendshapes

Authors:

Jara Alvarez Masso,

Alexandru Mihai Rogozea,

Jan Medvesek,

Saeid Mokaram,

Yijun YuAuthors Info & Claims

SA '21: SIGGRAPH Asia 2021 Emerging Technologies

Article No.: 9, Pages 1 - 2

https://doi.org/10.1145/3476122.3484850

Published: 14 December 2021 Publication History

Get Access

Abstract

We present a solution for generating realistic lips and tongue animations, using a novel hybrid method which makes use of both the advancements in deep learning and the theory behind speech and phonetics. Our solution generates highly accurate and natural animations of the jaw, lips and tongue through the use of additional phonetic information during the neural network training, and the procedural mapping of its outputs directly to FACS [Prince et al. 2015] based blendshapes, in order to comply to animation industry standards.

Supplementary Material

MP4 File (3476122.3484850.mp4)

presentation

Download
28.05 MB

References

[1]

C. Charalambous, Zerrin Yumak, and A. Stappen. 2019. Audio‐driven emotional speech animation for interactive virtual characters. Computer Animation and Virtual Worlds 30 (2019).

Google Scholar

[2]

Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, A. Ranjan, and Michael J. Black. 2019. Capture, Learning, and Synthesis of 3D Speaking Styles. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), 10093–10103.

Crossref

Google Scholar

[3]

P. Edwards, Chris Landreth, M. Poplawski, R. Malinowski, Sarah Watling, E. Fiume, and Karan Singh. 2020. JALI-Driven Expressive Facial Animation and Multilingual Speech in Cyberpunk 2077. Special Interest Group on Computer Graphics and Interactive Techniques Conference Talks (2020).

Digital Library

Google Scholar

[4]

Tero Karras, Timo Aila, S. Laine, Antti Herva, and J. Lehtinen. 2017. Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Transactions on Graphics (TOG) 36 (2017), 1 – 12.

Digital Library

Google Scholar

[5]

E. Prince, Katherine B. Martin, and D. Messinger. 2015. Facial Action Coding System.

Google Scholar

[6]

Sarah L. Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, J. Hodgins, and I. Matthews. 2017. A deep learning approach for generalized speech animation. ACM Transactions on Graphics (TOG) 36 (2017), 1 – 11.

Digital Library

Google Scholar

[7]

A. Thangthai, B. Milner, and Sarah L. Taylor. 2019. Synthesising visual speech using dynamic visemes and deep learning architectures. Comput. Speech Lang. 55(2019), 101–119.

Crossref

Google Scholar

[8]

Yang Zhou, Shan Xu, Chris Landreth, E. Kalogerakis, Subhransu Maji, and Karan Singh. 2018. VisemeNet: Audio-Driven Animator-Centric Speech Animation. ACM Trans. Graph. 37(2018), 161:1–161:10.

Digital Library

Google Scholar

Recommendations

Development of a silent speech interface driven by ultrasound and optical images of the tongue and lips

This article presents a segmental vocoder driven by ultrasound and optical images (standard CCD camera) of the tongue and lips for a ''silent speech interface'' application, usable either by a laryngectomized patient or for silent communication. The ...
A coupled HMM approach to video-realistic speech animation

We propose a coupled hidden Markov model (CHMM) approach to video-realistic speech animation, which realizes realistic facial animations driven by speaker independent continuous speech. Different from hidden Markov model (HMM)-based animation approaches ...
Phoneme reduction in automated speech for computer animation
SCCG '04: Proceedings of the 20th Spring Conference on Computer Graphics

Accurate facial animation is rapidly becoming a key feature in computer animation, from movie production to software agents. A lot of practical work in computer animation presently is done by 3ds max software. An advantage of this system is that one of ...

Comments

Information & Contributors

Information

Published In

SA '21: SIGGRAPH Asia 2021 Emerging Technologies

December 2021

39 pages

ISBN:9781450386852

DOI:10.1145/3476122

Editors:
Shuzo John Shiota
CEO, Polygon Pictures Inc., Japan
,
Ayumi Kimura
Research Manager at OLM Digital, Inc., Japan and Program Manager and Collaboration Coordinator at CMIC, Victoria University of Wellington, New Zealand
,
Kouta Minamizawa
Professor, Keio University Graduate School of Media Design, Japan

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 December 2021

Check for updates

Author Tags

Qualifiers

Abstract
Research
Refereed limited

Conference

SA '21

Sponsor:

SIGGRAPH

SA '21: SIGGRAPH Asia 2021

December 14 - 17, 2021

Tokyo, Japan

Acceptance Rates

Overall Acceptance Rate 178 of 869 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
227
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)1

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

Supplementary Material

References

Recommendations

Development of a silent speech interface driven by ultrasound and optical images of the tongue and lips

A coupled HMM approach to video-realistic speech animation

Phoneme reduction in automated speech for computer animation

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Login options

Full Access

View options

PDF

eReader

HTML Format

Share

Share this Publication link

Share on social media

Affiliations