Abstract
This paper presents a novel approach for generating 3D talking heads from raw audio inputs. Our method grounds on the idea that speech related movements can be comprehensively and efficiently described by the motion of a few control points located on the movable parts of the face, i.e., landmarks. The underlying musculoskeletal structure then allows us to learn how their motion influences the geometrical deformations of the whole face. The proposed method employs two distinct models to this aim: the first one learns to generate the motion of a sparse set of landmarks from the given audio. The second model expands such landmarks motion to a dense motion field, which is utilized to animate a given 3D mesh in neutral state. Additionally, we introduce a novel loss function, named Cosine Loss, which minimizes the angle between the generated motion vectors and the ground truth ones. Using landmarks in 3D talking head generation offers various advantages such as consistency, reliability, and obviating the need for manual-annotation. Our approach is designed to be identity-agnostic, enabling high-quality facial animations for any users without additional data or training. Code and models are available at: S2L+S2D.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baevski, A., Zhou, H., Mohamed, A., Auli, M.: wav2vec 2.0: a framework for self-supervised learning of speech representations. CoRR abs/2006.11477 (2020)
Bouritsas, G., Bokhnyak, S., Ploumpis, S., Bronstein, M.M., Zafeiriou, S.: Neural 3d morphable models: spiral convolutional networks for 3d shape representation learning and generation. CoRR abs/1905.02876 (2019)
Chung, J.S., Jamaludin, A., Zisserman, A.: You said that? CoRR abs/1705.02966 (2017)
Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., Black, M.J.: Capture, learning, and synthesis of 3d speaking styles. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
De Martino, J.M., Pini Magalhães, L., Violaro, F.: Facial animation based on context-dependent visemes. Comput. Graph. 30(6), 971–980 (2006). https://doi.org/10.1016/j.cag.2006.08.017, https://www.sciencedirect.com/science/article/pii/S0097849306001518
Edwards, P., Landreth, C., Fiume, E., Singh, K.: Jali: an animator-centric viseme model for expressive lip synchronization. ACM Trans. Graph. 35(4) (2016). https://doi.org/10.1145/2897824.2925984
Fan, Y., Lin, Z., Saito, J., Wang, W., Komura, T.: Faceformer: speech-driven 3d facial animation with transformers. CoRR abs/2112.05329 (2021)
Kalberer, G., Van Gool, L.: Face animation based on observed 3d speech dynamics. In: Proceedings Computer Animation 2001. Fourteenth Conference on Computer Animation (Cat. No.01TH8596), pp. 20–251 (2001). https://doi.org/10.1109/CA.2001.982373
Karras, T., Aila, T., Laine, S., Herva, A., Lehtinen, J.: Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Trans. Graph. 36(4) (2017). https://doi.org/10.1145/3072959.3073658
Otberdout, N., Ferrari, C., Daoudi, M., Berretti, S., Del Bimbo, A.: Sparse to dense dynamic 3d facial expression generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20385–20394 (2022)
Otberdout, N., Ferrari, C., Daoudi, M., Berretti, S., Del Bimbo, A.: Generating multiple 4d expression transitions by learning face landmark trajectories. IEEE Trans. Affect. Comput. (2023)
Richard, A., Zollhöfer, M., Wen, Y., la Torre, F.D., Sheikh, Y.: Meshtalk: 3d face animation from speech using cross-modality disentanglement. CoRR abs/2104.08223 (2021)
Suwajanakorn, S., Seitz, S.M., Kemelmacher-Shlizerman, I.: Synthesizing obama: learning lip sync from audio. ACM Trans. Graph. 36(4) (2017). https://doi.org/10.1145/3072959.3073640
Vaswani, A., et al.: Attention is all you need. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper1_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aaPaper.pdf
Vougioukas, K., Petridis, S., Pantic, M.: Realistic speech-driven facial animation with gans. CoRR abs/1906.06337 (2019)
Zhou, Y., Li, D., Han, X., Kalogerakis, E., Shechtman, E., Echevarria, J.: Makeittalk: speaker-aware talking head animation. CoRR abs/2004.12992 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Nocentini, F., Ferrari, C., Berretti, S. (2023). Learning Landmarks Motion from Speech for Speaker-Agnostic 3D Talking Heads Generation. In: Foresti, G.L., Fusiello, A., Hancock, E. (eds) Image Analysis and Processing – ICIAP 2023. ICIAP 2023. Lecture Notes in Computer Science, vol 14233. Springer, Cham. https://doi.org/10.1007/978-3-031-43148-7_29
Download citation
DOI: https://doi.org/10.1007/978-3-031-43148-7_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43147-0
Online ISBN: 978-3-031-43148-7
eBook Packages: Computer ScienceComputer Science (R0)