Abstract
This research paper proposes a real-time and realistic full facial animation method for virtual reality (VR) applications. Currently, VR applications lack natural upper-face animation, which limits the immersive experiences of self-avatars. Our proposed approach combines existing lip-sync methods for the lower part of the face with a deep-learning method for the upper part. This allows us to achieve natural full-face animation with minimal latency and high computational efficiency. We demonstrate the effectiveness of our approach through experimental results and show that it is suitable for use in VR applications. Our proposed method can help to enhance the realism and immersion of self-avatars in the metaverse.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., Black, M.J.: Capture, learning, and synthesis of 3D speaking styles. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10101–10111. IEEE (2019)
Karras, T., Aila, T., Laine, S., Herva, A., Lehtinen, J.: Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Trans. Graph. 36(4), 1–12 (2017). https://doi.org/10.1145/3072959.3073658
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization, January 2017. https://doi.org/10.48550/arXiv.1412.6980
Richard, A., Zollhöfer, M., Wen, Y., de la Torre, F., Sheikh, Y.: MeshTalk: 3D face animation from speech using cross-modality disentanglement. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1173–1182 (2021)
Taylor, S., et al.: A deep learning approach for generalized speech animation. ACM Trans. Graph. 36(4), 1–11 (2017). https://doi.org/10.1145/3072959.3073699
Vaswani, A., et al.: Attention is all you need, December 2017
Zhou, Y., Xu, Z., Landreth, C., Kalogerakis, E., Maji, S., Singh, K.: VisemeNet: audio-driven animator-centric speech animation. ACM Trans. Graph. 37(4), 161:1–161:10 (2018). https://doi.org/10.1145/3197517.3201292
Acknowledgement
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MIST) (No. 2021R1A2C1014210) and by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2023-RS-2022-00156353) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hwang, J., Park, K. (2023). Full-Face Animation for a Virtual Reality Avatar. In: Stephanidis, C., Antona, M., Ntoa, S., Salvendy, G. (eds) HCI International 2023 Posters. HCII 2023. Communications in Computer and Information Science, vol 1836. Springer, Cham. https://doi.org/10.1007/978-3-031-36004-6_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-36004-6_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36003-9
Online ISBN: 978-3-031-36004-6
eBook Packages: Computer ScienceComputer Science (R0)