Skip to main content
SpringerLink
Log in

Shallow Diffusion Motion Model for Talking Face Generation from Speech

  • Conference paper
  • First Online:
Web and Big Data (APWeb-WAIM 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13422))

Abstract

Talking face generation is synthesizing a lip synchronized talking face video by inputting an arbitrary face image and audio clips. People naturally conduct spontaneous head motions to enhance their speeches while giving talks. Head motion generation from the speech is inherently difficult due to the nondeterministic mapping from speech to head motions. Most existing works map speech to motion in a deterministic way by conditioning certain styles, leading to sub-optimal results. In this paper, we decompose the speech motion into two complementary parts: pose modes and rhythmic dynamics. Accordingly, we introduce a shallow diffusion motion model (SDM) by equipping a two-stream architecture, i.e., a pose mode branch for primary posture generation, and a rhythmic motion branch for rhythmic dynamics synthesis. On one hand, diverse pose modes are generated by conditional sampling in a latent space, guided by speech semantics. On the other hand, rhythmic dynamics are synced with the speech prosody. Extensive experiments demonstrate the superior performance against several baselines, in terms of fidelity, similarity, and syncing with speech.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Afouras, T., Chung, J.S., Zisserman, A.: Lrs3-ted: a large-scale dataset for visual speech recognition. arXiv preprint arXiv:1809.00496 (2018)

  2. Brand, M.: Voice puppetry. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 21–28 (1999)

    Google Scholar 

  3. Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1021–1030 (2017)

    Google Scholar 

  4. Chen, L., Li, Z., Maddox, R.K., Duan, Z., Xu, C.: Lip movements generation at a glance. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 520–535 (2018)

    Google Scholar 

  5. Chen, L., Maddox, R.K., Duan, Z., Xu, C.: Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7832–7841 (2019)

    Google Scholar 

  6. Chung, J.S., Nagrani, A., Zisserman, A.: Voxceleb2: deep speaker recognition. arXiv preprint arXiv:1806.05622 (2018)

  7. Chung, J.S., Zisserman, A.: Lip reading in the wild. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10112, pp. 87–103. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54184-6_6

    Chapter  Google Scholar 

  8. Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., Black, M.J.: Capture, learning, and synthesis of 3d speaking styles. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10101–10111 (2019)

    Google Scholar 

  9. Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4690–4699 (2019)

    Google Scholar 

  10. Eskimez, S.E., Maddox, R.K., Xu, C., Duan, Z.: Generating talking face landmarks from speech. In: Deville, Y., Gannot, S., Mason, R., Plumbley, M.D., Ward, D. (eds.) LVA/ICA 2018. LNCS, vol. 10891, pp. 372–381. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93764-9_35

    Chapter  Google Scholar 

  11. Eskimez, S.E., Zhang, Y., Duan, Z.: Speech driven talking face generation from a single image and an emotion condition. IEEE Trans. Multimedia 24, 3480–3490 (2021)

    Article  Google Scholar 

  12. Fišer, J., et al.: Example-based synthesis of stylized facial animations. ACM Trans. Graph. (TOG) 36(4), 1–11 (2017)

    Article  Google Scholar 

  13. Greenwood, D., Matthews, I., Laycock, S.: Joint learning of facial expression and head pose from speech. In: Interspeech (2018)

    Google Scholar 

  14. Guo, Y., Chen, K., Liang, S., Liu, Y.J., Bao, H., Zhang, J.: Ad-nerf: audio driven neural radiance fields for talking head synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5784–5794 (2021)

    Google Scholar 

  15. Gupta, A., Khan, F.F., Mukhopadhyay, R., Namboodiri, V.P., Jawahar, C.: Intelligent video editing: incorporating modern talking face generation algorithms in a video editor. In: Proceedings of the Twelfth Indian Conference on Computer Vision, Graphics and Image Processing, pp. 1–9 (2021)

    Google Scholar 

  16. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Adv. Neural Inf. Processi. Syst. 30, 1–12 (2017)

    Google Scholar 

  17. Jamaludin, A., Chung, J.S., Zisserman, A.: You said that?: synthesising talking faces from audio. Int. J. Comput. Vision 127(11), 1767–1779 (2019)

    Article  Google Scholar 

  18. Ji, X., Zhou, H., Wang, K., Wu, W., Loy, C.C., Cao, X., Xu, F.: Audio-driven emotional video portraits. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14080–14089 (2021)

    Google Scholar 

  19. Kim, H., et al.: Neural style-preserving visual dubbing. ACM Trans. Graph. (TOG) 38(6), 1–13 (2019)

    Google Scholar 

  20. Kingma, D., Salimans, T., Poole, B., Ho, J.: Variational diffusion models. Adv. Neural Inf. Process. Syst. 34, 21696–21707 (2021)

    Google Scholar 

  21. Kong, Z., Ping, W., Huang, J., Zhao, K., Catanzaro, B.: Diffwave: a versatile diffusion model for audio synthesis. In: 9th International Conference on Learning Representations, ICLR 2021 (2021)

    Google Scholar 

  22. Lam, M.W., Wang, J., Su, D., Yu, D.: Bddm: bilateral denoising diffusion models for fast and high-quality speech synthesis. In: International Conference on Learning Representations (2021)

    Google Scholar 

  23. Meshry, M., Suri, S., Davis, L.S., Shrivastava, A.: Learned spatial representations for few-shot talking-head synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13829–13838 (2021)

    Google Scholar 

  24. Qu, X., Wang, J., Xiao, J.: Enhancing data-free adversarial distillation with activation regularization and virtual interpolation. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 3340–3344. IEEE (2021)

    Google Scholar 

  25. Si, S., Wang, J., Peng, J., Xiao, J.: Towards speaker age estimation with label distribution learning. In: ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4618–4622 (2022). https://doi.org/10.1109/ICASSP43922.2022.9746378

  26. Song, Y., Zhu, J., Li, D., Wang, A., Qi, H.: Talking face generation by conditional recurrent adversarial network. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, pp. 919–925 (2019)

    Google Scholar 

  27. Sun, A., et al.: Reconstructing dual learning for neural voice conversion using relatively few samples. In: IEEE Automatic Speech Recognition and Understanding Workshop, pp. 946–953. IEEE (2021)

    Google Scholar 

  28. Suwajanakorn, S., Seitz, S.M., Kemelmacher-Shlizerman, I.: Synthesizing obama: learning lip sync from audio. ACM Trans. Graph. (ToG) 36(4), 1–13 (2017)

    Article  Google Scholar 

  29. Tang, H., Zhang, X., Wang, J., Cheng, N., Xiao, J.: Avqvc: one-shot voice conversion by vector quantization with applying contrastive learning. In: 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP2022), pp. 1–5. IEEE (2022)

    Google Scholar 

  30. Tang, H., et al.: TGAVC: Improving autoencoder voice conversion with text-guided and adversarial training. In: IEEE Automatic Speech Recognition and Understanding Workshop (ASRU2021), pp. 938–945. IEEE (2021)

    Google Scholar 

  31. Tang, J., Wu, Y., Li, M., Wang, Z.: Talking face generation based on information bottleneck and complementary representations. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 3443–3447 (2021)

    Google Scholar 

  32. Thies, J., Elgharib, M., Tewari, A., Theobalt, C., Nießner, M.: Neural voice puppetry: audio-driven facial reenactment. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12361, pp. 716–731. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58517-4_42

    Chapter  Google Scholar 

  33. Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2face: real-time face capture and reenactment of rgb videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387–2395 (2016)

    Google Scholar 

  34. Vougioukas, K., Petridis, S., Pantic, M.: Realistic speech-driven facial animation with gans. Int. J. Comput. Vision 128(5), 1398–1413 (2020)

    Article  Google Scholar 

  35. Wang, Q., Zhang, X., Wang, J., Cheng, N., Xiao, J.: Drvc: a framework of any-to-any voice conversion with self-supervised learning. In: 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP2022), pp. 3184–3188. IEEE (2022)

    Google Scholar 

  36. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  37. Wiles, O., Koepke, A., Zisserman, A.: X2face: a network for controlling face generation using images, audio, and pose codes. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 670–686 (2018)

    Google Scholar 

  38. Xu, L., Zhou, X.: A crowd-powered task generation method for study of struggling search. Data Sci. Eng. 6(4), 472–484 (2021)

    Article  Google Scholar 

  39. Yao, X., Fried, O., Fatahalian, K., Agrawala, M.: Iterative text-based editing of talking-heads using neural retargeting. ACM Trans. Graph. (TOG) 40(3), 1–14 (2021)

    Article  Google Scholar 

  40. Zakharov, E., Shysheya, A., Burkov, E., Lempitsky, V.: Few-shot adversarial learning of realistic neural talking head models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9459–9468 (2019)

    Google Scholar 

  41. Zhang, C., Ni, S., Fan, Z., Li, H., Zeng, M., Budagavi, M., Guo, X.: 3d talking face with personalized pose dynamics. IEEE Trans. Visualization Comput. Graph. 29, 1438–1449 (2021)

    Article  Google Scholar 

  42. Zhang, X., Wang, J., Cheng, N., Xiao, E., Xiao, J.: CycleGEAN: cycle generative enhanced adversarial network for voice conversion. In: IEEE Automatic Speech Recognition and Understanding Workshop (ASRU2021), pp. 1–6. IEEE (2021)

    Google Scholar 

  43. Zhang, X., Wang, J., Cheng, N., Xiao, J.: Susing: su-net for singing voice synthesis. In: International Joint Conference on Neural Networks, IJCNN 2022. IEEE (2022)

    Google Scholar 

  44. Zhang, X., Wang, J., Cheng, N., Xiao, J.: Tdass: target domain adaptation speech synthesis framework for multi-speaker low-resource tts. In: International Joint Conference on Neural Networks, IJCNN 2022. IEEE (2022)

    Google Scholar 

  45. Zhou, H., Liu, Y., Liu, Z., Luo, P., Wang, X.: Talking face generation by adversarially disentangled audio-visual representation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 9299–9306 (2019)

    Google Scholar 

  46. Zhou, H., Sun, Y., Wu, W., Loy, C.C., Wang, X., Liu, Z.: Pose-controllable talking face generation by implicitly modularized audio-visual representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4176–4186 (2021)

    Google Scholar 

  47. Zhou, Y., Xu, Z., Landreth, C., Kalogerakis, E., Maji, S., Singh, K.: Visemenet: audio-driven animator-centric speech animation. ACM Trans. Graph. (TOG) 37(4), 1–10 (2018)

    Article  Google Scholar 

Download references

Acknowledgement

This paper is supported by the Key Research and Development Program of Guangdong Province under grant No.2021B0101400003. Corresponding author is Jianzong Wang from Ping An Technology (Shenzhen) Co., Ltd (jzwang@188.com).

Author information

Authors and Affiliations

  1. Ping An Technology (Shenzhen) Co., Ltd., Shenzhen, China

    Xulong Zhang, Jianzong Wang, Ning Cheng & Jing Xiao

  2. Aquinas International Academy, La Palma, USA

    Edward Xiao

Authors
  1. Xulong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianzong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ning Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Edward Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jing Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianzong Wang .

Editor information

Editors and Affiliations

  1. Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Bohan Li

  2. Newcastle University, Callaghan, NSW, Australia

    Lin Yue

  3. Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Chuanqi Tao

  4. Jinan University, Guangzhou, China

    Xuming Han

  5. Free University of Bozen-Bolzano, Bolzano, Italy

    Diego Calvanese

  6. University of Tsukuba, Tsukuba, Japan

    Toshiyuki Amagasa

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, X., Wang, J., Cheng, N., Xiao, E., Xiao, J. (2023). Shallow Diffusion Motion Model for Talking Face Generation from Speech. In: Li, B., Yue, L., Tao, C., Han, X., Calvanese, D., Amagasa, T. (eds) Web and Big Data. APWeb-WAIM 2022. Lecture Notes in Computer Science, vol 13422. Springer, Cham. https://doi.org/10.1007/978-3-031-25198-6_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25198-6_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25197-9

  • Online ISBN: 978-3-031-25198-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation