Skip to main content
SpringerLink
Log in

Improved 3D Morphable Model for Facial Action Unit Synthesis

  • Conference paper
  • First Online:
Image and Graphics (ICIG 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12890))

Included in the following conference series:

  • 2272 Accesses

Abstract

To overcome the limitation of the conventional 3D face model on the synthesis of local facial expression movements, this paper proposes an improved 3D face model that combines facial action coding system (FACS) and 3D morphable model (3DMM). Our proposed 3D face model can be used for 3D facial expression synthesis with local action units (AUs). To be specific, AUs are introduced as prior knowledge into the 3D face model to capture the anatomically defined muscle movements of different facial expressions. Our proposed model extracts the parameters of a single AU, which can also be used to represent AU information to facilitate AU recognition. The qualitative and quantitative experimental results demonstrate that our proposed model can generate 3D faces with specific AU labels. These AU parameters of our proposed model perform better in AU classification than the global expression parameters of conventional 3DMM.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amos, B., Ludwiczuk, B., Satyanarayanan, M.: Openface: a general-purpose face recognition library with mobile applications. Tech. rep., CMU-CS-16-118, CMU School of Computer Science (2016)

    Google Scholar 

  2. Basso, C., Vetter, T., Blanz, V.: Regularized 3D morphable models. In: First IEEE International Workshop on Higher-Level Knowledge in 3D Modeling and Motion Analysis, 2003. HLK 2003, pp. 3–10. IEEE (2003)

    Google Scholar 

  3. Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–194 (1999)

    Google Scholar 

  4. Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K.: FaceWarehouse: a 3D facial expression database for visual computing. IEEE Trans. Visual. Comput. Graph. 20(3), 413–425 (2013)

    Google Scholar 

  5. Chen, Y., Song, G., Shao, Z., Cai, J., Cham, T.J., Zheng, J.: GeoConv: geodesic guided convolution for facial action unit recognition. arXiv preprint arXiv:2003.03055 (2020)

  6. Cosker, D., Krumhuber, E., Hilton, A.: A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling. In: 2011 International Conference on Computer Vision, pp. 2296–2303. IEEE (2011)

    Google Scholar 

  7. Deng, Y., Yang, J., Chen, D., Wen, F., Tong, X.: Disentangled and controllable face image generation via 3D imitative-contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5154–5163 (2020)

    Google Scholar 

  8. Ekman, P., Friesen, W.V.: Manual for the Facial Action Coding System. Consulting Psychologists Press (1978)

    Google Scholar 

  9. Li, W., Abtahi, F., Zhu, Z., Yin, L.: EAC-Net: a region-based deep enhancing and cropping approach for facial action unit detection. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pp. 103–110. IEEE (2017)

    Google Scholar 

  10. Lin, J., Yuan, Y., Shao, T., Zhou, K.: Towards high-fidelity 3D face reconstruction from in-the-wild images using graph convolutional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5891–5900 (2020)

    Google Scholar 

  11. Liu, Z., Dong, J., Zhang, C., Wang, L., Dang, J.: Relation modeling with graph convolutional networks for facial action unit detection. In: Ro, Y.M., et al. (eds.) Relation modeling with graph convolutional networks for facial action unit detection. LNCS, vol. 11962, pp. 489–501. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-37734-2_40

    Chapter  Google Scholar 

  12. Liu, Z., Liu, D., Wu, Y.: Region based adversarial synthesis of facial action units. In: Ro, Y.M., et al. (eds.) MMM 2020. LNCS, vol. 11962, pp. 514–526. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-37734-2_42

    Chapter  Google Scholar 

  13. Liu, Z., Song, G., Cai, J., Cham, T.J., Zhang, J.: Conditional adversarial synthesis of 3D facial action units. Neurocomputing 355, 200–208 (2019)

    Article  Google Scholar 

  14. Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)

  15. Niinuma, K., Ertugrul, I.O., Cohn, J.F., Jeni, L.A.: Synthetic expressions are better than real for learning to detect facial actions. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1248–1257 (2021)

    Google Scholar 

  16. Pham, T.T.D., Kim, S., Lu, Y., Jung, S.W., Won, C.S.: Facial action units-based image retrieval for facial expression recognition. IEEE Access 7, 5200–5207 (2019)

    Article  Google Scholar 

  17. Roth, J., Tong, Y., Liu, X.: Adaptive 3D face reconstruction from unconstrained photo collections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4197–4206 (2016)

    Google Scholar 

  18. Shao, Z., Liu, Z., Cai, J., Ma, L.: Deep adaptive attention for joint facial action unit detection and face alignment. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11217, pp. 725–740. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01261-8_43

    Chapter  Google Scholar 

  19. Shi, T., Yuan, Y., Fan, C., Zou, Z., Shi, Z., Liu, Y.: Face-to-parameter translation for game character auto-creation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 161–170 (2019)

    Google Scholar 

  20. Song, X., et al.: Unsupervised learning facial parameter regressor for action unit intensity estimation via differentiable renderer. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2842–2851 (2020)

    Google Scholar 

  21. Suwajanakorn, S., Kemelmacher-Shlizerman, I., Seitz, S.M.: Total moving face reconstruction. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 796–812. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_52

    Chapter  Google Scholar 

  22. Tewari, A., et al.: MOFA: model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 1274–1283 (2017)

    Google Scholar 

  23. Tran, L., Liu, X.: Nonlinear 3D face morphable model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7346–7355 (2018)

    Google Scholar 

  24. Vemulapalli, R., Agarwala, A.: A compact embedding for facial expression similarity. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5683–5692 (2019)

    Google Scholar 

  25. Ververas, E., Zafeiriou, S.: SliderGAN: synthesizing expressive face images by sliding 3D blendshape parameters. Int. J. Comput. Vis. 128(10), 2629–2650 (2020)

    Article  Google Scholar 

  26. Walecki, R., Rudovic, O., Pavlovic, V., Schuller, B., Pantic, M.: Deep structured learning for facial action unit intensity estimation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  27. Wu, R., Lu, S.: LEED: label-free expression editing via disentanglement. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12357, pp. 781–798. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58610-2_46

    Chapter  Google Scholar 

  28. Zhang, X., et al.: BP4D-spontaneous: a high-resolution spontaneous 3D dynamic facial expression database. Image Vis. Comput. 32(10), 692–706 (2014)

    Article  Google Scholar 

  29. Zhang, Z., et al.: Multimodal spontaneous emotion corpus for human behavior analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3438–3446 (2016)

    Google Scholar 

  30. Zhu, W., Wu, H., Chen, Z., Vesdapunt, N., Wang, B.: ReDa: reinforced differentiable attribute for 3D face reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4958–4967 (2020)

    Google Scholar 

Download references

Acknowledgment

This work is supported by the National Natural Science Foundation of China (No. 61503277).

Author information

Authors and Affiliations

  1. College of Intelligence and Computing, Tianjin University, Tianjin, 300350, China

    Minghui Wang & Zhilei Liu

Authors
  1. Minghui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhilei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhilei Liu .

Editor information

Editors and Affiliations

  1. Peking University, Beijing, China

    Yuxin Peng

  2. Tsinghua University, Beijing, China

    Shi-Min Hu

  3. Tampere University, Tampere, Finland

    Moncef Gabbouj

  4. Zhejiang University, Hangzhou, China

    Kun Zhou

  5. Technion – Israel Institute of Technology, Haifa, Israel

    Michael Elad

  6. Tsinghua University, Beijing, China

    Kun Xu

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, M., Liu, Z. (2021). Improved 3D Morphable Model for Facial Action Unit Synthesis. In: Peng, Y., Hu, SM., Gabbouj, M., Zhou, K., Elad, M., Xu, K. (eds) Image and Graphics. ICIG 2021. Lecture Notes in Computer Science(), vol 12890. Springer, Cham. https://doi.org/10.1007/978-3-030-87361-5_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-87361-5_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87360-8

  • Online ISBN: 978-3-030-87361-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation