Skip to main content
SpringerLink
Log in

Face Reenactment Based Facial Expression Recognition

  • Conference paper
  • First Online:
Advances in Visual Computing (ISVC 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12509))

Included in the following conference series:

Abstract

Representations used for Facial Expression Recognition (FER) are usually contaminated with identity specific features. In this paper, we propose a novel Reenactment-based Expression-Representation Learning Generative Adversarial Network (REL-GAN) that employs the concept of face reenactment to disentangle facial expression features from identity information. In this method, the facial expression representation is learned by reconstructing an expression image employing an encoder-decoder based generator. More specifically, our method learns the disentangled expression representation by transferring the expression information from the source image to the identity of the target image. Experiments performed on widely used datasets (BU-3DFE, CK+, Oulu-CASIA, SEFW) show that the proposed technique produces comparable or better results than state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Yang, H., Zhang, Z., Yin, L.: Identity-adaptive facial expression recognition through expression regeneration using conditional generative adversarial networks. In: International Conference on Automatic Face and Gesture Recognition, pp. 294–301 (2018)

    Google Scholar 

  2. Cai, J., Meng, Z., Khan, A.S., Li, Z., O’Reilly, J., Tong, Y.: Identity-free facial expression recognition using conditional generative adversarial network. CoRR abs/1903.08051 (2019). https://arxiv.org/abs/1903.08051

  3. Bai, M., Xie, W., Shen, L.: Disentangled feature based adversarial learning for facial expression recognition. In: IEEE International Conference on Image Processing, pp. 31–35 (2019)

    Google Scholar 

  4. Yang, H., Ciftci, U., Yin, L.: Facial expression recognition by de-expression residue learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2168–2177, April 2018

    Google Scholar 

  5. Tran, L., Yin, X., Liu, X.: Disentangled representation learning GAN for pose-invariant face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1415–1424 (2017)

    Google Scholar 

  6. Song, B.C., Lee, M.K., Choi, D.Y.: Facial expression recognition via relation-based conditional generative adversarial network. In: International Conference on Multimodal Interaction, pp. 35–39 (2019)

    Google Scholar 

  7. Ali, K., Isler, I., Hughes, C.E.: Facial expression recognition using human to animated-character expression translation. CoRR abs/1910.05595, https://arxiv.org/abs/1910.05595 (2019)

  8. Klaser, A., Marszałek, M., Schmid, C.: A spatio-temporal descriptor based on 3d-gradients. In: Proceedings British Machine Vision Conference (2008)

    Google Scholar 

  9. Chu, W.S., Torre, F., Cohn, J.F.: Selective transfer machine for personalized facial expression analysis. IEEE Trans. Pattern Anal. Mach. Intell. 39, 529–545 (2016)

    Article  Google Scholar 

  10. Jiang, B., Valstar, M.F., Pantic, M.: Action unit detection using sparse appearance descriptors in space-time video volumes. In: International Conference on Automatic Face and Gesture Recognition, pp. 314–321 (2011)

    Google Scholar 

  11. Kim, B.K., Lee, H., Roh, J., Lee, S.Y.: Hierarchical committee of deep CNNs with exponentially-weighted decision fusion for static facial expression recognition. In: Proceedings of the International Conference on Multimodal Interaction, pp. 427–434 (2015)

    Google Scholar 

  12. Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the International Conference on Multimodal Interaction, pp. 435–442 (2015)

    Google Scholar 

  13. Ng, H.W., Nguyen, V.W., Vonikakis, V., Winkler, S.: Deep learning for emotion recognition on small datasets using transfer learning. In: Proceedings of the International Conference on Multimodal Interaction, pp. 443–449 (2015)

    Google Scholar 

  14. Liu, M., Li, S., Shan, S., Wang, R., Chen, X.: Deeply learning deformable facial action parts model for dynamic expression analysis. In: Asian Conference on Computer Vision, pp. 143–157 (2014)

    Google Scholar 

  15. Li, S., Deng, W., Du, J.P.: Reliable crowd sourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2852–2861 (2017)

    Google Scholar 

  16. Cai, J., Meng, Z., Khan, A.S., Li, Z., O’Reilly, J., Tong, Y.: Island loss for learning discriminative features in facial expression recognition. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp. 302–309 (2018)

    Google Scholar 

  17. Meng, Z., Liu, P., Cai, J., Han, S., Tong Y.: Identity-aware convolutional neural network for facial expression recognition. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp. 558–565 (2017)

    Google Scholar 

  18. Lucey, P, Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101 (2010)

    Google Scholar 

  19. Zhao, G., Huang, X., Taini, M., Li, S.Z., Pietikainen, M.: Facial expression recognition from near-infrared videos. Image Vis. Comput. 29, 607–619 (2011)

    Article  Google Scholar 

  20. Ali, K., Hughes, C.E.: Facial expression recognition using disentangled adversarial learning. CoRR abs/1909.13135 (2019). https://arxiv.org/abs/1909.13135

  21. Zadeh, A., Lim, Y.C., Baltrusaitis, T., Morency, L.P.: Convolutional experts constrained local model for 3d facial landmark detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2519–2528 (2017)

    Google Scholar 

  22. Yin, L., Chen, X., Sun, Y., Worm, T., Reale, M.: A high-resolution 3d dynamic facial expression database. In: IEEE International Conference on Automatic Face and Gesture Recognition (2008)

    Google Scholar 

  23. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR abs/1511.06434 (2015). https://arxiv.org/abs/1511.06434

  24. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in neural information processing systems, pp. 2672–2680 (2014)

    Google Scholar 

  25. Liu, M., Shan, S., Wang, R., Chen, X.: Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1749–1756 (2014)

    Google Scholar 

  26. Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural net-works for facial expression recognition. In: Proceedings of the IEEE international conference on computer vision, pp. 2983–2991 (2015)

    Google Scholar 

  27. Zhao, X., et al.: Peak-piloted deep network for facial expression recognition. In: European Conference on Computer Vision, pp. 425–442 (2016)

    Google Scholar 

  28. Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.J.: A 3d facial expression database for facial behavior research. In: International Conference on Automatic Face and Gesture Recognition, pp. 211–216 (2006)

    Google Scholar 

  29. Wang, J., Yin, L., Wei, X., Sun, Y.: 3d facial expression recognition based on primitive surface feature distribution. In: IEEE Computer Vision and Pattern Recognition, pp. 1399–1406 (2006)

    Google Scholar 

  30. Berretti, S., Del Bimbo, A., Pala, P., Amor, B.B., Daoudi, M.: A set of selected sift features for 3d facial expression recognition. In: International Conference on Pattern Recognition, pp. 4125–4128 (2010)

    Google Scholar 

  31. Lopes, A.T., de Aguiar, E., De Souza, A.F., Oliveira-Santos, T.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn. 61, 610–628 (2017)

    Article  Google Scholar 

  32. Song, L., Lu, Z., He, R., Sun, Z., Tan, T.: Geometry guided adversarial facial expression synthesis. In: Proceedings of the ACM International Conference on Multimedia, pp. 627–635 (2018)

    Google Scholar 

  33. Dhall, A., Murthy, O.R., Goecke, R., Joshi, J., Gedeon, T.: Video and image based emotion recognition challenges in the wild: Emotiw. In: Proceedings of the ACM on International Conference on Multimodal Interaction, pp. 423–426 (2015)

    Google Scholar 

  34. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

  35. Hassner, T., Harel, S., Paz, E., Enbar, R.: Effective face frontalization in unconstrained images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4295–4304 (2015)

    Google Scholar 

  36. Liu, M., Li, S., Shan, S., Chen, X.: Au-aware deep networks for facial expression recognition. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1–6 (2013)

    Google Scholar 

  37. Levi, G., Hassner, T.: Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: Proceedings of the ACM on International Conference on Multimodal Interaction, pp. 503–510 (2015)

    Google Scholar 

  38. Ding, H, Zhou, S.K., Chellappa, R.: Facenet2expnet: regularizing a deep face recognition net for expression recognition. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp. 118–126 (2017)

    Google Scholar 

  39. Halawa, M., Wollhaf, M., Vellasques, E., Sanz, U. S., Hellwich, O.: Learning Disentangled Expression Representations from Facial Images. CoRR abs/2008.07001 (2020). https://arxiv.org/abs/2008.07001

  40. Alaghband, M., Yousefi, N., Garibay, I.: FePh: an annotated facial expression dataset for the RWTH-PHOENIX-Weather 2014 Dataset. CoRR abs/2003.08759 (2020). https://arxiv.org/abs/2003.08759

  41. Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: Proceedings British Machine Vision Conference (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Synthetic Reality Lab, CECS, University of Central Florida, Oviedo, USA

    Kamran Ali & Charles E. Hughes

Authors
  1. Kamran Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charles E. Hughes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kamran Ali .

Editor information

Editors and Affiliations

  1. University of Nevada Reno, Reno, NV, USA

    George Bebis

  2. Stony Brook University, Stony Brook, NY, USA

    Zhaozheng Yin

  3. Drexel University, Philadelphia, PA, USA

    Edward Kim

  4. RWTH Aachen University, Aachen, Germany

    Jan Bender

  5. University of Edinburgh, Edinburgh, UK

    Kartic Subr

  6. IBM Research – Cambridge, Cambridge, MA, USA

    Bum Chul Kwon

  7. University of Waterloo, Waterloo, ON, Canada

    Jian Zhao

  8. Graz University of Technology, Graz, Austria

    Denis Kalkofen

  9. The Hong Kong Polytechnic University, Hong Kong, Hong Kong

    George Baciu

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ali, K., Hughes, C.E. (2020). Face Reenactment Based Facial Expression Recognition. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2020. Lecture Notes in Computer Science(), vol 12509. Springer, Cham. https://doi.org/10.1007/978-3-030-64556-4_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-64556-4_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-64555-7

  • Online ISBN: 978-3-030-64556-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation