Skip to main content
SpringerLink
Log in

Action Unit Assisted Facial Expression Recognition

  • Conference paper
  • First Online:
Book cover Artificial Neural Networks and Machine Learning – ICANN 2019: Image Processing (ICANN 2019)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11729))

Included in the following conference series:

Abstract

Facial expression recognition is vital to many intelligent applications such as human-computer interaction and social networks. For machines, learning to classify six basic human expressions (anger, disgust, fear, happiness, sadness and surprise) is still a big challenge. This paper proposed a convolutional neural network based on AlexNet combining a Bayesian network. Besides traditional features, the relationships between facial action units (AU) and expressions are captured. Firstly, a convolutional neural network to extract features from images is constructed. Then, a Bayesian network is established to learn the dependencies of AUs and expressions from joint probabilities and conditional probabilities. Finally, ensemble learning is used to combine the features of expressions, AUs and dependencies between the two. Our experiments on popular datasets show that the proposed method performs well compared with latest approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Jain, A.K., Li, S.Z.: Handbook of Face Recognition. Springer, New York (2011). https://doi.org/10.1007/978-0-85729-932-1

    Book  MATH  Google Scholar 

  2. Sawyer, R., Smith, A., Rowe, J., et al.: Enhancing student models in game-based learning with facial expression recognition. In: Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization, pp. 192–201. ACM (2017)

    Google Scholar 

  3. Cuesta, U., Martnez-Martnez, L., Nino, J.I.: A case study in neuromarketing: analysis of the influence of music on advertising effectivenes through eye-tracking, facial emotion and GSR. Eur. J. Soc. Sci. Educ. Res. 5(2), 84–92 (2018)

    Article  Google Scholar 

  4. Rifai, S., Bengio, Y., Courville, A., Vincent, P., Mirza, M.: Disentangling factors of variation for facial expression recognition. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 808–822. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33783-3_58

    Chapter  Google Scholar 

  5. Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists, San Francisco (1978)

    Google Scholar 

  6. LeCun, Y., Bottou, L., Bengio, Y., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  7. Ciresan, D.C., Meier, U., Masci, J., et al.: Flexible, high performance convolutional neural networks for image classification. In: IJCAI Proceedings-International Joint Conference on Artificial Intelligence, vol. 22, no. 1, p. 1237 (2011)

    Google Scholar 

  8. Song, I., Kim, H.J., Jeon, P.B.: Deep learning for real-time robust facial expression recognition on a smartphone. In: 2014 IEEE International Conference on Consumer Electronics (ICCE), pp. 564–567. IEEE (2014)

    Google Scholar 

  9. Byeon, Y.H., Kwak, K.C.: Facial expression recognition using 3D convolutional neural network. Int. J. Adv. Comput. Sci. Appl. 5(12) (2014)

    Google Scholar 

  10. Izard, C.E.: The face of emotion (1971)

    Google Scholar 

  11. Ekman, P.: Universals and cultural differences in facial expressions of emotion. In: Nebraska Symposium on Motivation. University of Nebraska Press (1971)

    Google Scholar 

  12. Ekman, P., Oster, H.: Facial expressions of emotion. Ann. Rev. Psychol. 30(1), 527–554 (1979)

    Article  Google Scholar 

  13. Lopes, A.T., de Aguiar, E., De Souza, A.F., et al.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit. 61, 610–628 (2017)

    Article  Google Scholar 

  14. El Meguid, M.K.A., Levine, M.D.: Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers. IEEE Trans. Affect. Comput. 5(2), 141–154 (2014)

    Article  Google Scholar 

  15. Fan, X., Tjahjadi, T.: A spatial-temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences. Pattern Recognit. 48(11), 3407–3416 (2015)

    Article  Google Scholar 

  16. Liu, P., Han, S., Meng, Z., et al.: Facial expression recognition via a boosted deep belief network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1805–1812 (2014)

    Google Scholar 

  17. Ding, H., Zhou, S.K., Chellappa, R.: Facenet2expnet: regularizing a deep face recognition net for expression recognition. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), pp. 118–126. IEEE (2017)

    Google Scholar 

  18. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  19. Tian, Y.I., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 97–115 (2001)

    Article  Google Scholar 

  20. Tong, Y., Liao, W., Ji, Q.: Facial action unit recognition by exploiting their dynamic and semantic relationships. IEEE Trans. Pattern Anal. Mach. Intell. 29(10), 1683–1699 (2007)

    Article  Google Scholar 

  21. Liu, M., Li, S., Shan, S., et al.: Au-inspired deep networks for facial expression feature learning. Neurocomputing 159, 126–136 (2015)

    Article  Google Scholar 

  22. Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In: International Conference on Computer Vision, vol. 1, no. 2, p. 4 (2017)

    Google Scholar 

  23. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  24. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  25. Lucey, P., Cohn, J.F., Kanade, T., et al.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101. IEEE (2010)

    Google Scholar 

  26. Wang, S., Gan, Q., Ji, Q.: Expression-assisted facial action unit recognition under incomplete au annotation. Pattern Recognit. 61, 78–91 (2017)

    Article  Google Scholar 

  27. Hu, Q., Jiang, F., Mei, C., et al.: CCT: a cross-concat and temporal neural network for multi-label action unit detection. In: 2018 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2018)

    Google Scholar 

  28. Zhang, X., Yin, L., Cohn, J.F., et al.: A high-resolution spontaneous 3D dynamic facial expression database. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–6. IEEE (2013)

    Google Scholar 

  29. Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55(1), 119–139 (1997)

    Article  MathSciNet  Google Scholar 

  30. Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)

    MATH  Google Scholar 

  31. Breiman, L.: Stacked regressions. Mach. Learn. 24(1), 49–64 (1996)

    MATH  Google Scholar 

  32. Seewald, A.K.: Exploring the parameter state space of stacking. In: 2002 IEEE International Conference on Data Mining, Proceedings, pp. 685–688. IEEE (2002)

    Google Scholar 

  33. Sill, J., Takcs, G., Mackey, L., et al.: Feature-weighted linear stacking. arXiv preprint arXiv:0911.0460 (2009)

  34. Lyons, M.J., Akamatsu, S., Kamachi, M., et al.: The Japanese female facial expression (JAFFE) database. In: Proceedings of Third International Conference on Automatic Face and Gesture Recognition, pp. 14–16 (1998)

    Google Scholar 

  35. Li, S., Deng, W., Du, J.P.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2852–2861 (2017)

    Google Scholar 

  36. Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_16

    Chapter  Google Scholar 

  37. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  38. Gu, W., Xiang, C., Venkatesh, Y.V., et al.: Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recognit. 45(1), 80–91 (2012)

    Article  Google Scholar 

  39. Zhong, L., Liu, Q., Yang, P., et al.: Learning active facial patches for expression analysis. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2562–2569. IEEE (2012)

    Google Scholar 

  40. Devries, T., Biswaranjan, K., Taylor, G.W.: Multi-task learning of facial landmarks and expression. In: 2014 Canadian Conference on Computer and Robot Vision, pp. 98–103. IEEE (2014)

    Google Scholar 

  41. Ionescu, R.T., Popescu, M., Grozea, C.: Local learning to improve bag of visual words model for facial expression recognition. In: Workshop on Challenges in Representation Learning, ICML (2013)

    Google Scholar 

  42. Tang, Y.: Deep learning using linear support vector machines. arXiv preprint arXiv:1306.0239 (2013)

  43. Guo, Y., Tao, D., Yu, J., et al.: Deep neural networks with relativity learning for facial expression recognition. In: 2016 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 1–6. IEEE (2016)

    Google Scholar 

Download references

Acknowledgement

The work was supported by the Key Program for International S&T Cooperation Project of China (No. 2016YFE0129500).

Author information

Authors and Affiliations

  1. SEIEE, Shanghai Jiao Tong University, Shanghai, China

    Fangjun Wang & Liping Shen

Authors
  1. Fangjun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liping Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fangjun Wang .

Editor information

Editors and Affiliations

  1. Helmholtz Zentrum München - Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH), Neuherberg, Germany

    Igor V. Tetko

  2. Institute of Computer Science, Czech Academy of Sciences, Praha 8, Czech Republic

    Věra Kůrková

  3. Helmholtz Zentrum München - Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH), Neuherberg, Germany

    Pavel Karpov

  4. Helmholtz Zentrum München - Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH), Neuherberg, Germany

    Fabian Theis

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, F., Shen, L. (2019). Action Unit Assisted Facial Expression Recognition. In: Tetko, I., Kůrková, V., Karpov, P., Theis, F. (eds) Artificial Neural Networks and Machine Learning – ICANN 2019: Image Processing. ICANN 2019. Lecture Notes in Computer Science(), vol 11729. Springer, Cham. https://doi.org/10.1007/978-3-030-30508-6_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-30508-6_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-30507-9

  • Online ISBN: 978-3-030-30508-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation