Skip to main content
SpringerLink
Log in

Expression Recognition Using a Flow-Based Latent-Space Representation

  • Conference paper
  • First Online:
Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges (ICPR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13646))

Included in the following conference series:

  • 173 Accesses

Abstract

Facial expression Recognition is a growing and important field that has applications in fields such as medicine, security, education, and entertainment. While there have been encouraging approaches that have shown accurate results on a wide variety of datasets, in many cases it is still a difficult problem to explain the results. To enable deployment of expression recognition applications in-the-wild, being able to explain why an particular expression is classified is an important task. Considering this, we propose to model flow-based latent representations of facial expressions, which allows us to further analyze the features and grants us more granular control over which features are produced for recognition. Our work is focused on posed facial expressions with a tractable density of the latent space. We investigate the behaviour of these tractable latent space features in the case of subject dependent and independent expression recognition. We employ a flow-based generative approach with minimal supervision introduced during training and observe that traditional metrics give encouraging results. When subject independent expressions are evaluated, a shift towards a stochastic nature, in the probability space, is observed. We evaluate our flow-based representation on the BU-EEG dataset showing our approach provides good separation of classes, resulting in more explainable results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/rosinality/glow-pytorch.

References

  1. Al-modwahi, A.A.M., et al.: Facial expression recognition intelligent security system for real time surveillance. In: World Congress in Computer Science, Computer Engineering, and Applied Computing (2012)

    Google Scholar 

  2. Atanov, A., et al.: Semi-conditional normalizing flows for semi-supervised learning (2020)

    Google Scholar 

  3. Barrett, L.F., et al.: Emotional expressions reconsidered: challenges to inferring emotion from human facial movements. Psychol. Sci. Publ. Interest 20(1), 1–68 (2019)

    Article  Google Scholar 

  4. Berthouze, N., et al.: Emopain challenge 2020: Multimodal pain evaluation from facial and bodily expressions. arXiv preprint arXiv:2001.07739 (2020)

  5. Bojanowski, P., Joulin, A., Lopez-Paz, D., Szlam, A.: Optimizing the latent space of generative networks (2019)

    Google Scholar 

  6. Cowie, R.: Ethical issues in affective computing. In: The Oxford handbook of AC, pp. 334–348. Oxford University Press (2015)

    Google Scholar 

  7. Deng, L.: The MNIST database of handwritten digit images for machine learning research. IEEE Signal Process. Mag. 29(6), 141–142 (2012)

    Article  Google Scholar 

  8. Dinh, L., Sohl-Dickstein, J., Bengio, S.: Density estimation using real NVP. CoRR abs/1605.08803 (2016). http://arxiv.org/abs/1605.08803

  9. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124 (1971)

    Article  Google Scholar 

  10. Ertugrul, I.O., et al.: Cross-domain au detection: Domains, learning approaches, and measures. In: FG, pp. 1–8. IEEE (2019)

    Google Scholar 

  11. Escalante, H.J., et al.: Design of an explainable machine learning challenge for video interviews. In: IJCNN (2017)

    Google Scholar 

  12. Fabiano, D., Canavan, S.: Emotion recognition using fused physiological signals. In: ACII, pp. 42–48. IEEE (2019)

    Google Scholar 

  13. Goebel, R., et al.: Explainable AI: the new 42? In: Holzinger, A., Kieseberg, P., Tjoa, A.M., Weippl, E. (eds.) CD-MAKE 2018. LNCS, vol. 11015, pp. 295–303. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99740-7_21

    Chapter  Google Scholar 

  14. Goodfellow, I.J., et al.: Generative adversarial networks (2014)

    Google Scholar 

  15. Habler, E., Shabtai, A.: Using LSTM encoder-decoder algorithm for detecting anomalous ADS-B messages. Comput. Secur. 78, 155–173 (2018)

    Article  Google Scholar 

  16. Hasani, B., Mahoor, M.H.: Facial expression recognition using enhanced deep 3d convolutional neural networks. In: CVPRW (2017)

    Google Scholar 

  17. Hinduja, S., Canavan, S., Yin, L.: Recognizing perceived emotions from facial expressions. In: FG (2020)

    Google Scholar 

  18. Hu, X., et al.: Ten challenges for EEG-based affective computing. Brain Science Advances 5(1), 1–20 (2019). https://doi.org/10.1177/2096595819896200

  19. Izmailov, P., et al.: Semi-supervised learning with normalizing flows (2019)

    Google Scholar 

  20. Jack, R.E., Garrod, O.G., Yu, H., Caldara, R., Schyns, P.G.: Facial expressions of emotion are not culturally universal. Proc. Natl. Acad. Sci. 109(19), 7241–7244 (2012)

    Article  Google Scholar 

  21. Kandeel, A.A., et al.: Explainable model selection of a CNN for driver’s facial emotion identification. In: ICPRW (2021)

    Google Scholar 

  22. Khalfallah, J., Slama, J.B.H.: Facial expression recognition for intelligent tutoring systems in remote laboratories platform. Proc. Comput. Sci. 73, 274–281 (2015)

    Article  Google Scholar 

  23. Kingma, D.P., Dhariwal, P.: Glow: Generative flow with invertible 1 \(\times \) 1 convolutions (2018)

    Google Scholar 

  24. Kingma, D.P., Welling, M.: Auto-encoding variational bayes (2014)

    Google Scholar 

  25. Kobyzev, I., Prince, S., Brubaker, M.: Normalizing flows: an introduction and review of current methods. IEEE Trans. Pattern Anal. Mach. Intell. 1 (2020). https://doi.org/10.1109/TPAMI.2020.2992934

  26. Li, S., Deng, W.: A deeper look at facial expression dataset bias. IEEE Trans. Affect. Comput. (2020)

    Google Scholar 

  27. Li, X., et al.: An EEG-based multi-modal emotion database with both posed and authentic facial actions for emotion analysis. In: FG (2020)

    Google Scholar 

  28. Liu, M., Li, S., Shan, S., Wang, R., Chen, X.: Deeply learning deformable facial action parts model for dynamic expression analysis. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 143–157. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16817-3_10

    Chapter  Google Scholar 

  29. Lucas, J., Tucker, G., Grosse, R., Norouzi, M.: Understanding posterior collapse in generative latent variable models (2019)

    Google Scholar 

  30. Lucey, P., Cohn, J., Lucey, S., Matthews, I., Sridharan, S., Prkachin, K.M.: Automatically detecting pain using facial actions. In: ACIIW, pp. 1–8 (2009). https://doi.org/10.1109/ACII.2009.5349321

  31. Van der Maaten, L., Hinton, G.: Visualizing data using T-SNE. J. Mach. Learn. Res. 9(11) (2008)

    Google Scholar 

  32. McGarigal, K., Stafford, S., Cushman, S.: Discriminant Analysis, pp. 129–187 (2000)

    Google Scholar 

  33. Melhart, D., Liapis, A., Yannakakis, G.N.: The affect game annotation (again) dataset. arXiv preprint arXiv:2104.02643 (2021)

  34. Minaee, S., Abdolrashidi, A.: Deep-emotion: Facial expression recognition using attentional convolutional network. arXiv preprint arXiv:1902.01019 (2019)

  35. Muhammad, G., Alsulaiman, M., Amin, S.U., Ghoneim, A., Alhamid, M.F.: A facial-expression monitoring system for improved healthcare in smart cities. IEEE Access 5, 10871–10881 (2017). https://doi.org/10.1109/ACCESS.2017.2712788

    Article  Google Scholar 

  36. Nguyen, A., et al.: Plug & play generative networks: conditional iterative generation of images in latent space. In: CVPR (2017)

    Google Scholar 

  37. Nummenmaa, L., Hari, R., Hietanen, J.K., Glerean, E.: Maps of subjective feelings. Proc. Natl. Acad. Sci. 115(37), 9198–9203 (2018). https://doi.org/10.1073/pnas.1807390115, https://www.pnas.org/content/115/37/9198

  38. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32, pp. 8024–8035. Curran Associates, Inc. (2019)

    Google Scholar 

  39. Perov, I., et al.: Deepfacelab: a simple, flexible and extensible face swapping framework (2020)

    Google Scholar 

  40. Reynolds, D.: Gaussian Mixture Models, pp. 659–663. Springer, US (2009). https://doi.org/10.1007/978-0-387-73003-5_196

  41. Rezende, D., Mohamed, S.: Variational inference with normalizing flows. In: ICML (2015)

    Google Scholar 

  42. Rothkrantz, L., et al.: Facs-coding of facial expressions. Association for Computing Machinery (2009)

    Google Scholar 

  43. Rudovic, O., et al.: Personalized federated deep learning for pain estimation from face images. arXiv preprint arXiv:2101.04800 (2021)

  44. Shao, J., Qian, Y.: Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing. 355, 82–92 (2019). https://doi.org/10.1016/j.neucom.2019.05.005, https://www.sciencedirect.com/science/article/pii/S0925231219306137

  45. Song, Y., Morency, L.P., Davis, R.: Distribution-sensitive learning for imbalanced datasets. In: FGW (2013)

    Google Scholar 

  46. Sricharan, K., et al.: Semi-supervised conditional gans. arXiv preprint arXiv:1708.05789 (2017)

  47. Sun, B., Li, L., Zhou, G., He, J.: Facial expression recognition in the wild based on multimodal texture features. J. Electron. Imaging 25(6), 1–8 (2016)

    Article  Google Scholar 

  48. Takalkar, M.A., Xu, M.: Image based facial micro-expression recognition using deep learning on small datasets. In: 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–7. IEEE (2017)

    Google Scholar 

  49. Vinciarelli, A., Pantic, M., Bourlard, H.: Social signal processing: survey of an emerging domain. Image Vision Comput. 27(12), 1743–1759 (2009)

    Google Scholar 

  50. Weitz, K., et al.: Deep-learned faces of pain and emotions: elucidating the differences of facial expressions with the help of explainable AI methods. tm-Technisches Messen. 86(7–8), 404–412 (2019)

    Google Scholar 

  51. Widen, S.C., et al.: Anger and disgust: discrete or overlapping categories. In: APS Annual Convention (2004)

    Google Scholar 

  52. Xie, S., Hu, H., Chen, Y.: Facial expression recognition with two-branch disentangled generative adversarial network. IEEE Trans. Circuits Syst. Video Technol. (2020)

    Google Scholar 

  53. Yang, H., et al.: Identity-adaptive facial expression recognition through expression regeneration using conditional generative adversarial networks. In: FG (2018)

    Google Scholar 

Download references

Acknowledgement

This material is based upon the work supported in part by the National Science Foundation under grant CNS-2039373. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Author information

Authors and Affiliations

  1. University of South Florida, Tampa, FL, 33620, USA

    Saandeep Aathreya & Shaun Canavan

Authors
  1. Saandeep Aathreya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shaun Canavan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shaun Canavan .

Editor information

Editors and Affiliations

  1. York University, Toronto, ON, Canada

    Jean-Jacques Rousseau

  2. Ontario Tech University, Oshawa, ON, Canada

    Bill Kapralos

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Aathreya, S., Canavan, S. (2023). Expression Recognition Using a Flow-Based Latent-Space Representation. In: Rousseau, JJ., Kapralos, B. (eds) Pattern Recognition, Computer Vision, and Image Processing. ICPR 2022 International Workshops and Challenges. ICPR 2022. Lecture Notes in Computer Science, vol 13646. Springer, Cham. https://doi.org/10.1007/978-3-031-37745-7_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-37745-7_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-37744-0

  • Online ISBN: 978-3-031-37745-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation