Skip to main content
SpringerLink
Log in

Personalized Frame-Level Facial Expression Recognition in Video

  • Conference paper
  • First Online:
Pattern Recognition and Artificial Intelligence (ICPRAI 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13363))

Abstract

In this paper, the personalization of the video-based frame-level facial expression recognition is studied for multi-user systems if a small amount of short videos are available for each user. At first, embeddings of each video frame are computed using deep convolutional neural network pre-trained on a large emotional dataset of static images. Next, a dataset of videos is used to train a subject-independent emotion classifier, such as feed-forward neural network or frame attention network. Finally, it is proposed to fine-tune this neural classifier on the videos of each user of interest. As a result, every user is associated with his or her own emotional model. The classifier in a multi-user system is chosen by an appropriate video-based face recognition method. The experimental study with the RAMAS dataset demonstrates the significant (up to 25%) increase in accuracy of the proposed approach when compared to a subject-independent facial expression recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/HSE-asavchenko/face-emotion-recognition/blob/main/src/train_ramas.ipynb.

References

  1. Pietikäinen, M., Silven, O.: Challenges of artificial intelligence-from machine learning and computer vision to emotional intelligence. arXiv preprint arXiv:2201.01466 (2022)

  2. Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. (2020). https://doi.org/10.1109/TAFFC.2020.2981446

    Article  Google Scholar 

  3. Savchenko, A.V.: Facial expression and attributes recognition based on multi-task learning of lightweight neural networks. In: Proceedings of 19th International Symposium on Intelligent Systems and Informatics (SISY), pp. 119–124. IEEE (2021)

    Google Scholar 

  4. Cerezo, E., et al.: Real-time facial expression recognition for natural interaction. In: Martí, J., Benedí, J.M., Mendonça, A.M., Serrat, J. (eds.) IbPRIA 2007. LNCS, vol. 4478, pp. 40–47. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-72849-8_6

  5. Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.: Vggface2: a dataset for recognising faces across pose and age. In: Proceedings of 13th International Conference on Automatic Face & Gesture Recognition (FG), pp. 67–74. IEEE (2018)

    Google Scholar 

  6. Mollahosseini, A., Hasani, B., Mahoor, M.H.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2017)

    Article  Google Scholar 

  7. Perepelkina, O., Kazimirova, E., Konstantinova, M.: RAMAS: Russian multimodal corpus of dyadic interaction for affective computing. In: Karpov, A., Jokisch, O., Potapova, R. (eds.) SPECOM 2018. LNCS (LNAI), vol. 11096, pp. 501–510. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99579-3_52

    Chapter  Google Scholar 

  8. Ryumina, E., Verkholyak, O., Karpov, A.: Annotation confidence vs. training sample size: trade-off solution for partially-continuous categorical emotion recognition. In: Proceedings of Interspeech 2021, pp. 3690–3694 (2021). https://doi.org/10.21437/Interspeech.2021-1636

  9. Saleem, S.M., Zeebaree, S.R., Abdulrazzaq, M.B.: Real-life dynamic facial expression recognition: a review. J. Phys. Conf. Ser. 1963, 012010 (2021). IOP Publishing

    Google Scholar 

  10. Ben, X., et al.: Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans. Pattern Anal. Mach. Intell. (2021)

    Google Scholar 

  11. Saeed, A., Al-Hamadi, A., Niese, R., Elzobi, M.: Frame-based facial expression recognition using geometrical features. In: Advances in Human-Computer Interaction 2014 (2014)

    Google Scholar 

  12. Bargal, S.A., Barsoum, E., Ferrer, C.C., Zhang, C.: Emotion recognition in the wild from videos using images. In: Proceedings of the 18th International Conference on Multimodal Interaction (ICMI), pp. 433–436. ACM (2016)

    Google Scholar 

  13. Meng, D., Peng, X., Wang, K., Qiao, Y.: Frame attention networks for facial expression recognition in videos. In: Proceedings of the International Conference on Image Processing (ICIP), pp. 3866–3870. IEEE (2019)

    Google Scholar 

  14. Demochkina, P., Savchenko, A.V.: Neural network model for video-based facial expression recognition in-the-wild on mobile devices. In: Proceedings of International Conference on Information Technology and Nanotechnology (ITNT), pp. 1–5. IEEE (2021)

    Google Scholar 

  15. Savchenko, A.V., Khokhlova, Y.I.: About neural-network algorithms application in viseme classification problem with face video in audiovisual speech recognition systems. Optical Memory Neural Netw. 23(1), 34–42 (2014). https://doi.org/10.3103/S1060992X14010068

    Article  Google Scholar 

  16. Zhou, H., et al.: Exploring emotion features and fusion strategies for audio-video emotion recognition. In: Proceedings of International Conference on Multimodal Interaction (ICMI), pp. 562–566. ACM (2019)

    Google Scholar 

  17. Peña, A., Morales, A., Serna, I., Fierrez, J., Lapedriza, A.: Facial expressions as a vulnerability in face recognition. In: Proceedings of International Conference on Image Processing (ICIP), pp. 2988–2992. IEEE (2021)

    Google Scholar 

  18. Shahabinejad, M., Wang, Y., Yu, Y., Tang, J., Li, J.: Toward personalized emotion recognition: a face recognition based attention method for facial emotion recognition. In: Proceedings of 16th International Conference on Automatic Face and Gesture Recognition (FG), pp. 1–5. IEEE (2021)

    Google Scholar 

  19. Zhao, Y., Li, J., Zhang, S., Chen, L., Gong, Y.: Domain and speaker adaptation for Cortana speech recognition. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5984–5988. IEEE (2018)

    Google Scholar 

  20. Savchenko, L.V., Savchenko, A.V.: Speaker-aware training of speech emotion classifier with speaker recognition. In: Karpov, A., Potapova, R. (eds.) SPECOM 2021. LNCS (LNAI), vol. 12997, pp. 614–625. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87802-3_55

    Chapter  Google Scholar 

  21. Savchenko, A.V.: Phonetic words decoding software in the problem of Russian speech recognition. Autom. Remote. Control. 74(7), 1225–1232 (2013)

    Article  Google Scholar 

  22. Deng, J., Guo, J., Xue, N., Zafeiriou, S.: Arcface: additive angular margin loss for deep face recognition. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4690–4699. IEEE (2019)

    Google Scholar 

  23. Naas, S.A., Sigg, S.: Real-time emotion recognition for sales. In: Proceedings of 16th International Conference on Mobility, Sensing and Networking (MSN), pp. 584–591. IEEE (2020)

    Google Scholar 

  24. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

  25. Makarov, I., Bakhanova, M., Nikolenko, S., Gerasimova, O.: Self-supervised recurrent depth estimation with attention mechanisms. PeerJ Comput. Sci. 8, e865 (2022)

    Article  Google Scholar 

  26. Sokolova, A.D., Kharchevnikova, A.S., Savchenko, A.V.: Organizing multimedia data in video surveillance systems based on face verification with convolutional neural networks. In: van der Aalst, W.M.P., et al. (eds.) AIST 2017. LNCS, vol. 10716, pp. 223–230. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73013-4_20

  27. Perepelkina, O., Sterling, G., Konstantinova, M., Kazimirova, E.: RAMAS: the Russian acted multimodal affective set for affective computing and emotion recognition studies. In: Proceedings of European Society for Cognitive and Affective Neuroscience (ESCAN), pp. 86–86 (2018)

    Google Scholar 

  28. Savchenko, A.V.: Efficient facial representations for age, gender and identity recognition in organizing photo albums using multi-output convnet. PeerJ Comput. Sci. 5, e197 (2019)

    Google Scholar 

  29. Kollias, D., Zafeiriou, S.: Analysing affective behavior in the second ABAW2 competition. In: Proceedings of the International Conference on Computer Vision (ICCV), pp. 3652–3660. IEEE (2021)

    Google Scholar 

Download references

Acknowledgements

The work is supported by RSF (Russian Science Foundation) grant 20-71-10010.

Author information

Authors and Affiliations

  1. Laboratory of Algorithms and Technologies for Network Analysis, HSE University, Nizhny Novgorod, Russia

    Andrey V. Savchenko

Authors
  1. Andrey V. Savchenko
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrey V. Savchenko .

Editor information

Editors and Affiliations

  1. Télécom SudParis, Palaiseau, France

    Mounîm El Yacoubi

  2. École de Technologie Supérieure, Montreal, QC, Canada

    Eric Granger

  3. Hong Kong Baptist University, Kowloon, Kowloon, Hong Kong

    Pong Chi Yuen

  4. Indian Statistical Institute, Kolkata, India

    Umapada Pal

  5. Université Paris Cité, Paris, France

    Nicole Vincent

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Savchenko, A.V. (2022). Personalized Frame-Level Facial Expression Recognition in Video. In: El Yacoubi, M., Granger, E., Yuen, P.C., Pal, U., Vincent, N. (eds) Pattern Recognition and Artificial Intelligence. ICPRAI 2022. Lecture Notes in Computer Science, vol 13363. Springer, Cham. https://doi.org/10.1007/978-3-031-09037-0_37

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-09037-0_37

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-09036-3

  • Online ISBN: 978-3-031-09037-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation