Skip to main content

Facial Expression Recognition in Virtual Reality Simulations

  • Conference paper
  • First Online:
Optimization, Learning Algorithms and Applications (OL2A 2024)

Abstract

Facial expressions are an important channel for interpersonal communication and comprehension, since people externalize their emotions through a variety of facial expressions. Technology, in particular, deep learning algorithms, can detect and analyze human emotions in real time, which paves the way for advanced user interfaces or adjustable devices and applications. Based on this, the work described in this paper presents a system that identifies three groups of emotions, positive, negative, and neutral, for adjusting user experience in Virtual Reality (VR) games. This, however, introduces an additional challenge in classifying expressions, because of the partial occlusion of the face. To solve this problem, four CNNs were used: VGG-19, ResNet-18, EfficientNet-b1, and Mini-Xception, as well as the ensemble of the three most accurate models (VGG-19, EfficientNet-b1, ResNet-18) via max voting. As expected, this ensemble, which was designated as VERNet, was the most accurate model, with an accuracy of 85.7% without partial occlusion of the face and 82.7% with occlusion. When compared with the accuracy of VGG-19, the results of VERNet only differ by 1% when compared with VGG-19. The minor difference between the results with and without occlusion demonstrated that an outside camera can be a very robust solution for tracking human facial expressions in VR environments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.kaggle.com/jafarhussain786/datasets.

  2. 2.

    https://unsplash.com/.

  3. 3.

    https://www.pexels.com/search/face/.

  4. 4.

    https://pixabay.com/vectors/.

References

  1. Alarood, A.A., Faheem, M., Al-Khasawneh, M.A., Alzahrani, A.I.A., Alshdadi, A.A.: Secure medical image transmission using deep neural network in e-health applications. Healthc. Technol. Lett. 10(4), 87–98 (2023). https://doi.org/10.1049/htl2.12049

    Article  Google Scholar 

  2. Arriaga, O., Valdenegro-Toro, M., Plöger, P.: Real-time convolutional neural networks for emotion and gender classification. arXiv preprint arXiv:1710.07557 (2017)

  3. Barsoum, E., Zhang, C., Ferrer, C.C., Zhang, Z.: Training deep networks for facial expression recognition with crowd-sourced label distribution. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 279–283 (2016)

    Google Scholar 

  4. Bie, M., Liu, Q., Xu, H., Gao, Y., Che, X.: FEMFER: feature enhancement for multi-faces expression recognition in classroom images. Multimedia Tools Appl. (2023). https://doi.org/10.1007/s11042-023-15808-w

    Article  Google Scholar 

  5. Canedo, D., Neves, A.: Mood estimation based on facial expressions and postures. In: Proceedings of the RECPAD, pp. 49–50 (2020)

    Google Scholar 

  6. Georgescu, M.I., Ionescu, R.T.: Teacher-student training and triplet loss for facial expression recognition under occlusion. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 2288–2295. IEEE (2021)

    Google Scholar 

  7. Glikson, E., Woolley, A.W.: Human trust in artificial intelligence: review of empirical research. Acad. Manag. Ann. 14(2), 627–660 (2020). https://doi.org/10.5465/annals.2018.0057

    Article  Google Scholar 

  8. Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Korea, 3–7 November 2013. Proceedings, Part III 20, pp. 117–124. Springer (2013)

    Google Scholar 

  9. Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_16

    Chapter  Google Scholar 

  10. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  11. Hong, J.W., Cruz, I., Williams, D.: AI, you can drive my car: how we evaluate human drivers vs. self-driving cars. Comput. Hum. Behav. 125, 106944 (2021). https://doi.org/10.1016/j.chb.2021.106944

  12. Hong, K., Chalup, S.K., King, R.A.: A component based approach for classifying the seven universal facial expressions of emotion. In: 2013 IEEE Symposium on Computational Intelligence for Creativity and Affective Computing (CICAC), pp. 1–8 (2013). https://doi.org/10.1109/CICAC.2013.6595214

  13. Ikechukwu, A.V., Murali, S., Deepu, R., Shivamurthy, R.: ResNet-50 vs VGG-19 vs training from scratch: a comparative analysis of the segmentation and classification of Pneumonia from chest X-ray images. Glob. Trans. Proc. 2(2), 375–381 (2021)

    Article  Google Scholar 

  14. Jie, Y., et al.: Combined multi-layer feature fusion and edge detection method for distributed photovoltaic power station identification. Energies 13, 6742 (2020). https://doi.org/10.3390/en13246742

    Article  Google Scholar 

  15. Kumar, B., Bedi, R.K., Gupta, S.K.: Facial gesture recognition for emotion detection: a review of methods and advancements. In: Handbook of Research on AI-Based Technologies and Applications in the Era of the Metaverse, pp. 342–358. IGI Global (2023). https://doi.org/10.4018/978-1-6684-8851-5.ch018. https://www.igi-global.com/chapter/facial-gesture-recognition-for-emotion-detection/www.igi-global.com/chapter/facial-gesture-recognition-for-emotion-detection/326039

  16. Li, H., Wang, N., Ding, X., Yang, X., Gao, X.: Adaptively learning facial expression representation via CF labels and distillation. IEEE Trans. Image Process. 30, 2016–2028 (2021)

    Article  Google Scholar 

  17. Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2852–2861 (2017)

    Google Scholar 

  18. Lin, Y., Lan, Y., Wang, S.: A method for evaluating the learning concentration in head-mounted virtual reality interaction. Virtual Reality 27(2), 863–885 (2023)

    Article  Google Scholar 

  19. Lopes, J.C., Lopes, R.P.: A review of dynamic difficulty adjustment methods for serious games. In: Pereira, A.I., Košir, A., Fernandes, F.P., Pacheco, M.F., Teixeira, J.P., Lopes, R.P. (eds.) Optimization, Learning Algorithms and Applications. Communications in Computer and Information Science, pp. 144–159. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-23236-7_11

  20. Lopes, R.P., et al.: Digital technologies for innovative mental health rehabilitation. Electronics 10(18), 2260 (2021). https://doi.org/10.3390/electronics10182260. https://www.mdpi.com/2079-9292/10/18/2260

  21. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101. IEEE (2010)

    Google Scholar 

  22. Mollahosseini, A., Hasani, B., Mahoor, M.H.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2017)

    Article  Google Scholar 

  23. Mozaffari, L., Brekke, M.M., Gajaruban, B., Purba, D., Zhang, J.: Facial expression recognition using deep neural network. In: 2023 3rd International Conference on Applied Artificial Intelligence (ICAPAI), pp. 1–9. IEEE (2023)

    Google Scholar 

  24. Park, S., Ryu, J.: Exploring preservice teachers’ emotional experiences in an immersive virtual teaching simulation through facial expression recognition. Int. J. Hum.-Comput. Interact. 35(6), 521–533 (2019). https://doi.org/10.1080/10447318.2018.1469710

    Article  Google Scholar 

  25. Petrou, N., Christodoulou, G., Avgerinakis, K., Kosmides, P.: Lightweight mood estimation algorithm for faces under partial occlusion. In: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, pp. 402–407 (2023)

    Google Scholar 

  26. Rawal, N., Koert, D., Turan, C., Kersting, K., Peters, J., Stock-Homburg, R.: ExGenNet: learning to generate robotic facial expression using facial expression recognition. Front. Robot. AI 8 (2022). https://www.frontiersin.org/articles/10.3389/frobt.2021.730317

  27. Rodrigues, A.S.F., Lopes, J.C., Lopes, R.P., Teixeira, L.F.: Classification of facial expressions under partial occlusion for VR games. In: Pereira, A.I., Košir, A., Fernandes, F.P., Pacheco, M.F., Teixeira, J.P., Lopes, R.P. (eds.) Optimization, Learning Algorithms and Applications. Communications in Computer and Information Science, pp. 804–819. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-23236-7_55

  28. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  29. Sánchez, P.C., Bennett, C.C.: Facial expression recognition via transfer learning in cooperative game paradigms for enhanced social AI. J. Multimodal User Interfaces (2023). https://doi.org/10.1007/s12193-023-00410-z

    Article  Google Scholar 

  30. Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)

    Google Scholar 

  31. Yang, B., Jianming, W., Hattori, G.: Face mask aware robust facial expression recognition during the COVID-19 pandemic. In: 2021 IEEE International Conference on Image Processing (ICIP), pp. 240–244. IEEE (2021)

    Google Scholar 

  32. Yang, B., Wu, J., Hattori, G.: Facial expression recognition with the advent of face masks. In: Proceedings of the 19th International Conference on Mobile and Ubiquitous Multimedia, pp. 335–337 (2020)

    Google Scholar 

  33. Yeung, S., Downing, N.L., Fei-Fei, L., Milstein, A., et al.: Bedside computer vision-moving artificial intelligence from driver assistance to patient safety. N. Engl. J. Med. 378(14), 1271–1273 (2018)

    Article  Google Scholar 

  34. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

Download references

Acknowledgments

The authors are grateful to the Foundation for Science and Technology (FCT, Portugal) for financial support through national funds FCT/MCTES (PIDDAC) to CeDRI, UIDB/05757/2020 (DOI: 10.54499/UIDB/05757/2020) and UIDP/05757/2020 (DOI: 10.54499/UIDP/05757/2020) and SusTEC, LA/P/0007/2020 (DOI: 10.54499/LA/P/0007/2020).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Júlio Castro Lopes .

Editor information

Editors and Affiliations

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rodrigues, A.S., Lopes, J.C., Lopes, R.P. (2024). Facial Expression Recognition in Virtual Reality Simulations. In: Pereira, A.I., et al. Optimization, Learning Algorithms and Applications. OL2A 2024. Communications in Computer and Information Science, vol 2280. Springer, Cham. https://doi.org/10.1007/978-3-031-77426-3_3

Download citation

Publish with us

Policies and ethics