Abstract
Facial expressions are an important channel for interpersonal communication and comprehension, since people externalize their emotions through a variety of facial expressions. Technology, in particular, deep learning algorithms, can detect and analyze human emotions in real time, which paves the way for advanced user interfaces or adjustable devices and applications. Based on this, the work described in this paper presents a system that identifies three groups of emotions, positive, negative, and neutral, for adjusting user experience in Virtual Reality (VR) games. This, however, introduces an additional challenge in classifying expressions, because of the partial occlusion of the face. To solve this problem, four CNNs were used: VGG-19, ResNet-18, EfficientNet-b1, and Mini-Xception, as well as the ensemble of the three most accurate models (VGG-19, EfficientNet-b1, ResNet-18) via max voting. As expected, this ensemble, which was designated as VERNet, was the most accurate model, with an accuracy of 85.7% without partial occlusion of the face and 82.7% with occlusion. When compared with the accuracy of VGG-19, the results of VERNet only differ by 1% when compared with VGG-19. The minor difference between the results with and without occlusion demonstrated that an outside camera can be a very robust solution for tracking human facial expressions in VR environments.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alarood, A.A., Faheem, M., Al-Khasawneh, M.A., Alzahrani, A.I.A., Alshdadi, A.A.: Secure medical image transmission using deep neural network in e-health applications. Healthc. Technol. Lett. 10(4), 87–98 (2023). https://doi.org/10.1049/htl2.12049
Arriaga, O., Valdenegro-Toro, M., Plöger, P.: Real-time convolutional neural networks for emotion and gender classification. arXiv preprint arXiv:1710.07557 (2017)
Barsoum, E., Zhang, C., Ferrer, C.C., Zhang, Z.: Training deep networks for facial expression recognition with crowd-sourced label distribution. In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 279–283 (2016)
Bie, M., Liu, Q., Xu, H., Gao, Y., Che, X.: FEMFER: feature enhancement for multi-faces expression recognition in classroom images. Multimedia Tools Appl. (2023). https://doi.org/10.1007/s11042-023-15808-w
Canedo, D., Neves, A.: Mood estimation based on facial expressions and postures. In: Proceedings of the RECPAD, pp. 49–50 (2020)
Georgescu, M.I., Ionescu, R.T.: Teacher-student training and triplet loss for facial expression recognition under occlusion. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 2288–2295. IEEE (2021)
Glikson, E., Woolley, A.W.: Human trust in artificial intelligence: review of empirical research. Acad. Manag. Ann. 14(2), 627–660 (2020). https://doi.org/10.5465/annals.2018.0057
Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Korea, 3–7 November 2013. Proceedings, Part III 20, pp. 117–124. Springer (2013)
Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_16
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hong, J.W., Cruz, I., Williams, D.: AI, you can drive my car: how we evaluate human drivers vs. self-driving cars. Comput. Hum. Behav. 125, 106944 (2021). https://doi.org/10.1016/j.chb.2021.106944
Hong, K., Chalup, S.K., King, R.A.: A component based approach for classifying the seven universal facial expressions of emotion. In: 2013 IEEE Symposium on Computational Intelligence for Creativity and Affective Computing (CICAC), pp. 1–8 (2013). https://doi.org/10.1109/CICAC.2013.6595214
Ikechukwu, A.V., Murali, S., Deepu, R., Shivamurthy, R.: ResNet-50 vs VGG-19 vs training from scratch: a comparative analysis of the segmentation and classification of Pneumonia from chest X-ray images. Glob. Trans. Proc. 2(2), 375–381 (2021)
Jie, Y., et al.: Combined multi-layer feature fusion and edge detection method for distributed photovoltaic power station identification. Energies 13, 6742 (2020). https://doi.org/10.3390/en13246742
Kumar, B., Bedi, R.K., Gupta, S.K.: Facial gesture recognition for emotion detection: a review of methods and advancements. In: Handbook of Research on AI-Based Technologies and Applications in the Era of the Metaverse, pp. 342–358. IGI Global (2023). https://doi.org/10.4018/978-1-6684-8851-5.ch018. https://www.igi-global.com/chapter/facial-gesture-recognition-for-emotion-detection/www.igi-global.com/chapter/facial-gesture-recognition-for-emotion-detection/326039
Li, H., Wang, N., Ding, X., Yang, X., Gao, X.: Adaptively learning facial expression representation via CF labels and distillation. IEEE Trans. Image Process. 30, 2016–2028 (2021)
Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2852–2861 (2017)
Lin, Y., Lan, Y., Wang, S.: A method for evaluating the learning concentration in head-mounted virtual reality interaction. Virtual Reality 27(2), 863–885 (2023)
Lopes, J.C., Lopes, R.P.: A review of dynamic difficulty adjustment methods for serious games. In: Pereira, A.I., Košir, A., Fernandes, F.P., Pacheco, M.F., Teixeira, J.P., Lopes, R.P. (eds.) Optimization, Learning Algorithms and Applications. Communications in Computer and Information Science, pp. 144–159. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-23236-7_11
Lopes, R.P., et al.: Digital technologies for innovative mental health rehabilitation. Electronics 10(18), 2260 (2021). https://doi.org/10.3390/electronics10182260. https://www.mdpi.com/2079-9292/10/18/2260
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101. IEEE (2010)
Mollahosseini, A., Hasani, B., Mahoor, M.H.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2017)
Mozaffari, L., Brekke, M.M., Gajaruban, B., Purba, D., Zhang, J.: Facial expression recognition using deep neural network. In: 2023 3rd International Conference on Applied Artificial Intelligence (ICAPAI), pp. 1–9. IEEE (2023)
Park, S., Ryu, J.: Exploring preservice teachers’ emotional experiences in an immersive virtual teaching simulation through facial expression recognition. Int. J. Hum.-Comput. Interact. 35(6), 521–533 (2019). https://doi.org/10.1080/10447318.2018.1469710
Petrou, N., Christodoulou, G., Avgerinakis, K., Kosmides, P.: Lightweight mood estimation algorithm for faces under partial occlusion. In: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments, pp. 402–407 (2023)
Rawal, N., Koert, D., Turan, C., Kersting, K., Peters, J., Stock-Homburg, R.: ExGenNet: learning to generate robotic facial expression using facial expression recognition. Front. Robot. AI 8 (2022). https://www.frontiersin.org/articles/10.3389/frobt.2021.730317
Rodrigues, A.S.F., Lopes, J.C., Lopes, R.P., Teixeira, L.F.: Classification of facial expressions under partial occlusion for VR games. In: Pereira, A.I., Košir, A., Fernandes, F.P., Pacheco, M.F., Teixeira, J.P., Lopes, R.P. (eds.) Optimization, Learning Algorithms and Applications. Communications in Computer and Information Science, pp. 804–819. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-23236-7_55
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Sánchez, P.C., Bennett, C.C.: Facial expression recognition via transfer learning in cooperative game paradigms for enhanced social AI. J. Multimodal User Interfaces (2023). https://doi.org/10.1007/s12193-023-00410-z
Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)
Yang, B., Jianming, W., Hattori, G.: Face mask aware robust facial expression recognition during the COVID-19 pandemic. In: 2021 IEEE International Conference on Image Processing (ICIP), pp. 240–244. IEEE (2021)
Yang, B., Wu, J., Hattori, G.: Facial expression recognition with the advent of face masks. In: Proceedings of the 19th International Conference on Mobile and Ubiquitous Multimedia, pp. 335–337 (2020)
Yeung, S., Downing, N.L., Fei-Fei, L., Milstein, A., et al.: Bedside computer vision-moving artificial intelligence from driver assistance to patient safety. N. Engl. J. Med. 378(14), 1271–1273 (2018)
Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)
Acknowledgments
The authors are grateful to the Foundation for Science and Technology (FCT, Portugal) for financial support through national funds FCT/MCTES (PIDDAC) to CeDRI, UIDB/05757/2020 (DOI: 10.54499/UIDB/05757/2020) and UIDP/05757/2020 (DOI: 10.54499/UIDP/05757/2020) and SusTEC, LA/P/0007/2020 (DOI: 10.54499/LA/P/0007/2020).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Rodrigues, A.S., Lopes, J.C., Lopes, R.P. (2024). Facial Expression Recognition in Virtual Reality Simulations. In: Pereira, A.I., et al. Optimization, Learning Algorithms and Applications. OL2A 2024. Communications in Computer and Information Science, vol 2280. Springer, Cham. https://doi.org/10.1007/978-3-031-77426-3_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-77426-3_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-77425-6
Online ISBN: 978-3-031-77426-3
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)