Abstract
Understanding emotions of others is important for effective interactions among people. Therefore, it is likely similarly important in applications where people interact with or via virtual humans. However, while some studies have examined the recognisability of expressions by virtual avatars, it is currently unclear how generalisable the findings are across technologies and designs. To empirically examine how well people (N = 100) recognise dynamic facial expressions for a set of 12 proposed avatars, the expressions are tested at high (75%) and low intensity (25%) in the context of 2D computer screens. Also, the effects of the self-reported age, gender, mood and ability to recognise emotions by the annotator are examined. Then, these findings are compared to emotion recognition literature for avatars and real people with a similar context. Finally automated recognition models are applied to test automated emotion detection, as well as to establish what facial action units may contribute to the found patterns in recognisability of the proposed avatars. We conclude that on average the emotional expressions of the proposed avatars are recognisable and confusion patterns resemble those of real people, where specific emotion pairs are more difficult to distinguish. Negative effects are found for male avatar gender and the age of the participant, while no effect is found for the self-reported mood or ability to recognise emotion. Moreover, no difference is found in the mean recognition-rate between human and avatar-based studies, yet the variation among avatar recognition studies is substantial.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
del Aguila, J., González-Gualda, L.M., Játiva, M.A., Fernández-Sotos, P., Fernández-Caballero, A., García, A.S.: How interpersonal distance between avatar and human influences facial affect recognition in immersive virtual reality. Front. Psychol. 12, 2925 (2021)
Ahn, S.J.G., Fox, J., Bailenson, J.: Avatars. In: Bainbridge, W. (ed.) Leadership in Science and Technology: A Reference Handbook, pp. 695–702. Sage, Thousand Oaks, CA (2012)
Amini, R., Lisetti, C., Ruiz, G.: HapFACS 3.0: FACS-based facial expression generator for 3D speaking virtual characters. IEEE Tran. Affect. Comput. 6(4), 348–360 (2015)
ARtillery Intelligence: VR Usage & Consumer Attitudes, Wave VII - ARtillery Intelligence (2023). https://artilleryiq.com/reports/vr-usage-consumer-attitudes-wave-vii/
Barrett, L.F., Adolphs, R., Marsella, S., Martinez, A.M., Pollak, S.D.: Emotional expressions reconsidered: challenges to inferring emotion from human facial movements. Psychol. Sci. Public Interest 20(1), 1–68 (2019)
Binetti, N., Roubtsova, N., Carlisi, C., Cosker, D., Viding, E., Mareschal, I.: Genetic algorithms reveal profound individual differences in emotion recognition. Proc. Nat. Acad. Sci. United States Am. 119(45), e2201380119 (2022)
Calvo, M.G., Avero, P., Fernández-Martín, A., Recio, G.: Recognition thresholds for static and dynamic emotional faces. Emotion 16(8), 1186–1200 (2016)
Cao, Q., Yu, H., Charisse, P., Qiao, S., Stevens, B.: Is high-fidelity important for human-like virtual avatars in human computer interactions? Int. J. Netw. Dyn. Intell. 15–23 (2023)
Chamberland, J., Roy-Charland, A., Perron, M., Dickinson, J.: Distinction between fear and surprise: an interpretation-independent test of the perceptual-attentional limitation hypothesis. Soc. Neurosci. 12(6), 751–768 (2017)
Chang, D., Yin, Y., Li, Z., Tran, M., Soleymani, M.: LibreFace: an open-source toolkit for deep facial expression analysis. Preprint (2023)
Chattopadhyay, D., Ma, T., Sharifi, H., Martyn-Nemeth, P.: Computer-controlled virtual humans in patient-facing systems: systematic review and meta-analysis. J. Med. Internet Res. 22(7), e18839 (2020)
Cherbonnier, A., Michinov, N.: The recognition of emotions conveyed by emoticons and emojis: a systematic literature review. Technol. Mind, Behav. 3(2: Summer 2022) (2022)
Chevalier, P., Martin, J.C., Isableu, B., Bazile, C., Tapus, A.: Impact of sensory preferences of individuals with autism on the recognition of emotions expressed by two robots, an avatar, and a human. Auton. Robots 41(3), 613–635 (2017)
Cordaro, D.T., Sun, R., Keltner, D., Kamble, S., Huddar, N., McNeil, G.: Universals and cultural variations in 22 emotional expressions across five cultures. Emotion 18(1), 75–93 (2018)
Crivelli, C., Fridlund, A.J.: Inside-out: from basic emotions theory to the behavioral ecology view. J. Nonverbal Behav. 43(2), 161–194 (2019)
Daher, K., Bardelli, Z., Casas, J., Mugellini, E., Khaled, O.A., Lalanne, D.: Embodied conversational agent for emotional recognition training. In: ACHI 2020 : The Thirteenth International Conference on Advances in Computer-Human Interactions, pp. 384–390. IARIA, Valencia, Spain (2020)
de Araújo Luz Junior, J., Formico Rodrigues, M.A.: Comparative analysis of facial expression recognition systems for evaluating emotional states in virtual humans. In: ACM International Conference Proceeding Series, pp. 38–47 (2023)
Durupinar, F., Kim, J.: Facial emotion recognition of virtual humans with different genders, races, and ages. In: Proceedings - SAP 2022: ACM Symposium on Applied Perception, vol. 22 (2022)
Dwivedi, Y.K., et al.: Metaverse beyond the hype: multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. Int. J. Inf. Manage. 66, 102542 (2022)
Dyck, M., Winbeck, M., Leiberg, S., Chen, Y., Gur, R.C., Mathiak, K.: Recognition profile of emotions in natural and virtual faces. PLoS ONE 3(11), 1–8 (2008)
Ekdahl, D., Osler, L.: Expressive avatars: vitality in virtual worlds. Philos. Technol. 36(2), 1–28 (2023)
Ekman, P., Friesen, W.V.: Facial action coding system. Environ. Psychol. Nonverbal Behav. (1978)
Elfenbein, H.A., Ambady, N.: On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychol. Bull. 128(2), 203–235 (2002)
Fernández-Sotos, P., García, A.S., Vicente-Querol, M.A., Lahera, G., Rodriguez-Jimenez, R., Fernández-Caballero, A.: Validation of dynamic virtual faces for facial affect recognition. PLOS ONE 16(1), e0246001 (2021)
Fischer, A.H., Kret, M.E., Broekens, J.: Gender differences in emotion perception and self-reported emotional intelligence: a test of the emotion sensitivity hypothesis. PLOS ONE 13(1), e0190712 (2018)
Fujimura, T., Umemura, H.: Development and validation of a facial expression database based on the dimensional and categorical model of emotions. Cogn. Emotion 32(8), 1663–1670 (2018)
García, A.S., Fernández-Sotos, P., Vicente-Querol, M.A., Lahera, G., Rodriguez-Jimenez, R., Fernández-Caballero, A.: Design of reliable virtual human facial expressions and validation by healthy people. Integr. Comput.-Aided Eng. 27(3), 287–299 (2020)
Geraets, C.N., et al.: Virtual reality facial emotion recognition in social environments: an eye-tracking study. Internet Interventions 25, 100432 (2021)
Gonçalves, A.R., Fernandes, C., Pasion, R., Ferreira-Santos, F., Barbosa, F., Marques-Teixeira, J.: Effects of age on the identification of emotions in facial expressions: a metaanalysis. PeerJ 2018(7) (2018)
Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_16
Gutiérrez-Maldonado, J., Rus-Calafell, M., González-Conde, J.: Creation of a new set of dynamic virtual reality faces for the assessment and training of facial emotion recognition ability. Virtual Reality 18(1), 61–71 (2014)
van Haeringen, E.S., Veltmeijer, E.A., Gerritsen, C.: Empirical validation of an agent-based model of emotion contagion. IEEE Trans. Affect. Comput. (2023)
Hayes, G.S., et al.: Task characteristics influence facial emotion recognition age-effects: a meta-analytic review. Psychol. Aging 35(2), 295–315 (2020)
Hofer, M., Hüsser, A., Prabhu, S.: The effect of an avatar’s emotional expressions on players’ fear reactions: the mediating role of embodiment. Comput. Hum. Behav. 75, 883–890 (2017)
Israelashvili, J., Fischer, A.: Recognition of emotion from verbal and nonverbal expressions and its relation to effective communication: a preliminary evidence of a positive link. J. Intell. 11(1) (2023)
Jack, R.E., Garrod, O.G., Yu, H., Caldara, R., Schyns, P.G.: Facial expressions of emotion are not culturally universal. Proc. Nat. Acad. Sci. United States Am. 109(19), 7241–7244 (2012)
Keltner, D., Sauter, D., Tracy, J., Cowen, A.: Emotional expression: advances in basic emotion theory. J. Nonverbal Behav. 43(2), 133–160 (2019)
Krumhuber, E.G., Küster, D., Namba, S., Shah, D., Calvo, M.G.: Emotion recognition from posed and spontaneous dynamic expressions: human observers versus machine analysis. Emotion 21(2), 447–451 (2021)
Krumhuber, E.G., Küster, D., Namba, S., Skora, L.: Human and machine validation of 14 databases of dynamic facial expressions. Behav. Res. Methods 53(2), 686–701 (2021)
Kyrlitsias, C., Michael-Grigoriou, D.: Social interaction with agents and avatars in immersive virtual environments: a survey. Front. Virtual Reality 2, 786665 (2022)
Lee, J., Kim, S., Kim, S., Park, J., Sohn, K.: Context-aware emotion recognition networks. In" Proceedings of the IEEE International Conference on Computer Vision, vol. 2019-Octob, pp. 10142–10151 (2019)
Leung, F.Y.N., et al.: Emotion recognition across visual and auditory modalities in autism spectrum disorder: a systematic review and meta-analysis. Develop. Rev. 63, 101000 (2022)
Li, S., Deng, W.: Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans. Image Process. 28(1), 356–370 (2019)
Luo, C., Song, S., Xie, W., Shen, L., Gunes, H.: Learning multi-dimensional edge feature-based au relation graph for facial action unit recognition. In: Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pp. 1239–1246 (2022)
Manierka, M.S., Rezaei, R., Palacios, S., Haigh, S.M., Hutsler, J.J.: In the mood to be social: Affective state influences facial emotion recognition in healthy adults. Emotion (Washington, D.C.) 21(7), 1576–1581 (2021)
Mao, J., Xu, R., Yin, X., Chang, Y., Nie, B., Huang, A.: POSTER++: a simpler and stronger facial expression recognition network. Preprint (2023)
Marcos-Pablos, S., González-Pablos, E., Martín-Lorenzo, C., Flores, L.A., Gómez-García-Bermejo, J., Zalama, E.: Virtual avatar for emotion recognition in patients with schizophrenia: a pilot study. Front. Human Neurosci. 10, 12 (2016)
Martinez, L., Falvello, V.B., Aviezer, H., Todorov, A.: Contributions of facial expressions and body language to the rapid perception of dynamic emotions. Cogn. Emoti. 30(5), 939–952 (2016)
Matsumoto, D., Willingham, B.: Spontaneous facial expressions of emotion of congenitally and noncongenitally blind individuals. J. Pers. Soc. Psychol. 96(1), 1–10 (2009)
Miao, F., Kozlenkova, I.V., Wang, H., Xie, T., Palmatier, R.W.: An emerging theory of avatar marketing. J. Market. 86(1), 67–90 (2022)
Mollahosseini, A., Hasani, B., Mahoor, M.H.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(01), 18–31 (2019)
Monferrer, M., et al.: Facial emotion recognition in patients with depression compared to healthy controls when using human avatars. Sci. Rep. 13(1), 1–10 (2023)
Muros, N.I., et al.: Facial affect recognition by patients with schizophrenia using human avatars. J. Clin. Med. 10(9), 1904 (2021)
O’Rourke, S.R., Branford, K.R., Brooks, T.L., Ives, L.T., Nagendran, A., Compton, S.N.: The emotional and behavioral impact of delivering bad news to virtual versus real standardized patients: a pilot study. Teach. Learn. Med. 32(2), 139–149 (2019)
Pham, L., Vu, T.H., Tran, T.A.: Facial expression recognition using residual masking network. In: Proceedings - International Conference on Pattern Recognition, pp. 4513–4519 (2020)
Rafiee, Y., Schacht, A.: Sex differences in emotion recognition: investigating the moderating effects of stimulus features. Cogn. Emot. (2023)
Richoz, A.R., Lao, J., Pascalis, O., Caldara, R.: Tracking the recognition of static and dynamic facial expressions of emotion across the life span. J. Vis. 18(9), 5–5 (2018)
Rodger, H., et al.: The recognition of facial expressions of emotion in deaf and hearing individuals. Heliyon 7(5) (2021)
Rymarczyk, K., Zurawski, Ł., Jankowiak-Siuda, K., Szatkowska, I.: Emotional empathy and facial mimicry for static and dynamic facial expressions of fear and disgust. Front. Psychol. 7(NOV), 1853 (2016)
Saquinaula, A., Juarez, A., Geigel, J., Bailey, R., Alm, C.O.: Emotional empathy and facial mimicry of avatar faces. In: Proceedings - 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2022, pp. 770–771 (2022)
Schmid, P.C., Schmid Mast, M.: Mood effects on emotion recognition. Motiv. Emot. 34(3), 288–292 (2010)
Sollfrank, T., et al.: The effects of dynamic and static emotional facial expressions of humans and their avatars on the EEG: an ERP and ERD/ERS study. Front. Neurosci. 15, 651044 (2021)
Suma, T., Sonia, B., Agyemang Baffour, K., Oyekoya, O.: The effects of avatar voice and facial expression intensity on emotional recognition and user perception. In: Proceedings - SIGGRAPH Asia 2023: Technical Communications, SA Technical Communications 2023 (2023)
Sun, Y., Won, A.S.: Despite appearances: comparing emotion recognition in abstract and humanoid avatars using nonverbal behavior in social virtual reality. Front. Virtual Reality 2, 694453 (2021)
Torregrossa, L.J., et al.: Decoupling of spontaneous facial mimicry from emotion recognition in schizophrenia. Psychiatry Res. 275, 169–176 (2019)
Trautmann, S.A., Fehr, T., Herrmann, M.: Emotions in motion: dynamic compared to static facial expressions of disgust and happiness reveal more widespread emotion-specific activations. Brain Res. 1284, 100–115 (2009)
Vicente-Querol, M.Á., García, A.S., Fernández-Sotos, P., Rodriguez-Jimenez, R., Fernández-Caballero, A.: Development and validation of basic virtual human facial emotion expressions. In: Ferrández Vicente, J.M., Álvarez-Sánchez, J.R., de la Paz López, F., Toledo Moreo, J., Adeli, H. (eds.) IWINAC 2019. LNCS, vol. 11486, pp. 222–231. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-19591-5_23
Wingenbach, T.S., Ashwin, C., Brosnan, M.: Validation of the Amsterdam dynamic facial expression set - bath intensity variations (ADFES-BIV): a set of videos expressing low, intermediate, and high intensity emotions. PLOS ONE 11(1), e0147112 (2016)
Zane, E., Yang, Z., Pozzan, L., Guha, T., Narayanan, S., Grossman, R.B.: Motion-capture patterns of voluntarily mimicked dynamic facial expressions in children and adolescents with and without ASD. J. Autism Dev. Disorders 49(3), 1062–1079 (2019)
Zhang, J., Chen, Q., Lu, J., Wang, X., Liu, L., Feng, Y.: Emotional expression by artificial intelligence chatbots to improve customer satisfaction: underlying mechanism and boundary conditions. Tourism Manage. 100, 104835 (2024)
Zupan, B., Eskritt, M.: Facial and vocal emotion recognition in adolescence: a systematic review. adolescent research review (2023)
Acknowledgements
This work is part of the research programme Innovational Research Incentives Scheme Vidi SSH 2017 with project number 016.Vidi.185.178, which is financed by the Dutch Research Council (NWO).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
van Haeringen, E., Otte, M., Gerritsen, C. (2025). Human Recognition of Emotions Expressed by Human-Like Avatars on 2D Screens. In: Oliehoek, F.A., Kok, M., Verwer, S. (eds) Artificial Intelligence and Machine Learning. BNAIC/Benelearn 2023. Communications in Computer and Information Science, vol 2187. Springer, Cham. https://doi.org/10.1007/978-3-031-74650-5_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-74650-5_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-74649-9
Online ISBN: 978-3-031-74650-5
eBook Packages: Artificial Intelligence (R0)