Skip to main content
Springer Nature Link
Log in

Human Recognition of Emotions Expressed by Human-Like Avatars on 2D Screens

  • Conference paper
  • First Online:
Artificial Intelligence and Machine Learning (BNAIC/Benelearn 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2187))

Included in the following conference series:

  • 16 Accesses

Abstract

Understanding emotions of others is important for effective interactions among people. Therefore, it is likely similarly important in applications where people interact with or via virtual humans. However, while some studies have examined the recognisability of expressions by virtual avatars, it is currently unclear how generalisable the findings are across technologies and designs. To empirically examine how well people (N = 100) recognise dynamic facial expressions for a set of 12 proposed avatars, the expressions are tested at high (75%) and low intensity (25%) in the context of 2D computer screens. Also, the effects of the self-reported age, gender, mood and ability to recognise emotions by the annotator are examined. Then, these findings are compared to emotion recognition literature for avatars and real people with a similar context. Finally automated recognition models are applied to test automated emotion detection, as well as to establish what facial action units may contribute to the found patterns in recognisability of the proposed avatars. We conclude that on average the emotional expressions of the proposed avatars are recognisable and confusion patterns resemble those of real people, where specific emotion pairs are more difficult to distinguish. Negative effects are found for male avatar gender and the age of the participant, while no effect is found for the self-reported mood or ability to recognise emotion. Moreover, no difference is found in the mean recognition-rate between human and avatar-based studies, yet the variation among avatar recognition studies is substantial.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.reallusion.com.

  2. 2.

    https://www.unity.com.

  3. 3.

    https://www.qualtrics.com.

  4. 4.

    https://www.prolific.com/academic-researchers.

  5. 5.

    https://posit.co.

  6. 6.

    https://paperswithcode.com/task/facial-expression-recognition.

  7. 7.

    https://paperswithcode.com/task/facial-action-unit-detection.

References

  1. del Aguila, J., González-Gualda, L.M., Játiva, M.A., Fernández-Sotos, P., Fernández-Caballero, A., García, A.S.: How interpersonal distance between avatar and human influences facial affect recognition in immersive virtual reality. Front. Psychol. 12, 2925 (2021)

    Google Scholar 

  2. Ahn, S.J.G., Fox, J., Bailenson, J.: Avatars. In: Bainbridge, W. (ed.) Leadership in Science and Technology: A Reference Handbook, pp. 695–702. Sage, Thousand Oaks, CA (2012)

    Google Scholar 

  3. Amini, R., Lisetti, C., Ruiz, G.: HapFACS 3.0: FACS-based facial expression generator for 3D speaking virtual characters. IEEE Tran. Affect. Comput. 6(4), 348–360 (2015)

    Article  Google Scholar 

  4. ARtillery Intelligence: VR Usage & Consumer Attitudes, Wave VII - ARtillery Intelligence (2023). https://artilleryiq.com/reports/vr-usage-consumer-attitudes-wave-vii/

  5. Barrett, L.F., Adolphs, R., Marsella, S., Martinez, A.M., Pollak, S.D.: Emotional expressions reconsidered: challenges to inferring emotion from human facial movements. Psychol. Sci. Public Interest 20(1), 1–68 (2019)

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Binetti, N., Roubtsova, N., Carlisi, C., Cosker, D., Viding, E., Mareschal, I.: Genetic algorithms reveal profound individual differences in emotion recognition. Proc. Nat. Acad. Sci. United States Am. 119(45), e2201380119 (2022)

    Article  CAS  Google Scholar 

  7. Calvo, M.G., Avero, P., Fernández-Martín, A., Recio, G.: Recognition thresholds for static and dynamic emotional faces. Emotion 16(8), 1186–1200 (2016)

    Article  PubMed  Google Scholar 

  8. Cao, Q., Yu, H., Charisse, P., Qiao, S., Stevens, B.: Is high-fidelity important for human-like virtual avatars in human computer interactions? Int. J. Netw. Dyn. Intell. 15–23 (2023)

    Google Scholar 

  9. Chamberland, J., Roy-Charland, A., Perron, M., Dickinson, J.: Distinction between fear and surprise: an interpretation-independent test of the perceptual-attentional limitation hypothesis. Soc. Neurosci. 12(6), 751–768 (2017)

    PubMed  Google Scholar 

  10. Chang, D., Yin, Y., Li, Z., Tran, M., Soleymani, M.: LibreFace: an open-source toolkit for deep facial expression analysis. Preprint (2023)

    Google Scholar 

  11. Chattopadhyay, D., Ma, T., Sharifi, H., Martyn-Nemeth, P.: Computer-controlled virtual humans in patient-facing systems: systematic review and meta-analysis. J. Med. Internet Res. 22(7), e18839 (2020)

    Article  PubMed  PubMed Central  Google Scholar 

  12. Cherbonnier, A., Michinov, N.: The recognition of emotions conveyed by emoticons and emojis: a systematic literature review. Technol. Mind, Behav. 3(2: Summer 2022) (2022)

    Google Scholar 

  13. Chevalier, P., Martin, J.C., Isableu, B., Bazile, C., Tapus, A.: Impact of sensory preferences of individuals with autism on the recognition of emotions expressed by two robots, an avatar, and a human. Auton. Robots 41(3), 613–635 (2017)

    Article  Google Scholar 

  14. Cordaro, D.T., Sun, R., Keltner, D., Kamble, S., Huddar, N., McNeil, G.: Universals and cultural variations in 22 emotional expressions across five cultures. Emotion 18(1), 75–93 (2018)

    Article  PubMed  Google Scholar 

  15. Crivelli, C., Fridlund, A.J.: Inside-out: from basic emotions theory to the behavioral ecology view. J. Nonverbal Behav. 43(2), 161–194 (2019)

    Article  Google Scholar 

  16. Daher, K., Bardelli, Z., Casas, J., Mugellini, E., Khaled, O.A., Lalanne, D.: Embodied conversational agent for emotional recognition training. In: ACHI 2020 : The Thirteenth International Conference on Advances in Computer-Human Interactions, pp. 384–390. IARIA, Valencia, Spain (2020)

    Google Scholar 

  17. de Araújo Luz Junior, J., Formico Rodrigues, M.A.: Comparative analysis of facial expression recognition systems for evaluating emotional states in virtual humans. In: ACM International Conference Proceeding Series, pp. 38–47 (2023)

    Google Scholar 

  18. Durupinar, F., Kim, J.: Facial emotion recognition of virtual humans with different genders, races, and ages. In: Proceedings - SAP 2022: ACM Symposium on Applied Perception, vol. 22 (2022)

    Google Scholar 

  19. Dwivedi, Y.K., et al.: Metaverse beyond the hype: multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. Int. J. Inf. Manage. 66, 102542 (2022)

    Article  Google Scholar 

  20. Dyck, M., Winbeck, M., Leiberg, S., Chen, Y., Gur, R.C., Mathiak, K.: Recognition profile of emotions in natural and virtual faces. PLoS ONE 3(11), 1–8 (2008)

    Article  Google Scholar 

  21. Ekdahl, D., Osler, L.: Expressive avatars: vitality in virtual worlds. Philos. Technol. 36(2), 1–28 (2023)

    Article  Google Scholar 

  22. Ekman, P., Friesen, W.V.: Facial action coding system. Environ. Psychol. Nonverbal Behav. (1978)

    Google Scholar 

  23. Elfenbein, H.A., Ambady, N.: On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychol. Bull. 128(2), 203–235 (2002)

    Article  PubMed  Google Scholar 

  24. Fernández-Sotos, P., García, A.S., Vicente-Querol, M.A., Lahera, G., Rodriguez-Jimenez, R., Fernández-Caballero, A.: Validation of dynamic virtual faces for facial affect recognition. PLOS ONE 16(1), e0246001 (2021)

    Article  PubMed  PubMed Central  Google Scholar 

  25. Fischer, A.H., Kret, M.E., Broekens, J.: Gender differences in emotion perception and self-reported emotional intelligence: a test of the emotion sensitivity hypothesis. PLOS ONE 13(1), e0190712 (2018)

    Article  PubMed  PubMed Central  Google Scholar 

  26. Fujimura, T., Umemura, H.: Development and validation of a facial expression database based on the dimensional and categorical model of emotions. Cogn. Emotion 32(8), 1663–1670 (2018)

    Article  Google Scholar 

  27. García, A.S., Fernández-Sotos, P., Vicente-Querol, M.A., Lahera, G., Rodriguez-Jimenez, R., Fernández-Caballero, A.: Design of reliable virtual human facial expressions and validation by healthy people. Integr. Comput.-Aided Eng. 27(3), 287–299 (2020)

    Article  Google Scholar 

  28. Geraets, C.N., et al.: Virtual reality facial emotion recognition in social environments: an eye-tracking study. Internet Interventions 25, 100432 (2021)

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Gonçalves, A.R., Fernandes, C., Pasion, R., Ferreira-Santos, F., Barbosa, F., Marques-Teixeira, J.: Effects of age on the identification of emotions in facial expressions: a metaanalysis. PeerJ 2018(7) (2018)

    Google Scholar 

  30. Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_16

    Chapter  Google Scholar 

  31. Gutiérrez-Maldonado, J., Rus-Calafell, M., González-Conde, J.: Creation of a new set of dynamic virtual reality faces for the assessment and training of facial emotion recognition ability. Virtual Reality 18(1), 61–71 (2014)

    Article  Google Scholar 

  32. van Haeringen, E.S., Veltmeijer, E.A., Gerritsen, C.: Empirical validation of an agent-based model of emotion contagion. IEEE Trans. Affect. Comput. (2023)

    Google Scholar 

  33. Hayes, G.S., et al.: Task characteristics influence facial emotion recognition age-effects: a meta-analytic review. Psychol. Aging 35(2), 295–315 (2020)

    Article  PubMed  Google Scholar 

  34. Hofer, M., Hüsser, A., Prabhu, S.: The effect of an avatar’s emotional expressions on players’ fear reactions: the mediating role of embodiment. Comput. Hum. Behav. 75, 883–890 (2017)

    Article  Google Scholar 

  35. Israelashvili, J., Fischer, A.: Recognition of emotion from verbal and nonverbal expressions and its relation to effective communication: a preliminary evidence of a positive link. J. Intell. 11(1) (2023)

    Google Scholar 

  36. Jack, R.E., Garrod, O.G., Yu, H., Caldara, R., Schyns, P.G.: Facial expressions of emotion are not culturally universal. Proc. Nat. Acad. Sci. United States Am. 109(19), 7241–7244 (2012)

    Article  ADS  CAS  Google Scholar 

  37. Keltner, D., Sauter, D., Tracy, J., Cowen, A.: Emotional expression: advances in basic emotion theory. J. Nonverbal Behav. 43(2), 133–160 (2019)

    Article  PubMed  PubMed Central  Google Scholar 

  38. Krumhuber, E.G., Küster, D., Namba, S., Shah, D., Calvo, M.G.: Emotion recognition from posed and spontaneous dynamic expressions: human observers versus machine analysis. Emotion 21(2), 447–451 (2021)

    Article  PubMed  Google Scholar 

  39. Krumhuber, E.G., Küster, D., Namba, S., Skora, L.: Human and machine validation of 14 databases of dynamic facial expressions. Behav. Res. Methods 53(2), 686–701 (2021)

    Article  PubMed  Google Scholar 

  40. Kyrlitsias, C., Michael-Grigoriou, D.: Social interaction with agents and avatars in immersive virtual environments: a survey. Front. Virtual Reality 2, 786665 (2022)

    Article  Google Scholar 

  41. Lee, J., Kim, S., Kim, S., Park, J., Sohn, K.: Context-aware emotion recognition networks. In" Proceedings of the IEEE International Conference on Computer Vision, vol. 2019-Octob, pp. 10142–10151 (2019)

    Google Scholar 

  42. Leung, F.Y.N., et al.: Emotion recognition across visual and auditory modalities in autism spectrum disorder: a systematic review and meta-analysis. Develop. Rev. 63, 101000 (2022)

    Article  Google Scholar 

  43. Li, S., Deng, W.: Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans. Image Process. 28(1), 356–370 (2019)

    Article  ADS  MathSciNet  CAS  PubMed  Google Scholar 

  44. Luo, C., Song, S., Xie, W., Shen, L., Gunes, H.: Learning multi-dimensional edge feature-based au relation graph for facial action unit recognition. In: Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pp. 1239–1246 (2022)

    Google Scholar 

  45. Manierka, M.S., Rezaei, R., Palacios, S., Haigh, S.M., Hutsler, J.J.: In the mood to be social: Affective state influences facial emotion recognition in healthy adults. Emotion (Washington, D.C.) 21(7), 1576–1581 (2021)

    Google Scholar 

  46. Mao, J., Xu, R., Yin, X., Chang, Y., Nie, B., Huang, A.: POSTER++: a simpler and stronger facial expression recognition network. Preprint (2023)

    Google Scholar 

  47. Marcos-Pablos, S., González-Pablos, E., Martín-Lorenzo, C., Flores, L.A., Gómez-García-Bermejo, J., Zalama, E.: Virtual avatar for emotion recognition in patients with schizophrenia: a pilot study. Front. Human Neurosci. 10, 12 (2016)

    Google Scholar 

  48. Martinez, L., Falvello, V.B., Aviezer, H., Todorov, A.: Contributions of facial expressions and body language to the rapid perception of dynamic emotions. Cogn. Emoti. 30(5), 939–952 (2016)

    Article  Google Scholar 

  49. Matsumoto, D., Willingham, B.: Spontaneous facial expressions of emotion of congenitally and noncongenitally blind individuals. J. Pers. Soc. Psychol. 96(1), 1–10 (2009)

    Article  PubMed  Google Scholar 

  50. Miao, F., Kozlenkova, I.V., Wang, H., Xie, T., Palmatier, R.W.: An emerging theory of avatar marketing. J. Market. 86(1), 67–90 (2022)

    Article  Google Scholar 

  51. Mollahosseini, A., Hasani, B., Mahoor, M.H.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(01), 18–31 (2019)

    Article  Google Scholar 

  52. Monferrer, M., et al.: Facial emotion recognition in patients with depression compared to healthy controls when using human avatars. Sci. Rep. 13(1), 1–10 (2023)

    Article  Google Scholar 

  53. Muros, N.I., et al.: Facial affect recognition by patients with schizophrenia using human avatars. J. Clin. Med. 10(9), 1904 (2021)

    Article  PubMed  PubMed Central  Google Scholar 

  54. O’Rourke, S.R., Branford, K.R., Brooks, T.L., Ives, L.T., Nagendran, A., Compton, S.N.: The emotional and behavioral impact of delivering bad news to virtual versus real standardized patients: a pilot study. Teach. Learn. Med. 32(2), 139–149 (2019)

    Article  PubMed  Google Scholar 

  55. Pham, L., Vu, T.H., Tran, T.A.: Facial expression recognition using residual masking network. In: Proceedings - International Conference on Pattern Recognition, pp. 4513–4519 (2020)

    Google Scholar 

  56. Rafiee, Y., Schacht, A.: Sex differences in emotion recognition: investigating the moderating effects of stimulus features. Cogn. Emot. (2023)

    Google Scholar 

  57. Richoz, A.R., Lao, J., Pascalis, O., Caldara, R.: Tracking the recognition of static and dynamic facial expressions of emotion across the life span. J. Vis. 18(9), 5–5 (2018)

    Article  PubMed  Google Scholar 

  58. Rodger, H., et al.: The recognition of facial expressions of emotion in deaf and hearing individuals. Heliyon 7(5) (2021)

    Google Scholar 

  59. Rymarczyk, K., Zurawski, Ł., Jankowiak-Siuda, K., Szatkowska, I.: Emotional empathy and facial mimicry for static and dynamic facial expressions of fear and disgust. Front. Psychol. 7(NOV), 1853 (2016)

    Google Scholar 

  60. Saquinaula, A., Juarez, A., Geigel, J., Bailey, R., Alm, C.O.: Emotional empathy and facial mimicry of avatar faces. In: Proceedings - 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2022, pp. 770–771 (2022)

    Google Scholar 

  61. Schmid, P.C., Schmid Mast, M.: Mood effects on emotion recognition. Motiv. Emot. 34(3), 288–292 (2010)

    Article  Google Scholar 

  62. Sollfrank, T., et al.: The effects of dynamic and static emotional facial expressions of humans and their avatars on the EEG: an ERP and ERD/ERS study. Front. Neurosci. 15, 651044 (2021)

    Article  PubMed  PubMed Central  Google Scholar 

  63. Suma, T., Sonia, B., Agyemang Baffour, K., Oyekoya, O.: The effects of avatar voice and facial expression intensity on emotional recognition and user perception. In: Proceedings - SIGGRAPH Asia 2023: Technical Communications, SA Technical Communications 2023 (2023)

    Google Scholar 

  64. Sun, Y., Won, A.S.: Despite appearances: comparing emotion recognition in abstract and humanoid avatars using nonverbal behavior in social virtual reality. Front. Virtual Reality 2, 694453 (2021)

    Article  Google Scholar 

  65. Torregrossa, L.J., et al.: Decoupling of spontaneous facial mimicry from emotion recognition in schizophrenia. Psychiatry Res. 275, 169–176 (2019)

    Article  PubMed  PubMed Central  Google Scholar 

  66. Trautmann, S.A., Fehr, T., Herrmann, M.: Emotions in motion: dynamic compared to static facial expressions of disgust and happiness reveal more widespread emotion-specific activations. Brain Res. 1284, 100–115 (2009)

    Article  CAS  PubMed  Google Scholar 

  67. Vicente-Querol, M.Á., García, A.S., Fernández-Sotos, P., Rodriguez-Jimenez, R., Fernández-Caballero, A.: Development and validation of basic virtual human facial emotion expressions. In: Ferrández Vicente, J.M., Álvarez-Sánchez, J.R., de la Paz López, F., Toledo Moreo, J., Adeli, H. (eds.) IWINAC 2019. LNCS, vol. 11486, pp. 222–231. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-19591-5_23

    Chapter  Google Scholar 

  68. Wingenbach, T.S., Ashwin, C., Brosnan, M.: Validation of the Amsterdam dynamic facial expression set - bath intensity variations (ADFES-BIV): a set of videos expressing low, intermediate, and high intensity emotions. PLOS ONE 11(1), e0147112 (2016)

    Article  PubMed  PubMed Central  Google Scholar 

  69. Zane, E., Yang, Z., Pozzan, L., Guha, T., Narayanan, S., Grossman, R.B.: Motion-capture patterns of voluntarily mimicked dynamic facial expressions in children and adolescents with and without ASD. J. Autism Dev. Disorders 49(3), 1062–1079 (2019)

    Article  Google Scholar 

  70. Zhang, J., Chen, Q., Lu, J., Wang, X., Liu, L., Feng, Y.: Emotional expression by artificial intelligence chatbots to improve customer satisfaction: underlying mechanism and boundary conditions. Tourism Manage. 100, 104835 (2024)

    Article  Google Scholar 

  71. Zupan, B., Eskritt, M.: Facial and vocal emotion recognition in adolescence: a systematic review. adolescent research review (2023)

    Google Scholar 

Download references

Acknowledgements

This work is part of the research programme Innovational Research Incentives Scheme Vidi SSH 2017 with project number 016.Vidi.185.178, which is financed by the Dutch Research Council (NWO).

Author information

Authors and Affiliations

  1. Vrije Universiteit, De Boelelaan 1111, 1081HV, Amsterdam, The Netherlands

    Erik van Haeringen, Marco Otte & Charlotte Gerritsen

Authors
  1. Erik van Haeringen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marco Otte
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charlotte Gerritsen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Erik van Haeringen .

Editor information

Editors and Affiliations

  1. Delft University of Technology, Delft, The Netherlands

    Frans A. Oliehoek

  2. Delft University of Technology, Delft, The Netherlands

    Manon Kok

  3. Delft University of Technology, Delft, The Netherlands

    Sicco Verwer

Ethics declarations

Disclosure of Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

van Haeringen, E., Otte, M., Gerritsen, C. (2025). Human Recognition of Emotions Expressed by Human-Like Avatars on 2D Screens. In: Oliehoek, F.A., Kok, M., Verwer, S. (eds) Artificial Intelligence and Machine Learning. BNAIC/Benelearn 2023. Communications in Computer and Information Science, vol 2187. Springer, Cham. https://doi.org/10.1007/978-3-031-74650-5_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-74650-5_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-74649-9

  • Online ISBN: 978-3-031-74650-5

  • eBook Packages: Artificial Intelligence (R0)

Publish with us

Policies and ethics

Search

Navigation