Abstract
The Metaverse is defined as an interconnected network of 3D environments that enable multisensory interactions with virtual environments, digital objects, and people. A central component of a Metaverse is the realism of human interaction with the environment and each other, which is achieved through emotional expression. However, there are limitations in realistically representing human emotions and facial expressions in real time through virtual avatars. In this paper, the authors present a research project called Meta-EmoVis aimed at improving the realism of the Metaverse by detecting human facial expressions and translating them into emotional feedback that influences aspects of the virtual avatar and environment. The Emoj tool is used to collect data and map facial expressions to basic emotions, as well as detect the human engagement level. The goal is to create a world of real social interaction and emotional sharing in virtual spaces that are sensitive and adaptive to human emotional and mental states. Preliminary user tests to evaluate the effectiveness of the application are presented.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mystakidis, S.: Metaverse. Encyclopedia 2(1), 486–497 (2022)
Dionisio, J.D.N., Burn, W.G., Gilbert, R.: 3D virtual worlds and the metaverse: current status and future possibilities. ACM Comput. Surv. (CSUR) 45(3), 1–38 (2013)
McCquiggan, S.W., et al.: Modeling parallel and reactive empathy in virtual agents: an inductive approach. In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, vol. 1, pp. 167–174 (2008)
Ochs, M., Pelachaud, C., Sadek, D.: An empathic virtual dialog agent to improve human-machine interaction. In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, vol. 1, pp. 89–96 (2008)
Boukricha, H., Wachsmuth, I.: Mechanism, modulation, and expression of empathy in a virtual human. In: 2011 IEEE Workshop on Affective Computational Intelligence (WACI), pp. 1–8. IEEE (2011)
Darwin, C.: The Expression of the Emotions in Man and Animals. The University of Chicago Press (1969)
Ekman, P., Friesen, W.V., Ellsworth, P.: Emotion in the Human Face: Guidelines for Research and an Integration of Findings. Pergamon Press Inc. (1972)
https://www.emojlab.com/. Accessed 15 Feb 2023
Dincelli, E., Yayla, A.: Immersive virtual reality in the age of the Metaverse: a hybrid-narrative review based on the technology affordance perspective. J. Strateg. Inf. Syst. 31(2), 101717 (2022)
Baker-Brunnauer, J.: Ethical Challenges for the Metaverse Development (2022)
Slater, M., et al.: Visual realism enhances realistic response in an immersive virtual environment. IEEE Comput. Graphics Appl. 29(3), 76–84 (2009)
Bailenson, J.N., et al.: Real-time classification of evoked emotions using facial feature tracking and physiological responses. Int. J. Hum Comput Stud. 66(5), 303–317 (2008)
Borth, D., et al.: Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: Proceedings of the 21st ACM international conference on Multimedia, pp. 223–232 (2013)
Ekman, P., et al.: Basic Emotions. Handbook of Cognition and Emotion, vol. 98(45–60), p. 16 (1999)
Russell, J.A.: Core affect and the psychological construction of emotion. Psychol. Rev. 110(1), 145 (2003)
Hanada, M.: Correspondence analysis of color–emotion associations. Color. Res. Appl. 43(2), 224–237 (2018)
Naz, K.A.Y.A., Helen, H.: Color-emotion associations: past experience and personal preference. In: AIC 2004 Color and Paints, Interim Meeting of the International Color Association, Proceedings, Jose Luis Caivano, vol. 31 (2004)
Takahshi, F., Kawabata, Y.: The association between colors and emotions for emotional words and facial expressions. Color. Res. Appl. 43(2), 247–257 (2018)
Kralj Novak, P., et al.: Sentiment of emojis. PloS One 10(12), e0144296 (2015)
Ekmann, P.: Universal facial expressions in emotion. Stud. Psychol. 15(2), 140 (1973)
Acknowledgement
the authors want to thank Siyu Hu and Weiyu Li, student of the Virtual and Physical Prototyping course, School of Design, Academic Year 2020/2021, for their contribution to the project.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Spadoni, E., Carulli, M., Mengoni, M., Luciani, M., Bordegoni, M. (2023). Empowering Virtual Humans’ Emotional Expression in the Metaverse. In: Antona, M., Stephanidis, C. (eds) Universal Access in Human-Computer Interaction. HCII 2023. Lecture Notes in Computer Science, vol 14021. Springer, Cham. https://doi.org/10.1007/978-3-031-35897-5_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-35897-5_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-35896-8
Online ISBN: 978-3-031-35897-5
eBook Packages: Computer ScienceComputer Science (R0)