Abstract
The emergence of Artificial Intelligence (AI) marks a significant milestone in innovations, particularly with the advent of Virtual Beings (VBs) and Mixed Reality. VBs have transitioned from rudimentary programmed characters to elaborate, interactive entities capable of sophisticated human engagement. Enhanced with emotional intelligence, adaptive learning, and context-sensitivity, VBs offer nuanced interactions within both digital and real-world settings. A key breakthrough in this field is the development of affective VBs, which possess the ability to comprehend and react to human emotions, challenging the traditional view of AI as emotionless and strictly logical. This evolution prompts a reexamination of AI’s societal role and the dynamics of Human-Computer Interaction. This study focuses on the complexities of VBs, particularly through the implementation of a virtual being named PLEA, manifested in both worlds: the virtual and the physical one through a robotic head. It discusses the utility of such agents in various applications and employs ethnographic communication methodologies for data collection and analysis to unearth interaction patterns. Additionally, it examines human reactions to PLEA through a user-centered design approach, highlighting interactions based solely on facial expressions between PLEA and human participants. This investigation aims to lay the groundwork for developing multidisciplinary methods to collect, analyze, and abstract data from real-time interactions and feedback sessions, advancing the discourse on AI’s integration into human social environments.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Genay, A., Lecuyer, A., Hachet, M.: Being an avatar “for real”: a survey on virtual embodiment in augmented reality. IEEE Trans. Visual Comput. Graph. 28(12), 5071–5090 (2021). https://doi.org/10.1109/TVCG.2021.3099290
Paiva, A., Leite, I., Boukricha, H., Wachsmuth, I.: Empathy in virtual agents and robots: a survey. ACM Trans. Interact. Intell. Syst. 7(3), 1–40 (2017). https://doi.org/10.1145/2912150
Azarnov, D.A., Chubarov, A.A., Samsonovich, A.V.: Virtual actor with social-emotional intelligence. Procedia Comput. Sci. 123, 76–85 (2018). https://doi.org/10.1016/j.procs.2018.01.013
Hendrikse, S.C., Kluiver, S., Treur, J., Wilderjans, T.F., Dikker, S., Koole, S.L.: How virtual agents can learn to synchronize: an adaptive joint decision-making model of psychotherapy. Cogn. Syst. Res. 79, 138–215 (2023). https://doi.org/10.1016/j.cogsys.2022.12.009
Koren, L., Stipancic, T., Ricko, A., Benic, J.: Context-driven method in realization of optimized human-robot interaction. Tehnicki Glasnik 16(3), 320–327 (2022). https://doi.org/10.31803/tg-20220504100707
Koren, L., Stipancic, T., Ricko, A., Orsag, L.: Person localization model based on a fusion of acoustic and visual inputs. Electronics 11(3), 440 (2022). https://doi.org/10.3390/electronics11030440
Park, S., Whang, M.: Empathy in human-robot interaction: designing for social robots. Int. J. Environ. Res. Public Health 19(3), 1889 (2022). https://doi.org/10.3390/ijerph19031889
Fruchter, R., Nishida, T., Rosenberg, D.: Social intelligence design for social computing. In: International Conference on Human-Computer Interaction, pp. 545–558. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-05061-9_38
Mills, K.A., Scholes, L., Brown, A.: Virtual reality and embodiment in multimodal meaning making. Writ. Commun. 39(3), 335–369 (2022). https://doi.org/10.1177/07410883221083517
Genay, A., Lécuyer, A., Hachet, M.: Being an avatar “for real”: a survey on virtual embodiment in augmented reality. IEEE Trans. Visual Comput. Graph. 28(12), 5071–5090 (2021). https://doi.org/10.1109/tvcg.2021.3099290
Stipancic, T., Koren, L., Korade, D., Rosenberg, D.: PLEA: a social robot with teaching and interacting capabilities. J. Pacific Rim Psychol. 15 (2021). https://doi.org/10.1177/18344909211037019
Rosenberg, E.L., Ekman, P. (eds.): What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press(2020). https://doi.org/10.1093/acprof:oso/9780195179644.001.0001
Tarnowski, P., Kolodziej, M., Majkowski, A., Rak, R.J.: Emotion recognition using facial expressions. In: International Conference on Computational Science (2017). https://doi.org/10.1016/j.procs.2017.05.025
Melzer, A., Shafir, T., Tsachor, R.P.: How do we recognize emotion from movement? Specific motor components contribute to the recognition of each emotion. Front. Psychol. 10 (2019). https://doi.org/10.3389/fpsyg.2019.01389
Poria, S., Chaturvedi, I., Cambria, E., Hussain, A.: Convolutional MKL based multimodal emotion recognition and sentiment analysis. In: 16th IEEE International Conference on Data Mining (ICDM), Barcelona (2016). https://doi.org/10.1109/ICDM.2016.0055
Savchenko, A.V.: Facial expression and attributes recognition basedon multi-task learning of lightweight neuralnetworks. In: IEEE 19th International Symposium on Intelligent Systems and Informatics, Subotica (2021). https://doi.org/10.1109/SISY52375.2021.9582508
Koren, L., Stipancic, T., Ricko, A., Orsag, L.: Multimodal emotion analysis based on visual, acoustic and linguistic features. In: Meiselwitz, G. (ed.) Social Computing and Social Media: Design, User Experience and Impact (HCII 2022). LNCS, vol. 13315. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-05061-9_23
Bavelas, J.B., Gerwing, J.: The listener as addressee in face-to-face dialogue. Int. J. Listen. 25(3), 178–198 (2011). https://doi.org/10.1080/10904018.2010.508675
Rawal, N., Stock-Homburg, R.M.: Facial emotion expressions in human–robot interaction: a survey. Int. J. Soc. Robot. 14(7), 1583–1604 (2022). https://arxiv.org/abs/2103.07169
El Kamali, M., et al.: NESTORE: mobile chatbot and tangible vocal assistant to support older adults’ wellbeing. In: Proceedings of the 2nd Conference on Conversational User Interfaces, pp. 1–3 (2020). https://doi.org/10.1145/3405755.3406167
Scoglio, A.A., Reilly, E.D., Gorman, J.A., Drebing, C.E.: Use of social robots in mental health and well-being research: systematic review. J. Med. Internet Res. 21(7) (2019). https://doi.org/10.2196/13322
Dimberg, U., Andréasson, P., Thunberg, M.: Emotional empathy and facial reactions to facial expressions. J. Psychophysiol. 25(1), 26 (2011). https://doi.org/10.1027/0269-8803/a000029
Adriana, T., Matarić, M.J.: Emulating empathy in socially assistive robotics. In: Proceedings of the AAAI Spring Symposium: Multidisciplinary Collaboration for Socially Assistive Robotics, pp. 93–96 (2007)
Reeves, B., Nass, C.I.: The media equation: how people treat computers, television, and new media like real people and places. In: Center for the Study of Language and Information. Cambridge University Press (1996). https://doi.org/10.1016/s0898-1221(97)82929-x
Złotowski, J., et al.: Model of dual anthropomorphism: the relationship between the media equation effect and implicit anthropomorphism. Int. J. Soc. Robot. 10(5), 701–714 (2018). https://doi.org/10.1007/s12369-018-0476-5
Reuten, A., Van Dam, M., Naber, M.: Pupillary responses to robotic and human emotions: the uncanny valley and media equation confirmed. Front. Psychol. 9, 774 (2018). https://doi.org/10.3389/fpsyg.2018.00774
Klowait, N.: The quest for appropriate models of human-likeness: anthropomorphism in media equation research. AI Soc. 33(4), 527–536 (2018). https://doi.org/10.1007/s00146-017-0746-z
Clifford, N., Corina, Y.: The man who lied to his laptop: what machines teach us about human relationships. Current/Penguin (2010). https://doi.org/10.5860/choice.48-3960
Tudor Car, L., et al.: Conversational agents in health care: scoping review and conceptual analysis. J. Med. Internet Res. 22(8), e17158 (2020). https://doi.org/10.2196/17158
Shvo, M., Buhmann, J., Kapadia, M.: An interdependent model of personality, motivation, emotion, and mood for intelligent virtual agents. In: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, pp. 65–72 (2019). https://doi.org/10.1145/3308532.3329474
Kramer, N., Manzeschke, A.: Social reactions to socially interactive agents and their ethical implications. In: The Handbook on Socially Interactive Agents: 20 years of Research on Embodied Conversational Agents, Intelligent Virtual Agents, and Social Robotics Volume 1: Methods, Behavior, Cognition, pp. 77–104 (2021). https://doi.org/10.1145/3477322.3477326
Canal, F.Z., et al.: A survey on facial emotion recognition techniques: a state-of-the-art literature review. Inf. Sci. 582, 593–617 (2022). https://doi.org/10.1016/j.ins.2021.10.005
Stipancic, T., Jerbic, B., Curkovic, P.: Bayesian Approach to Robot Group Control. LNEE, vol. 130, pp. 109–119. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-2317-1_9
Acknowledgments
This work has been supported in part by the Croatian Science Foundation under the project “Affective Multimodal Interaction based on Constructed Robot Cognition—AMICORC (UIP-2020-02-7184)”. Special thanks to iCOM ICT Research (https://icomict.org), Art AI Festival, Leicester (https://www.art-ai.io), and to students and colleagues of the LAPIS research group and FAMENA UNIZG for their active support and participation in this research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Stipancic, T., Koren, L., Rosenberg, D., Harwood, T., Benic, J. (2024). PLEA: The Embodied Virtual Being. In: Coman, A., Vasilache, S. (eds) Social Computing and Social Media. HCII 2024. Lecture Notes in Computer Science, vol 14703. Springer, Cham. https://doi.org/10.1007/978-3-031-61281-7_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-61281-7_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-61280-0
Online ISBN: 978-3-031-61281-7
eBook Packages: Computer ScienceComputer Science (R0)