Skip to main content

PLEA: The Embodied Virtual Being

  • Conference paper
  • First Online:
Social Computing and Social Media (HCII 2024)

Abstract

The emergence of Artificial Intelligence (AI) marks a significant milestone in innovations, particularly with the advent of Virtual Beings (VBs) and Mixed Reality. VBs have transitioned from rudimentary programmed characters to elaborate, interactive entities capable of sophisticated human engagement. Enhanced with emotional intelligence, adaptive learning, and context-sensitivity, VBs offer nuanced interactions within both digital and real-world settings. A key breakthrough in this field is the development of affective VBs, which possess the ability to comprehend and react to human emotions, challenging the traditional view of AI as emotionless and strictly logical. This evolution prompts a reexamination of AI’s societal role and the dynamics of Human-Computer Interaction. This study focuses on the complexities of VBs, particularly through the implementation of a virtual being named PLEA, manifested in both worlds: the virtual and the physical one through a robotic head. It discusses the utility of such agents in various applications and employs ethnographic communication methodologies for data collection and analysis to unearth interaction patterns. Additionally, it examines human reactions to PLEA through a user-centered design approach, highlighting interactions based solely on facial expressions between PLEA and human participants. This investigation aims to lay the groundwork for developing multidisciplinary methods to collect, analyze, and abstract data from real-time interactions and feedback sessions, advancing the discourse on AI’s integration into human social environments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Genay, A., Lecuyer, A., Hachet, M.: Being an avatar “for real”: a survey on virtual embodiment in augmented reality. IEEE Trans. Visual Comput. Graph. 28(12), 5071–5090 (2021). https://doi.org/10.1109/TVCG.2021.3099290

    Article  Google Scholar 

  2. Paiva, A., Leite, I., Boukricha, H., Wachsmuth, I.: Empathy in virtual agents and robots: a survey. ACM Trans. Interact. Intell. Syst. 7(3), 1–40 (2017). https://doi.org/10.1145/2912150

    Article  Google Scholar 

  3. Azarnov, D.A., Chubarov, A.A., Samsonovich, A.V.: Virtual actor with social-emotional intelligence. Procedia Comput. Sci. 123, 76–85 (2018). https://doi.org/10.1016/j.procs.2018.01.013

    Article  Google Scholar 

  4. Hendrikse, S.C., Kluiver, S., Treur, J., Wilderjans, T.F., Dikker, S., Koole, S.L.: How virtual agents can learn to synchronize: an adaptive joint decision-making model of psychotherapy. Cogn. Syst. Res. 79, 138–215 (2023). https://doi.org/10.1016/j.cogsys.2022.12.009

    Article  Google Scholar 

  5. Koren, L., Stipancic, T., Ricko, A., Benic, J.: Context-driven method in realization of optimized human-robot interaction. Tehnicki Glasnik 16(3), 320–327 (2022). https://doi.org/10.31803/tg-20220504100707

  6. Koren, L., Stipancic, T., Ricko, A., Orsag, L.: Person localization model based on a fusion of acoustic and visual inputs. Electronics 11(3), 440 (2022). https://doi.org/10.3390/electronics11030440

    Article  Google Scholar 

  7. Park, S., Whang, M.: Empathy in human-robot interaction: designing for social robots. Int. J. Environ. Res. Public Health 19(3), 1889 (2022). https://doi.org/10.3390/ijerph19031889

    Article  Google Scholar 

  8. Fruchter, R., Nishida, T., Rosenberg, D.: Social intelligence design for social computing. In: International Conference on Human-Computer Interaction, pp. 545–558. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-05061-9_38

  9. Mills, K.A., Scholes, L., Brown, A.: Virtual reality and embodiment in multimodal meaning making. Writ. Commun. 39(3), 335–369 (2022). https://doi.org/10.1177/07410883221083517

    Article  Google Scholar 

  10. Genay, A., Lécuyer, A., Hachet, M.: Being an avatar “for real”: a survey on virtual embodiment in augmented reality. IEEE Trans. Visual Comput. Graph. 28(12), 5071–5090 (2021). https://doi.org/10.1109/tvcg.2021.3099290

    Article  Google Scholar 

  11. Stipancic, T., Koren, L., Korade, D., Rosenberg, D.: PLEA: a social robot with teaching and interacting capabilities. J. Pacific Rim Psychol. 15 (2021). https://doi.org/10.1177/18344909211037019

  12. Rosenberg, E.L., Ekman, P. (eds.): What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press(2020). https://doi.org/10.1093/acprof:oso/9780195179644.001.0001

  13. Tarnowski, P., Kolodziej, M., Majkowski, A., Rak, R.J.: Emotion recognition using facial expressions. In: International Conference on Computational Science (2017). https://doi.org/10.1016/j.procs.2017.05.025

  14. Melzer, A., Shafir, T., Tsachor, R.P.: How do we recognize emotion from movement? Specific motor components contribute to the recognition of each emotion. Front. Psychol. 10 (2019). https://doi.org/10.3389/fpsyg.2019.01389

  15. Poria, S., Chaturvedi, I., Cambria, E., Hussain, A.: Convolutional MKL based multimodal emotion recognition and sentiment analysis. In: 16th IEEE International Conference on Data Mining (ICDM), Barcelona (2016). https://doi.org/10.1109/ICDM.2016.0055

  16. Savchenko, A.V.: Facial expression and attributes recognition basedon multi-task learning of lightweight neuralnetworks. In: IEEE 19th International Symposium on Intelligent Systems and Informatics, Subotica (2021). https://doi.org/10.1109/SISY52375.2021.9582508

  17. Koren, L., Stipancic, T., Ricko, A., Orsag, L.: Multimodal emotion analysis based on visual, acoustic and linguistic features. In: Meiselwitz, G. (ed.) Social Computing and Social Media: Design, User Experience and Impact (HCII 2022). LNCS, vol. 13315. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-05061-9_23

  18. Bavelas, J.B., Gerwing, J.: The listener as addressee in face-to-face dialogue. Int. J. Listen. 25(3), 178–198 (2011). https://doi.org/10.1080/10904018.2010.508675

    Article  Google Scholar 

  19. Rawal, N., Stock-Homburg, R.M.: Facial emotion expressions in human–robot interaction: a survey. Int. J. Soc. Robot. 14(7), 1583–1604 (2022). https://arxiv.org/abs/2103.07169

  20. El Kamali, M., et al.: NESTORE: mobile chatbot and tangible vocal assistant to support older adults’ wellbeing. In: Proceedings of the 2nd Conference on Conversational User Interfaces, pp. 1–3 (2020). https://doi.org/10.1145/3405755.3406167

  21. Scoglio, A.A., Reilly, E.D., Gorman, J.A., Drebing, C.E.: Use of social robots in mental health and well-being research: systematic review. J. Med. Internet Res. 21(7) (2019). https://doi.org/10.2196/13322

  22. Dimberg, U., Andréasson, P., Thunberg, M.: Emotional empathy and facial reactions to facial expressions. J. Psychophysiol. 25(1), 26 (2011). https://doi.org/10.1027/0269-8803/a000029

    Article  Google Scholar 

  23. Adriana, T., Matarić, M.J.: Emulating empathy in socially assistive robotics. In: Proceedings of the AAAI Spring Symposium: Multidisciplinary Collaboration for Socially Assistive Robotics, pp. 93–96 (2007)

    Google Scholar 

  24. Reeves, B., Nass, C.I.: The media equation: how people treat computers, television, and new media like real people and places. In: Center for the Study of Language and Information. Cambridge University Press (1996). https://doi.org/10.1016/s0898-1221(97)82929-x

  25. Złotowski, J., et al.: Model of dual anthropomorphism: the relationship between the media equation effect and implicit anthropomorphism. Int. J. Soc. Robot. 10(5), 701–714 (2018). https://doi.org/10.1007/s12369-018-0476-5

  26. Reuten, A., Van Dam, M., Naber, M.: Pupillary responses to robotic and human emotions: the uncanny valley and media equation confirmed. Front. Psychol. 9, 774 (2018). https://doi.org/10.3389/fpsyg.2018.00774

    Article  Google Scholar 

  27. Klowait, N.: The quest for appropriate models of human-likeness: anthropomorphism in media equation research. AI Soc. 33(4), 527–536 (2018). https://doi.org/10.1007/s00146-017-0746-z

    Article  Google Scholar 

  28. Clifford, N., Corina, Y.: The man who lied to his laptop: what machines teach us about human relationships. Current/Penguin (2010). https://doi.org/10.5860/choice.48-3960

    Article  Google Scholar 

  29. Tudor Car, L., et al.: Conversational agents in health care: scoping review and conceptual analysis. J. Med. Internet Res. 22(8), e17158 (2020). https://doi.org/10.2196/17158

  30. Shvo, M., Buhmann, J., Kapadia, M.: An interdependent model of personality, motivation, emotion, and mood for intelligent virtual agents. In: Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, pp. 65–72 (2019). https://doi.org/10.1145/3308532.3329474

  31. Kramer, N., Manzeschke, A.: Social reactions to socially interactive agents and their ethical implications. In: The Handbook on Socially Interactive Agents: 20 years of Research on Embodied Conversational Agents, Intelligent Virtual Agents, and Social Robotics Volume 1: Methods, Behavior, Cognition, pp. 77–104 (2021). https://doi.org/10.1145/3477322.3477326

  32. Canal, F.Z., et al.: A survey on facial emotion recognition techniques: a state-of-the-art literature review. Inf. Sci. 582, 593–617 (2022). https://doi.org/10.1016/j.ins.2021.10.005

    Article  Google Scholar 

  33. Stipancic, T., Jerbic, B., Curkovic, P.: Bayesian Approach to Robot Group Control. LNEE, vol. 130, pp. 109–119. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-2317-1_9

Download references

Acknowledgments

This work has been supported in part by the Croatian Science Foundation under the project “Affective Multimodal Interaction based on Constructed Robot Cognition—AMICORC (UIP-2020-02-7184)”. Special thanks to iCOM ICT Research (https://icomict.org), Art AI Festival, Leicester (https://www.art-ai.io), and to students and colleagues of the LAPIS research group and FAMENA UNIZG for their active support and participation in this research.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Tomislav Stipancic .

Editor information

Editors and Affiliations

Ethics declarations

Disclosure of Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Stipancic, T., Koren, L., Rosenberg, D., Harwood, T., Benic, J. (2024). PLEA: The Embodied Virtual Being. In: Coman, A., Vasilache, S. (eds) Social Computing and Social Media. HCII 2024. Lecture Notes in Computer Science, vol 14703. Springer, Cham. https://doi.org/10.1007/978-3-031-61281-7_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-61281-7_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-61280-0

  • Online ISBN: 978-3-031-61281-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics