Skip to main content
SpringerLink
Log in

Virtual intimacy in human-embodied conversational agent interactions: the influence of multimodality on its perception

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Interacting with an embodied conversational agent (ECA) in a professional context addresses social considerations to satisfy customer-relationship. This paper presents an experimental study about the perception of virtual intimacy in human-ECA interactions. We explore how an ECA’s multimodal communication affects our perception of virtual intimacy. To this end, we developed a virtual Tourism Information counselor capable of exhibiting verbal and nonverbal intimate behaviors according to several modalities (voice, chatbox, both media), and we built a corpus of videos showing interactions between the agent and a human tourist. We interrogated observers about their perception of the agent’s level of intimacy. Our results confirm the human ability to perceive intimacy in an ECA displaying multimodal behaviors, although the contribution of nonverbal communication remains unclear. Our study suggests that using voice channel increases the perception of virtual intimacy and offers further evidence that human-inspired design of ECAs is needed. Finally, we demonstrate that intimate cues do not disturb the comprehension of task-related information and are valuable for an attentional focus on the agent’s animation. We discuss the concept of virtual intimacy in relation to interpersonal intimacy, and we question its perception in terms of attentional mechanisms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Humans are basically sensitive to eye gaze in others [36], and in the present study, the amplitude of the agent’s eye-moves allowed the external observers to detect whether the agent’s gazed at the interlocutor or not. In non intimate condition, the agent gazed at the tourist at least once when she was talking and when she was listening, that was around 50% of the time.

  2. “I am Léa”.

  3. “Personally, I feel that the Loire’s sides are an ideal place for a wild lunch”.

  4. “You should stop there by the way”.

  5. “I promise you’ll have an incredible look on the river”.

  6. I, Me....

  7. “You are right”.

  8. https://app.prolific.ac.

  9. https://www.facebook.com/.

  10. https://www.mturk.com/.

References

  1. Anderson JR (1985) Cognitive psychology and its implications. WH Freeman/Times Books/Henry Holt & Co, New York

    Google Scholar 

  2. Argyle M, Dean J (1965) Eye-contact, distance and affiliation. Sociometry 28(3):289–304

    Google Scholar 

  3. Bartneck C, Reichenbach J, Breemen vA (2004) In your face, robot! the influence of a characters embodiment on how users perceive its emotional expressions. In: Proceedings of the design and emotion. pp 32–51

  4. Beale R, Creed C (2009) Affective interaction: how emotional agents affect users. Int J Hum Comput Stud 67(9):755–776

    Google Scholar 

  5. Bente G, Kramer NC, Petersen A, de Ruiter JP (2001) Computer animated movement and person perception: methodological advances in nonverbal behavior research. J Nonverbal Behav 25(3):151–166

    Google Scholar 

  6. Bickmore T, Cassell J (2001) Relational agents: a model and implementation of building user trust. In: Proceedings of the sigchi conference on human factors in computing systems. pp. 396–403

  7. Bickmore T, Gruber A, Picard R (2005) Establishing the computer-patient working alliance in automated health behavior change interventions. Patient Educ Couns 59(1):21–30

    Google Scholar 

  8. Bickmore T, Schulman D, Yin L (2009) Engagement vs. deceit: virtual humans with human autobiographies. In: International conference on intelligent virtual agents. pp 6–19

  9. Buisine S (2007) Conception et evaluation d’agents conversationnels multimodaux bidirectionnels (Unpublished doctoral dissertation). Atelier national de Reproduction des Theses

  10. Cerekovic A, Aran O, Gatica-Perez D (2016) Rapport with virtual agents: what do human social cues and personality explain? IEEE Trans Affect Comput 8(3):382–395

    Google Scholar 

  11. Cerezo E, Baldassarri S, Seron F (2007) Interactive agents for multimodal emotional user interaction. In: Proceedings of ofiadis international conference interfaces and human computer interaction. pp 35–42

  12. Clavel C, Plessier J, Martin J-C, Ach L, Morel B (2009) Combining facial and postural expressions of emotions in a virtual character. In: International conference on intelligent virtual agents. pp. 287–300

  13. Constant E, Vallet F, Nandrino J-L, Christophe V (2016) Personal assessment of intimacy in relationships: validity and measurement invariance across gender. Eur Rev Appl Psychol 66(3):109–116

    Google Scholar 

  14. Courgeon M, Clavel C, Tan N, Martin J-C (2011) Front view vs. side view of facial and postural expressions of emotions in a virtual character. In: Transactions on edutainment VI. Springer, Berlin, pp 132–143

  15. de Borst AW, de Gelder B (2015) Is it the real deal? perception of virtual characters versus humans: an affective cognitive neuroscience perspective. Front Psychol 6:576

    Google Scholar 

  16. de Borst AW, Sanchez-Vives MV, Slater M, de Gelder B (2018) First person experience of threat modulates cortical network encoding human peripersonal space. bioRxiv, 314971

  17. Decety J, Jackson PL (2004) The functional architecture of human empathy. Behav Cogn Neurosci Rev 3(2):71–100

    Google Scholar 

  18. DeVault D, Artstein R, Benn G, Dey T, Fast E, Gainer A et al (2014) Simsensei kiosk: a virtual human interviewer for healthcare decision support. In: Proceedings of the 2014 international conference on autonomous agents and multi-agent systems. pp 1061–1068

  19. Ekman P, Friesen WV (2003) Unmasking the face: a guide to recognizing emotions from facial clues. Malor Books, Ishk

  20. Flanagan JR, Johansson RS (2003) Action plans used in action observation. Nature 424(6950):769

    Google Scholar 

  21. Gratch J, Wang N, Gerten J, Fast E, Duffy R (2007) Creating rapport with virtual agents. In: International workshop on intelligent virtual agents. pp 125–138

  22. Gunkel DJ (2012) Communication and artificial intelligence: opportunities and challenges for the 21st century. Communication+1 1(1):1–25

    Google Scholar 

  23. Hinde RA (1981) The bases of a science of interpersonal relationships. Pers Relationsh 1:1–22

    Google Scholar 

  24. Hojat M (2007) Empathy inpatient care: antecedents, development, measurement, and outcomes. Springer, Berlin

    Google Scholar 

  25. Kang S-H, Gratch J, Sidner C, Artstein R, Huang L, Morency L-P (2012) Towards building a virtual counselor: modeling nonverbal behavior during intimate self-disclosure. In: Proceedings of the 11th international conference on autonomous agents and multiagent systems-volume 1. pp 63–70

  26. Kenrick DT, Griskevicius V, Neuberg SL, Schaller M (2010) Renovating the pyramid of needs: contemporary extensions built upon ancient foundations. Perspect Psychol Sci 5(3):292–314

    Google Scholar 

  27. Klein JT, Shepherd SV (2009) Social attention and the brain. Curr Biol 19:958–962

    Google Scholar 

  28. Kramer NC, Lucas G, Schmitt L, Gratch J (2018) Social snacking with a virtual agent-on the interrelation of need to belong and effects of social responsiveness when interacting with artificial entities. Int J Hum Comput Stud 109:112–121

    Google Scholar 

  29. Kunter M, Frenzel A, Nagy G, Baumert J, Pekrun R (2011) Teacher enthusiasm: dimensionality and context specificity. Contemp Educ Psychol 36(4):289–301

    Google Scholar 

  30. Laurenceau J-P, Barrett LF, Rovine MJ (2005) The interpersonal process model of intimacy in marriage: a daily-diary and multilevel modeling approach. J Offamily Psychol 19(2):314

    Google Scholar 

  31. Leonard RC, Knott LE, Lee EB, Singh S, Smith AH, Kanter J, Wetterneck CT (2014) The development of the functional analytic psychotherapy intimacy scale. Psychol Rec 64(4):647657

    Google Scholar 

  32. Liang T-P, Li Y-W, Turban E (2009) Personalized services as empathic responses: the role of intimacy. In: PACIS 2009 Proceedings. 73

  33. Liew TW, Zin NAM, Sahari N (2017) Exploring the affective, motivational and cognitive effects of pedagogical agent enthusiasm in a multimedia learning environment. Hum Centric Comput Inf Sci 7:9

    Google Scholar 

  34. Lisetti C, Amini R, Yasavur U, Rishe N (2013) I can help you change! an empathic virtual agent delivers behavior change health interventions. ACM Trans Manag Inf Syst (TMIS) 4(4):19

    Google Scholar 

  35. Lomanowska AM, Guitton MJ (2016) Online intimacy and well-being in the digital age. Internet Interv 4:138–144

    Google Scholar 

  36. Macrae CN, Hood BM, Milne AB, Rowe AC, Mason MF (2002) Are you looking at me? eye gaze and person perception. Psychol Sci 13(5):460–464

    Google Scholar 

  37. Matsuyama Y, Bhardwaj A, Zhao R, Romeo O, Akoju S, Cassell J (2016) Socially-aware animated intelligent personal assistant agent. In: Proceedings of the 17th annual meeting of the special interest group on discourse and dialogue. pp 224–227

  38. Mendelson M, Aboud F (2012) Mcgill friendship questionnairerespondents affection (MFQ-RA). Measur Instrum Database Soc Sci 4:1–20

    Google Scholar 

  39. Miller R (2015) Intimate relationships, 7th edn. McGraw-Hill Education, New York

    Google Scholar 

  40. Moon Y (1998) Intimate self-disclosure exhanges: using computers to build reciprocal relationships with consumers. Division of Research, Harvard Business School, Boston

    Google Scholar 

  41. Mori M (1970) Bukimi no tani [the uncanny valley]. Energy 7:33–35

    Google Scholar 

  42. Morton TL (1978) Intimacy and reciprocity of exchange: a comparison of spouses and strangers. J Personal Soc Psychol 36(1):72

    Google Scholar 

  43. Moser E, Derntl B, Robinson S, Fink B, Gur RC, Grammer K (2007) Amygdala activation at 3T in response to human and avatar facial expressions of emotions. J Neurosci Methods 161(1):126–133

    Google Scholar 

  44. Muhlberger A, Wieser MJ, Herrmann MJ, Weyers P, Troger C, Pauli P (2009) Early cortical processing of natural and artificial emotional faces differs between lower and higher socially anxious persons. J Neural Transm 116(6):735–746

    Google Scholar 

  45. Ochs M, Ding Y, Fourati N, Chollet M, Ravenet B, Pecune F, Pelachaud C (2014) Vers des agents con-versationnels animes dotes d’emotions et d’attitudes sociales. J d’Interact Pers Syst (JIPS) 3(2):1

    Google Scholar 

  46. Okun MA, Shepard SA, Eisenberg N (2000) The relations of emotionality and regulation to dispositional empathy-related responding among volunteers-in-training. Personal Individ Differ 28(2):367382

    Google Scholar 

  47. Park S (2015) The effects of social cue principles on cognitive load, situational interest, motivation, and achievement in pedagogical agent multimedia learning. J Educ Technol Soc 18(4):211–229

    Google Scholar 

  48. Picard RW (1999) Affective computing for HCI. In: Proceedings HCI international, vol 1, pp 829–833

  49. Potdevin D, Clavel C, Sabouret N (2018) Virtual intimacy, this little something between us: a study about human perception of intimate behaviors in embodied conversational agents. In: Proceedings of the 18th international conference on intelligent virtual agents, pp 165–172

  50. Prager KJ (1995) Guilford series on personal relationships. The psychology of intimacy. Guilford Press, New York

    Google Scholar 

  51. Prager KJ (1997) The psychology ofintimacy. Guilford Press, New York

    Google Scholar 

  52. Prager KJ (2000) Intimacy in personal relationships. In: Hendrick C, Hendrick CC (eds) Close relationships: a sourcebook. Sage Publications, Inc., pp 229–242

  53. Ravenet B, Bevacqua E, Cafaro A, Ochs M, Pelachaud C (2016) Perceiving attitudes expressed through nonverbal behaviors in immersive virtual environments. In: Proceedings of the 9th international conference on motion in games. pp 175–180

  54. Ravenet B, Ochs M, Pelachaud C (2013). From a user-created corpus of virtual agents non-verbal behavior to a computational model of interpersonal attitudes. In: International conference on intelligent virtual agents. pp 263–274

  55. Register LM, Henley TB (1992) The phenomenology of intimacy. J Soc Pers Relationsh 9(4):467–481

    Google Scholar 

  56. Reis HT, Shaver P et al (1988) Intimacy as an interpersonal process. Handb Pers Relationsh 24(3):367–389

    Google Scholar 

  57. Reis HT, Patrick BC (1996) Attachment and intimacy: component processes. In: Handbook of basic principles, social psychology, pp 523–563

  58. Ruijten PA, Midden CJ, Ham J (2016) Ambiguous agents: the influence of consistency of an artificial agents social cues on emotion recognition, recall, and persuasiveness. Int J Hum Comput Interact 32(9):734–744

    Google Scholar 

  59. Sabbadini A, Kogan I, Golinelli P (2018) Psychoanalytic perspectives on virtual intimacy and communication in film. Routledge, Abingdon

    Google Scholar 

  60. Schaefer MT, Olson DH (1981) Assessing intimacy: the pair inventory. J Marital Fam Therapy 7(1):47–60

    Google Scholar 

  61. Slater M, Antley A, Davison A, Swapp D, Guger C, Barker C, Sanchez-Vives MV (2006) A virtual reprise of the stanley milgram obedience experiments. PLoS ONE 1(1):e39

    Google Scholar 

  62. Stern BB (1997) Advertising intimacy: relationship marketing and the services consumer. J Advert 26(4):7–19

    Google Scholar 

  63. Sternberg RJ (1997) Construct validation of a triangular love scale. Eur J Soc Psychol 27(3):313–335

    Google Scholar 

  64. Swartout WR, Gratch J, Hill RW Jr, Hovy E, Marsella S, Rickel J (2006) Toward virtual humans. AI Mag 27(2):96

    Google Scholar 

  65. Sweller J (2011) Cognitive load theory. In: Mestre JP, Ross BH (eds) The psychology of learning and motivation, vol 55. Elsevier Academic Press, pp 37–76

  66. Timmerman GM (1991) A concept analysis of intimacy. Issues Ment Health Nurs 12(1):19–30

    Google Scholar 

  67. Verhagen T, Van Nes J, Feldberg F, Van Dolen W (2014) Virtual customer service agents: using social presence and personalization to shape online service encounters. J Comput Mediat Commun 19(3):529–545

    Google Scholar 

  68. Waring E (1985) Measurement of intimacy: conceptual and methodological issues of studying close relationships. Psychol Med 15(1):9–14

    Google Scholar 

  69. Wilhelm K, Parker G (1988) The development of a measure of intimate bonds. Psychol Med 18(1):225–234

    Google Scholar 

  70. Zhao R, Papangelis A, Cassell J (2014) Towards a dyadic computational model of rapport management for human-virtual agent interaction. In: International conference on intelligent virtual agents. pp 514–527

Download references

Acknowledgements

This work was supported by Agence Nationale de la Recherche Technologique. We would like to sincerely thank DAVI team, and particularly Audrey Pagnier and Aymeric David for their contribution in the creation of Léa and their support in the design of the experiment. We also thank all the participants who participated in our experiment.

Author information

Authors and Affiliations

  1. LIMSI, CNRS, Univ. Paris-Sud, Univ. Paris-Saclay, Orsay, France

    Delphine Potdevin, Céline Clavel & Nicolas Sabouret

  2. DAVI, Puteaux, France

    Delphine Potdevin

Authors
  1. Delphine Potdevin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Céline Clavel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicolas Sabouret
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Delphine Potdevin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Potdevin, D., Clavel, C. & Sabouret, N. Virtual intimacy in human-embodied conversational agent interactions: the influence of multimodality on its perception. J Multimodal User Interfaces 15, 25–43 (2021). https://doi.org/10.1007/s12193-020-00337-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-020-00337-9

Keywords

Search

Navigation