Abstract
In human face-to-face conversations, non-verbal behaviors (NVB), such as gaze, facial expressions, gestures, and body postures, can improve communication effectiveness, by creating a smooth interaction between the interlocutors - called rapport. During human interactions with embodied conversational agents (ECAs) (a.k.a. virtual humans), a key issue for the success of the interaction is the ability of an ECA to establish and maintain some level of rapport with its human counterpart. This need is particularly important for ECAs who interact in contexts involving socio-emotional content, such as education and entertainment, or in the role of health assistants delivering healthcare interventions, as in the context of this study. Because clinical psychologists are trained in establishing and maintaining rapport, we designed an ECA that learns offline from such an expert, which NVBs to display, when to display them, when not to display them, in real time. We describe our data-driven machine learning approach to modeling rapport from a corpus of annotated videos of counseling sessions, that were conducted by a licensed practicing clinical psychologist with role-playing patients. Results of a randomly controlled experiment show that, in its role of delivering a brief screening health intervention, our ECA improved user’s attitude, intention to (re-)use the ECA system, perceived enjoyment, perceived sociability, perceived usefulness, social presence, and trust.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bartneck, C., Kulic, D., Croft, E.: Measuring the anthropomorphism, animacy, likeability, perceived intelligence and perceived safety of robots. In: Proceedings of the Metrics for Human-Robot Interaction Workshop in Affiliation with the 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI 2008), Technical report 471, vol. 471, pp. 37–44. University of Hertfordshire, Amsterdam (2008)
Cassell, J., Vilhjálmsson, H., Bickmore, T.W.: BEAT: the behavior expression animation toolkit. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 2001), pp. 477–486. ACM (2001). https://doi.org/10.1145/383259.383315
Chartrand, T.L., Bargh, J.A.: The chameleon effect: the perception-behavior link and social interaction. J. Pers. Soc. Psychol. 76(6), 893–910 (1999)
Ekman, P., Freisen, W.V., Ancoli, S.: Facial signs of emotional experience (1980). https://doi.org/10.1037/h0077722
Foster, M.E., Oberlander, J.: Corpus-based generation of head and eyebrow motion for an embodied conversational agent. Lang. Resour. Eval. 41(3–4), 305–323 (2008). https://doi.org/10.1007/s10579-007-9055-3
Grahe, J.: The importance of nonverbal cues in judging rapport. J. Nonverbal Behav. 23(4), 253–269 (1999)
Gratch, J., et al.: Virtual rapport. In: Gratch, J., Young, M., Aylett, R., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 14–27. Springer, Heidelberg (2006). https://doi.org/10.1007/11821830_2
Gratch, J., Wang, N., Gerten, J., Fast, E., Duffy, R.: Creating rapport with virtual agents. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 125–138. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74997-4_12
Gratch, J., et al.: Can virtual humans be more engaging than real ones? In: Jacko, J.A. (ed.) HCI 2007. LNCS, vol. 4552, pp. 286–297. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73110-8_30
Heerink, M., Krose, B., Evers, V., Wielinga, B.: Measuring acceptance of an assistive social robot: a suggested toolkit. In: The 18th IEEE International Symposium on Robot and Human Interactive Communication, RO-MAN 2009, pp. 528–533. IEEE (2009)
Hester, R.K., Squires, D.D., Delaney, H.D.: The Drinker’s Check-up: 12-month outcomes of a controlled clinical trial of a stand-alone software program for problem drinkers. J. Subst. Abuse Treat. 28(2), 159–169 (2005)
Huang, L., Morency, L.-P., Gratch, J.: Learning backchannel prediction model from parasocial consensus sampling: a subjective evaluation. In: Allbeck, J., Badler, N., Bickmore, T., Pelachaud, C., Safonova, A. (eds.) IVA 2010. LNCS (LNAI), vol. 6356, pp. 159–172. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15892-6_17
Huang, L., Morency, L.-P., Gratch, J.: Virtual rapport 2.0. In: Vilhjálmsson, H.H., Kopp, S., Marsella, S., Thórisson, K.R. (eds.) IVA 2011. LNCS (LNAI), vol. 6895, pp. 68–79. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23974-8_8
Joo, H., Simon, T., Cikara, M., Sheikh, Y.: Towards social artificial intelligence: nonverbal social signal prediction in a triadic interaction. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2019, pp. 10865–10875 (2019). https://doi.org/10.1109/CVPR.2019.01113
Kang, S.h., Watt, J.H., Gratch, J.: Associations between interactants’ personality traits and their feelings of rapport in interactions with virtual humans. Paper Presented at the Annual Meeting of the International Communication Association, Marriott, Chicago, IL, pp. 1–25 (2009)
Kipp, M.: Anvil - a generic annotation tool for multimodal dialogue. In: Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech), pp. 1367–1370 (2001)
Kipp, M.: Creativity meets automation: combining nonverbal action authoring with rules and machine learning. In: Gratch, J., Young, M., Aylett, R., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 230–242. Springer, Heidelberg (2006). https://doi.org/10.1007/11821830_19
Lee, J., Marsella, S.C.: Learning a model of speaker head nods using gesture corpora. In: Decker, Sichman, Sierra, Castelfranchi (eds.) 8th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2009). No. Aamas, International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org), Budapest, Hungary (2009)
Lee, J., Marsella, S.: Nonverbal behavior generator for embodied conversational agents. In: Gratch, J., Young, M., Aylett, R., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 243–255. Springer, Heidelberg (2006). https://doi.org/10.1007/11821830_20
Lee, J., Prendinger, H., Neviarouskaya, A., Marsella, S.: Learning models of speaker head nods with affective information. In: 3rd International Conference on Affective Computing and Intelligent Interaction (ACII 2009), pp. 1–6. IEEE (2009)
Maura E. Stokes, Davis, C.S., Koch, G.G.: Categorical Data Analysis Using the SAS System, 2nd edn. SAS Institute and Wiley (2003)
Miller, W.R., Rollnick, S.: Motivational Interviewing: Preparing People for Change, vol. 2, 2nd edn. Guilford Press, New York (2002). https://doi.org/10.1026/1616-3443.34.1.66
R. Amini, C.L., Ruiz, G.: HapFACS 3.0: FACS-based facial expression generator for 3D speaking virtual characters. IEEE Trans. Affect. Comput. PP(99), 1–13 (2015)
Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989)
Tickle-Degnen, L., Rosenthal, R.: The nature of rapport and its nonverbal correlates. Psychol. Inq. 1(4), 285–293 (1990)
Toutanova, K., Klein, D., Manning, C.D., Singer, Y.: Feature-rich part-of-speech tagging with a cyclic dependency network. In: Proceedings of HLT-NAACL, pp. 252–259 (2003)
Acknowledgment
This work was partially funded by research grant number CISE IIS-1423260 from the National Science Foundation to Florida International University.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Amini, R., Boustani, M., Lisetti, C. (2021). Modeling Rapport for Conversations About Health with Autonomous Avatars from Video Corpus of Clinician-Client Therapy Sessions. In: Duffy, V.G. (eds) Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Human Body, Motion and Behavior. HCII 2021. Lecture Notes in Computer Science(), vol 12777. Springer, Cham. https://doi.org/10.1007/978-3-030-77817-0_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-77817-0_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77816-3
Online ISBN: 978-3-030-77817-0
eBook Packages: Computer ScienceComputer Science (R0)