Abstract
How can conversational agents be better designed to build rapport with human beings? Related work on creating rapport through conversational agents has largely focused on nonverbal contingent envelope feedback. There is relatively little known about how forms of emotional feedback play a role in building rapport between agents and humans. This paper describes a study in which people told stories to an agent that provided emotional feedback in the form of facial expressions. Rapport was measured through the length of the stories, the fluency of their speech, and the user’s own subjective experience. Surprisingly, results indicated that inappropriate emotional feedback increased story length, which was the opposite of previous studies on envelope feedback that had shorter stories in unresponsive conditions. This paper explains the factors particular to emotional feedback that could cause this difference.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Tickle-Degnen, L., Rosenthal, R.: The Nature of Rapport and Its Nonverbal Correlates. Psychological Inquiry 1(4), 285–293 (1990)
Cassell, J., Thorisson, K.R.: The Power of a Nod and a Glance: Envelope vs. Emotional Feedback in Animated Conversational Agents. Applied Artificial Intelligence 13, 519–538 (1999)
Bailenson, J.N., Yee, N.: Digital Chameleons: Automatic Assimilation of Nonverbal Gestures in Immersive Virtual Environments. Psychological Science 16, 814–819 (2005)
Gratch, J., Okhmatovskaia, A., Lamothe, F., Marsella, S.C., Morales, M., van der Werf, R.J., Morency, L.-P.: Virtual Rapport. In: Gratch, J., Young, M., Aylett, R.S., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 14–27. Springer, Heidelberg (2006)
Gratch, J., Wang, N., Gerten, J., Fast, E., Duffy, R.: Creating Rapport with Virtual Agents. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 125–138. Springer, Heidelberg (2007)
Kang, S.-H., et al.: Does the contingency of agents’ nonverbal feedback affect users’ social anxiety? In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, Portugal, vol. 1. International Foundation for Autonomous Agents and Multiagent Systems, Estoril (2008)
Wang, N., Gratch, J.: Don’t just stare at me! In: Proceedings of the 28th International Conference on Human Factors in Computing Systems. ACM, Atlanta (2010)
Krumhuber, E., et al.: Effects of Dynamic Attributes of Smiles in Human and Synthetic Faces: A Simulated Job Interview Setting. Journal of Nonverbal Communication 33, 1–15 (2009)
Bickmore, T., Picard, R.: Establishing and maintaining long-term human-computer relationships. ACM Transactions on Computer-Human Interactions, 12(2), 293–327 (2005)
Bickmore, T., Cassell, J.: Relational Agents: A Model and Implementation of Building User Trust. In: CHI (2001)
Koster, T.: The Persuasive Qualities of an Empathic Agent. In: 5th Twente Student Conference on IT. University of Twente, Twente (2006)
Johnson, W.L., Rizzo, P., Bosma, W., Kole, S., Ghijsen, M., van Welbergen, H.: Generating Socially Appropriate Tutorial Dialog. In: André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.) ADS 2004. LNCS (LNAI), vol. 3068, pp. 254–264. Springer, Heidelberg (2004)
Wong, J.W.-E.: Establishing Rapport with Conversational Agents: Comparing the Effect of Envelope versus Emotional Feedback. In: Department of Communications and New Media. National University of Singapore, Singapore (2012)
Thiebaux, M., et al.: SmartBody: Behavior Realization for Embodied Conversational Agents. In: Autonomous Agents and Multi-Agent Systems (AAMAS). International Foundation for Autonomous Agents and Multiagent Systems, Estoril (2008)
Ekman, P., Friesen, W.V.: Facial Action Coding System. Consulting Psychologists Press, Inc., Palo Alto (1978)
Ekman, P.: The directed facial action task. In: Coan, J.A., Allen, J.J.B. (eds.) Handbook of Emotion Elicitation and Assessment. Oxford University Press, Oxford (2007)
Clark, H.H., Brennan, S.E.: Grounding in Communication. In: Levine, L.B.R.J.M., Teasley, S.D. (eds.) Perspectives on Socially Shared Cognition, pp. 127–149. American Psychological Association, Washington D.C (1991)
Myers, D.: Social Psychology. McGraw-Hill College (1999)
Rickenberg, R., Reeves, B.: The effects of animated characters on anxiety, task performance, and evaluations of user interfaces. In: SIGCHI Conference on Human Factors in Computing Systems, The Hague, The Netherlands (2000)
Pertaub, D.-P., Slater, M.: An Experiment on Public Speaking Anxiety in Response to Three Different Types of Virtual Audience. Presence: Teleoperators and Virtual Environments 11(1), 68–78 (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wong, J.WE., McGee, K. (2012). Frown More, Talk More: Effects of Facial Expressions in Establishing Conversational Rapport with Virtual Agents. In: Nakano, Y., Neff, M., Paiva, A., Walker, M. (eds) Intelligent Virtual Agents. IVA 2012. Lecture Notes in Computer Science(), vol 7502. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33197-8_43
Download citation
DOI: https://doi.org/10.1007/978-3-642-33197-8_43
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33196-1
Online ISBN: 978-3-642-33197-8
eBook Packages: Computer ScienceComputer Science (R0)