Skip to main content
SpringerLink
Log in

Socially-Aware Virtual Agents: Automatically Assessing Dyadic Rapport from Temporal Patterns of Behavior

  • Conference paper
  • First Online:
Intelligent Virtual Agents (IVA 2016)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10011))

Included in the following conference series:

Abstract

This work focuses on data-driven discovery of the temporally co-occurring and contingent behavioral patterns that signal high and low interpersonal rapport. We mined a reciprocal peer tutoring corpus reliably annotated for nonverbals like eye gaze and smiles, conversational strategies like self-disclosure and social norm violation, and for rapport (in 30 s thin slices). We then performed a fine-grained investigation of how the temporal profiles of sequences of interlocutor behaviors predict increases and decreases of rapport, and how this rapport management manifests differently in friends and strangers. We validated the discovered behavioral patterns by predicting rapport against our ground truth via a forecasting model involving two-step fusion of learned temporal associated rules. Our framework performs significantly better than a baseline linear regression method that does not encode temporal information among behavioral features. Implications for the understanding of human behavior and social agent design are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    sociallyawarerobotassistant.net.

References

  1. Allwood, J., Cerrato, L.: A study of gestural feedback expressions. In: First Nordic Symposium on Multimodal Communication, pp. 7–22. Copenhagen (2003)

    Google Scholar 

  2. Ambady, N., Rosenthal, R.: Thin slices of expressive behavior as predictors of interpersonal consequences: a meta-analysis. Psychol. Bull. 111(2), 256 (1992)

    Article  Google Scholar 

  3. Bevacqua, E., Mancini, M., Pelachaud, C.: A listening agent exhibiting variable behaviour. In: Prendinger, H., Lester, J., Ishizuka, M. (eds.) IVA 2008. LNCS (LNAI), vol. 5208, pp. 262–269. Springer, Heidelberg (2008). doi:10.1007/978-3-540-85483-8_27

    Chapter  Google Scholar 

  4. Cappella, J.N.: On defining conversational coordination and rapport. Psychol. Inq. 1(4), 303–305 (1990)

    Article  Google Scholar 

  5. Cassell, J., Gill, A.J., Tepper, P.A.: Coordination in conversation and rapport. In: Proceedings of the workshop on Embodied Language Processing, pp. 41–50. ACL (2007)

    Google Scholar 

  6. Chiu, C.C., Morency, L.-P., Marsella, S.: Predicting co-verbal gestures: a deep and temporal modeling approach, pp. 152–166 (2015)

    Google Scholar 

  7. Chollet, M., Ochs, M., Pelachaud, C.: From non-verbal signals sequence mining to Bayesian networks for interpersonal attitudes expression. In: Bickmore, T., Marsella, S., Sidner, C. (eds.) IVA 2014. LNCS (LNAI), vol. 8637, pp. 120–133. Springer, Heidelberg (2014). doi:10.1007/978-3-319-09767-1_15

    Google Scholar 

  8. Chou, C.-Y., Chan, T.-W.: Reciprocal tutoring: design with cognitive load sharing. Int. J. Artif. Intell. Educ. 26(1), 1–24 (2015)

    Google Scholar 

  9. D’Mello, S., Olney, A., Williams, C., Hays, P.: Gaze tutor: a gaze-reactive intelligent tutoring system. Int. J. Hum. Comput. Stud. 70(5), 377–398 (2012)

    Article  Google Scholar 

  10. Fujimoto, D.T.: Listener Responses in Interaction: A Case for Abandoning the Term. Backchannel (2009)

    Google Scholar 

  11. Goffman, E.: Interaction Ritual: Essays in Face to Face Behavior. AldineTransaction, New Brunswick (2005)

    Google Scholar 

  12. Gratch, J., Okhmatovskaia, A., Lamothe, F., Marsella, S., Morales, M., Werf, R.J., Morency, L.-P.: Virtual rapport. In: Gratch, J., Young, M., Aylett, R., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 14–27. Springer, Heidelberg (2006). doi:10.1007/11821830_2

    Chapter  Google Scholar 

  13. Gratch, J., Wang, N., Gerten, J., Fast, E., Duffy, R.: Creating rapport with virtual agents. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 125–138. Springer, Heidelberg (2007). doi:10.1007/978-3-540-74997-4_12

    Chapter  Google Scholar 

  14. Gravano, A., Hirschberg, J.: Backchannel-inviting cues in task-oriented dialogue. In: INTERSPEECH, pp. 1019–1022 (2009)

    Google Scholar 

  15. Guillame-Bert, M., Crowley, J.L.: Learning temporal association rules on symbolic time sequences, pp. 159–174 (2012)

    Google Scholar 

  16. Guillame-Bert, M., Dubrawski, A.: Learning temporal rules to forecast events in multivariate time sequences

    Google Scholar 

  17. Heylen, D., Bevacqua, E., Tellier, M., Pelachaud, C.: Searching for prototypical facial feedback signals. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 147–153. Springer, Heidelberg (2007). doi:10.1007/978-3-540-74997-4_14

    Chapter  Google Scholar 

  18. Huang, L., Morency, L.-P., Gratch, J.: Virtual rapport 2.0. In: Vilhjálmsson, H.H., Kopp, S., Marsella, S., Thórisson, K.R. (eds.) IVA 2011. LNCS (LNAI), vol. 6895, pp. 68–79. Springer, Heidelberg (2011). doi:10.1007/978-3-642-23974-8_8

    Chapter  Google Scholar 

  19. Johnson, D.W.: Student-student interaction: the neglected variable in education. Educ. Researcher 10(1), 5–10 (1981)

    Article  Google Scholar 

  20. Kang, S.-H., Gratch, J., Sidner, C., Artstein, R., Huang, L., Morency, L.-P., Towards building a virtual counselor: modeling nonverbal behavior during intimate self-disclosure. In: Proceedings of the 11th ICAAMS, vol. 1, pp. 63–70 (2012)

    Google Scholar 

  21. Keltner, D., Buswell, B.N.: Embarrassment: its distinct form and appeasement functions. Psychol. Bull. 122(3), 250 (1997)

    Article  Google Scholar 

  22. Klem, A.M., Connell, J.P.: Relationships matter: Linking teacher support to student engagement and achievement. J. Sch. Health 74(7), 262–273 (2004)

    Article  Google Scholar 

  23. Kreijns, K., Kirschner, P.A., Jochems, W.: Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: a review of the research. Comput. Hum. Behav. 19(3), 335–353 (2003)

    Article  Google Scholar 

  24. Laurenceau, J.-P., Barrett, L.F., Pietromonaco, P.R.: Intimacy as an interpersonal process: the importance of self-disclosure, partner disclosure, and perceived partner responsiveness in interpersonal exchanges. J. Pers. Soc. Psychol. 74(5), 1238 (1998)

    Article  Google Scholar 

  25. Matsuyama, Y., Bhardwaj, A., Zhao, R., Romero, O.J., Akoju, S., CassellJ.: Socially-aware animated intelligent personal assistant agent. In: 17th Annual SIGdial Meeting on Discourse and Dialogue (2016)

    Google Scholar 

  26. Nakano, YI., Nihonyanagi, S., Takase, Y., Hayashi, Y., Okada, S.: Predicting participation styles using co-occurrence patterns of nonverbal behaviors in collaborative learning, pp. 91–98 (2015)

    Google Scholar 

  27. Ogan, A., Finkelstein, S., Walker, E., Carlson, R., Cassell, J.: Rudeness and rapport: insults and learning gains in peer tutoring. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 11–21. Springer, Heidelberg (2012). doi:10.1007/978-3-642-30950-2_2

    Chapter  Google Scholar 

  28. Ramanarayanan, V., Leong, CW., Chen, L., Feng, G., Suendermann-Oeft, D.: Evaluating speech, face, emotion and body movement time-series features for automated multimodal presentation scoring, pp. 23–30 (2015)

    Google Scholar 

  29. Rummel, N., Walker, E., Aleven, V.: Different futures of adaptive collaborative learning support. Int. J. Artif. Intell. Educ. 26(2), 784–795 (2016)

    Article  Google Scholar 

  30. Sharpley, A.M., Irvine, J.W., Sharpley, C.F.: An examination of the effectiveness of a cross-age tutoring program in mathematics for elementary school children. Am. Educ. Res. J. 20(1), 103–111 (1983)

    Article  Google Scholar 

  31. Sinha, T., Cassell, J.: Fine-grained analyses of interpersonal processes and their effect on learning. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (eds.) AIED 2015. LNCS (LNAI), vol. 9112, pp. 781–785. Springer, Heidelberg (2015). doi:10.1007/978-3-319-19773-9_115

    Chapter  Google Scholar 

  32. Sinha, T., Cassell, J., We click, we align, we learn: impact of influence and convergence processes on student learning and rapport building. In: 17th ACM International Conference on Multimodal Interaction Proceedings of the 2015 Workshop on Modeling Interpersonal Synchrony. ACM (2015)

    Google Scholar 

  33. Sinha, T., Zhao, R., Cassell, J.: Exploring socio-cognitive effects of conversational strategy congruence in peer tutoring. In: 17th ACM International Conference on Multimodal Interaction Proceedings of the 2015 Workshop on Modeling Interpersonal Synchrony. ACM (2015)

    Google Scholar 

  34. Tickle-Degnen, L., Rosenthal, R.: The nature of rapport and its nonverbal correlates. Psychol. Inquiry 1(4), 285–293 (1990)

    Article  Google Scholar 

  35. Woertwein, T., Chollet, M., Schauerte, B., Stiefelhagen, R., Morency, L.P., Scherer, S.: Multimodal public speaking performance assessment, pp. 43–50 (2015)

    Google Scholar 

  36. Yu, Z., Gerritsen, D., Ogan, A., Black, A., Cassell, J.: Automatic prediction of friendship via multi-model dyadic features. In: 14th Annual SIGdial Meeting on Discourse and Dialogue, Metz, France (2013)

    Google Scholar 

  37. Zhao, R., Papangelis, A., Cassell, J.: Towards a dyadic computational model of rapport management for human-virtual agent interaction. In: Bickmore, T., Marsella, S., Sidner, C. (eds.) IVA 2014. LNCS (LNAI), vol. 8637, pp. 514–527. Springer, Heidelberg (2014). doi:10.1007/978-3-319-09767-1_62

    Google Scholar 

  38. Zhao, R., Sinha, T., Black, A., Cassell, J.: Automatic recognition of conversational strategies in the service of a socially-aware dialog system. In: 17th Annual SIGDIAL Meeting on Discourse and Dialogue (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Language Technologies Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, 15213, USA

    Ran Zhao, Tanmay Sinha, Alan W. Black & Justine Cassell

Authors
  1. Ran Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tanmay Sinha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alan W. Black
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Justine Cassell
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ran Zhao .

Editor information

Editors and Affiliations

  1. University of Southern California, Los Angeles, California, USA

    David Traum

  2. University of Southern California, Los Angeles, California, USA

    William Swartout

  3. US Army Research Laboratory, Los Angeles, California, USA

    Peter Khooshabeh

  4. Universität Bielefeld, Bielefeld, Nordrhein-Westfalen, Germany

    Stefan Kopp

  5. University of Southern California, Los Angeles, California, USA

    Stefan Scherer

  6. University of Southern California, Los Angeles, California, USA

    Anton Leuski

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Zhao, R., Sinha, T., Black, A.W., Cassell, J. (2016). Socially-Aware Virtual Agents: Automatically Assessing Dyadic Rapport from Temporal Patterns of Behavior. In: Traum, D., Swartout, W., Khooshabeh, P., Kopp, S., Scherer, S., Leuski, A. (eds) Intelligent Virtual Agents. IVA 2016. Lecture Notes in Computer Science(), vol 10011. Springer, Cham. https://doi.org/10.1007/978-3-319-47665-0_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-47665-0_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-47664-3

  • Online ISBN: 978-3-319-47665-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation