Skip to main content
SpringerLink
Log in

Modeling User’s Social Attitude in a Conversational System

  • Chapter
  • First Online:
Book cover Emotions and Personality in Personalized Services

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

With the growing number of conversational systems that find their way in our daily life, new questions and challenges arise. Even though natural conversation with agent-based systems has been improved in the recent years, e.g., by better speech recognition algorithms, they still lack the ability to understand nonverbal behavior and conversation dynamics—a key part of human natural interaction. To make a step towards intuitive and natural interaction with virtual agents, social robots, and other conversational systems, this chapter proposes a probabilistic framework that models the dynamics of interpersonal cues reflecting the user’s social attitude within the context they occur.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://genie.sis.pitt.edu/.

References

  1. Anderson, K., André, E., Baur, T., Bernardini, S., Chollet, M., Chryssafidou, E., Damian, I., Ennis, C., Egges, A., Gebhard, P., Jones, H., Ochs, M., et al.: The tardis framework: intelligent virtual agents for social coaching in job interviews. In: Proceedings of the Tenth International Conference on Advances in Computer Entertainment Technology (ACE-13). Enschede, The Netherlands, November 2013, LNCS 8253 (2013)

    Google Scholar 

  2. Batrinca, L.M., Stratou, G., Shapiro, A., Morency, L.P., Scherer, S.: Cicero - towards a multimodal virtual audience platform for public speaking training. In: Aylett, R., Krenn, B., Pelachaud, C., Shimodaira, H. (eds.) Proceedings of 13th International Conference on Intelligent Virtual Agents, IVA 2013, Edinburgh, UK, August 29–31, 2013. Lecture Notes in Computer Science, vol. 8108, pp. 116–128, Springer (2013)

    Google Scholar 

  3. Baur, T., Damian, I., Gebhard, P., Porayska-Pomsta, K., Andre, E.: A job interview simulation: social cue-based interaction with a virtual character. In: 2013 IEEE/ASE International Conference on Social Computing (SocialCom), pp. 220–227, Washington D.C., USA (2013)

    Google Scholar 

  4. Baur, T., Damian, I., Lingenfelser, F., Wagner, J., André, E.: Nova: automated analysis of nonverbal signals in social interactions. In: Salah, A., Hung, H., Aran, O., Gunes, H. (eds.) Human Behavior Understanding. LNCS, vol. 8212, pp. 160–171, Springer International Publishing (2013)

    Google Scholar 

  5. Baur, T., Mehlmann, G., Damian, I., Lingenfelser, F., Wagner, J., Lugrin, B., André, E., Gebhard, P.: Context-aware automated analysis and annotation of social human-agent interactions. ACM Trans. Interact. Intell. Syst. (TiiS) 5(2), 11 (2015)

    Google Scholar 

  6. Beck, J.E.: Engagement tracing: using response times to model student disengagement. In: Looi, C., McCalla, G.I., Bredeweg, B., Breuker, J. (eds.) Artificial Intelligence in Education—Supporting Learning through Intelligent and Socially Informed Technology, Proceedings of the 12th International Conference on Artificial Intelligence in Education, AIED 2005, July 18–22, 2005, Amsterdam, The Netherlands. Frontiers in Artificial Intelligence and Applications, vol. 125, pp. 88–95, IOS Press (2005)

    Google Scholar 

  7. Broekens, J., Heerink, M., Rosendal, H.: Assistive social robots in elderly care: a review. Gerontechnology 8(2) (2009)

    Google Scholar 

  8. Camurri, A., Volpe, G., De Poli, G., Leman, M.: Communicating expressiveness and affect in multimodal interactive systems. IEEE MultiMedia 12(1) (2005)

    Google Scholar 

  9. Caridakis, G., Wagner, J., Raouzaiou, A., Lingenfelser, F., Karpouzis, K., André, E.: A cross-cultural, multimodal, affective corpus for gesture expressivity analysis. J. Multimodal User Interfaces 7(1–2), 121–134 (2013)

    Article  Google Scholar 

  10. Conati, C., Maclaren, H.: Empirically building and evaluating a probabilistic model of user affect. User Model. User-Adap. Inter. 19(3), 267–303 (2009)

    Article  Google Scholar 

  11. Damian, I., Baur, T., André, E.: Investigating social cue-based interaction in digital learning games. In: Proceedings of the 1st International Workshop on Intelligent Digital Games for Empowerment and Inclusion (IDGEI 2013) Held in Conjunction with the 8th Foundations of Digital Games 2013 (FDG), ACM, SASDG Digital Library, Chania, Crete, Greece (2013)

    Google Scholar 

  12. De Carolis, B., Novielli, N.: Recognizing signals of social attitude in interacting with ambient conversational systems. J. Multimodal User Interfaces 8(1), 43–60 (2014)

    Google Scholar 

  13. D’Mello, S., Chipman, P., Graesser, A.: Posture as a predictor of learner’s affective engagement. In: Proceedings of the 29th Annual Cognitive Science Society, pp. 905–991, Cognitive Science Society (2007)

    Google Scholar 

  14. Eagly, A.H., Chaiken, S.: Attitude structure and function. In: Fiske, S.T., Gilbert, D.T., Lindzey, G. (eds.) The handbook of social psychology, vol. 1, pp. 269–322, 4th edn. McGraw-Hill (1998)

    Google Scholar 

  15. Endraß, B., André, E., Rehm, M., Nakano, Y.I.: Investigating culture-related aspects of behavior for virtual characters. Auton. Agent. Multi-Agent Syst. 27(2), 277–304 (2013)

    Article  Google Scholar 

  16. Gebhard, P., Mehlmann, G., Kipp, M.: Visual scenemaker: a tool for authoring interactive virtual characters. J. Multimodal User Interfaces 6, 3–11 (2012)

    Article  Google Scholar 

  17. Gebhard, P., Baur, T., Damian, I., Mehlmann, G., Wagner, J., André, E.: Exploring interaction strategies for virtual characters to induce stress in simulated job interviews. In: Proceedings of AAMAS (2014)

    Google Scholar 

  18. Greenwald, A.G., Banaji, M.R.: Implicit social cognition: attitudes, self-esteem, and stereotypes. Psychol. Rev. 102(1), 4 (1995)

    Article  Google Scholar 

  19. Hoque, M.E., Courgeon, M., Martin, J.C., Mutlu, B., Picard, R.W.: MACH: my automated conversation coach. In: Mattern, F., Santini, S., Canny, J.F., Langheinrich, M., Rekimoto, J. (eds.) The 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, UbiComp ’13, Zurich, Switzerland, September 8–12, 2013, pp. 697–706, ACM (2013)

    Google Scholar 

  20. Hung, H., Gatica-Perez, D.: Estimating cohesion in small groups using audio-visual nonverbal behavior. Trans. Multimedia 12(6), 563–575 (2010)

    Article  Google Scholar 

  21. Kang, S.H., Gratch, J., Sidner, C.L., Artstein, R., Huang, L., Morency, L.P.: Towards building a virtual counselor: modeling nonverbal behavior during intimate self-disclosure. In: van der Hoek, W., Padgham, L., Conitzer, V., Winikoff, M. (eds.) International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2012, Valencia, Spain, June 4–8, 2012 (3 Volumes), pp. 63–70, IFAAMAS (2012)

    Google Scholar 

  22. Kim, J., André, E.: Emotion recognition based on physiological changes in music listening. IEEE Trans. Pattern Anal. Mach. Intell. 30(12), 2067–2083 (2008)

    Article  Google Scholar 

  23. Kleinsmith, A., Bianchi-Berthouze, N.: Form as a cue in the automatic recognition of non-acted affective body expressions. In: D’Mello, S., Graesser, A., Schuller, B., Martin, J.C. (eds.) Affective Computing and Intelligent Interaction, LNCS, vol. 6974, pp. 155–164. Springer, Berlin (2011)

    Chapter  Google Scholar 

  24. Lingenfelser, F., Wagner, J., André, E., McKeown, G., Curran, W.: An event driven fusion approach for enjoyment recognition in real-time. In: Proceedings of the ACM International Conference on Multimedia, MM’14, pp. 377–386. ACM, New York, NY, USA (2014)

    Google Scholar 

  25. Mahmoud, M., Robinson, P.: Interpreting hand-over-face gestures. In: D’Mello, S.K., Graesser, A.C., Schuller, B.W., Martin, J. (eds.) Proceedings of Fourth International Conference on Affective Computing and Intelligent Interaction, ACII 2011, Memphis, TN, USA, October 9–12, 2011, Part II. Lecture Notes in Computer Science, vol. 6975, pp. 248–255, Springer (2011)

    Google Scholar 

  26. Mahmoud, M., Morency, L.P., Robinson, P.: Automatic multimodal descriptors of rhythmic body movement. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, pp. 429–436, ACM (2013)

    Google Scholar 

  27. Mancini, M., Ach, L., Bantegnie, E., Baur, T., Berthouze, N., Datta, D., Ding, Y., Dupont, S., Griffin, H., Lingenfelser, F., Niewiadomski, R., Pelachaud, C., Pietquin, O., Piot, B., Urbain, J., Volpe, G., Wagner, J.: Laugh when you’re winning. In: Rybarczyk, Y., Cardoso, T., Rosas, J., Camarinha-Matos, L. (eds.) Innovative and Creative Developments in Multimodal Interaction Systems, IFIP Advances in Information and Communication Technology, vol. 425, pp. 50–79. Springer, Berlin (2014)

    Google Scholar 

  28. Mehlmann, G., Janowski, K., Baur, T., Häring, M., André, E., Gebhard, P.: Modeling gaze mechanisms for grounding in hri. In: Proceedings of the 21th European Conference on Artificial Intelligence. ECAI 2014, Prague, Czech Republic, August 18–22, 2014, Frontiers in Artificial Intelligence and Applications, pp. 1069–1070. IOS Press Ebooks, Amsterdam, The Netherlands (2014)

    Google Scholar 

  29. Michelet, S., Karp, K., Delaherche, E., Achard, C., Chetouani, M.: Automatic imitation assessment in interaction. Human Behavior Understanding. Lecture Notes in Computer Science, vol. 7559, pp. 161–173. Springer, Berlin (2012)

    Google Scholar 

  30. Morency, L.P.: Modeling human communication dynamics. IEEE Signal Process. Mag. 27(5), 112–116 (2010)

    Article  Google Scholar 

  31. Nakano, Y.I., Ishii, R.: Estimating user’s engagement from eye-gaze behaviors in human-agent conversations. In: Proceedings of the 15th International Conference on Intelligent User Interfaces, IUI ’10, pp. 139–148. ACM, New York, NY, USA (2010)

    Google Scholar 

  32. Niewiadomski, R., Hofmann, J., Urbain, J., Platt, T., Wagner, J., Piot, B., Cakmak, H., Pammi, S., Baur, T., Dupont, S., Geist, M., Lingenfelser, F., McKeown, G., Pietquin, O., Ruch, W.: Laugh-aware virtual agent and its impact on user amusement. In: Proceedings of the 2013 International Conference on Autonomous Agents and Multi-agent systems. AAMAS’13, pp. 619–626. International Foundation for Autonomous Agents and Multiagent Systems, Richland (2013)

    Google Scholar 

  33. Pantic, M., Sebe, N., Cohn, J.F., Huang, T.: Affective multimodal human-computer interaction. In: Proceedings of the 13th Annual ACM International Conference on Multimedia. MULTIMEDIA’05, pp. 669–676. ACM, New York, NY, USA (2005)

    Google Scholar 

  34. Pease, A.: Body Language. Sheldon Press, London (1988)

    Google Scholar 

  35. Petridis, S., Gunes, H., Kaltwang, S., Pantic, M.: Static vs. dynamic modeling of human nonverbal behavior from multiple cues and modalities. In: Crowley, J.L., Ivanov, Y.A., Wren, C.R., Gatica-Perez, D., Johnston, M., Stiefelhagen, R. (eds.) Proceedings of the 11th International Conference on Multimodal Interfaces, ICMI 2009, Cambridge, Massachusetts, USA, November 2–4, 2009, pp. 23–30, ACM (2009)

    Google Scholar 

  36. Reeves, B., Nass, C.: How people treat computers, television, and new media like real people and places. CSLI Publications and Cambridge university press, Cambridge (1996)

    Google Scholar 

  37. Rich, C., Ponsleur, B., Holroyd, A., Sidner, C.L.: Recognizing engagement in human-robot interaction. In: Proceedings of the 5th ACM/IEEE International Conference on Human-robot interaction, HRI’10, pp. 375–382. IEEE Press, Piscataway (2010)

    Google Scholar 

  38. Rosenberg, M.J., Hovland, C.I.: Cognitive, affective, and behavioral components of attitudes. Attitude organization and change: an analysis of consistency among attitude components 3, 1–14 (1960)

    Google Scholar 

  39. Russell, S.J., Norvig, P.: Artificial Intelligence: a modern approach, 2nd int. edn. Prentice Hall, Upper Saddle River (2003)

    Google Scholar 

  40. Salam, H., Chetouani, M.: A multi-level context-based modelling of engagement in human-robot interaction. In: International Workshop on Context Based Affect Recognition (2015)

    Google Scholar 

  41. Sandbach, G., Zafeiriou, S., Pantic, M., Yin, L.: Static and dynamic 3d facial expression recognition: a comprehensive survey. Image Vision Comput. 30(10), 683–697 (2012)

    Article  Google Scholar 

  42. Sanghvi, J., Castellano, G., Leite, I., Pereira, A., McOwan, P.W., Paiva, A.: Automatic analysis of affective postures and body motion to detect engagement with a game companion. In: Billard, A., Adams, P.H.K.J.A., Jr., Trafton, J.G. (eds.) Proceedings of the 6th International Conference on Human Robot Interaction, HRI 2011, Lausanne, Switzerland, March 6-9, 2011, pp. 305–312, ACM (2011)

    Google Scholar 

  43. Scherer, S., Marsella, S., Stratou, G., Xu, Y., Morbini, F., Egan, A., Rizzo, A., Morency, L.P.: Perception markup language: Towards a standardized representation of perceived nonverbal behaviors. In: Nakano, Y., Neff, M., Paiva, A., Walker, M. (eds.) Intelligent Virtual Agents, LNCS, vol. 7502, pp. 455–463. Springer, Berlin (2012)

    Chapter  Google Scholar 

  44. Sebe, N., Cohen, I., Gevers, T., Huang, T.S.: Emotion recognition based on joint visual and audio cues. In: Proceedings of the 18th International Conference on Pattern Recognition—Volume 01, ICPR’06, pp. 1136–1139. IEEE Computer Society, Washington, DC, USA (2006)

    Google Scholar 

  45. Sidner, C.L., Kidd, C.D., Lee, C., Lesh, N.: Where to look: a study of human-robot engagement. In: IUI ’04: Proceedings of the 9th International Conference on Intelligent user Interfaces, pp. 78–84. ACM Press, New York, NY, USA (2004)

    Google Scholar 

  46. Traum, D.R., DeVault, D., Lee, J., Wang, Z., Marsella, S.: Incremental dialogue understanding and feedback for multiparty, multimodal conversation. In: Nakano, Y., Neff, M., Paiva, A., Walker, M.A. (eds.) Proceedings of 12th International Conference on Intelligent Virtual Agents, IVA 2012, Santa Cruz, CA, USA, September, 12–14, 2012. Lecture Notes in Computer Science, vol. 7502, pp. 275–288, Springer (2012)

    Google Scholar 

  47. Vail, A.K., Grafsgaard, J.F., Wiggins, J.B., Lester, J.C., Boyer, K.E.: Predicting learning and engagement in tutorial dialogue: a personality-based model. In: Salah, A.A., Cohn, J.F., Schuller, B.W., Aran, O., Morency, L., Cohen, P.R. (eds.) Proceedings of the 16th International Conference on Multimodal Interaction, ICMI 2014, Istanbul, Turkey, November 12–16, 2014, pp. 255–262, ACM (2014)

    Google Scholar 

  48. Valstar, M.: Automatic facial expression analysis. In: Mandal, M.K., Awasthi, A. (eds.) Understanding Facial Expressions in Communication, pp. 143–172. Springer India, New York (2015)

    Google Scholar 

  49. Vogt, T., André, E., Bee, N.: Emovoice—a framework for online recognition of emotions from voice. In: Perception in Multimodal Dialogue Systems, 4th IEEE Tutorial and Research Workshop on Perception and Interactive Technologies for Speech-Based Systems, Kloster Irsee, Germany, LNCS, pp. 188–199, Springer (2008)

    Google Scholar 

  50. Wagner, J., Lingenfelser, F., Baur, T., Damian, I., Kistler, F., André, E.: The social signal interpretation (ssi) framework—multimodal signal processing and recognition in real-time. In: Proceedings of ACM MULTIMEDIA 2013, Barcelona (2013)

    Google Scholar 

  51. Whitehill, J., Serpell, Z., Lin, Y., Foster, A., Movellan, J.R.: The faces of engagement: automatic recognition of student engagementfrom facial expressions. T. Affect. Comput. 5(1), 86–98 (2014)

    Article  Google Scholar 

  52. Yu, C., Aoki, P.M., Woodruff, A.: Detecting User Engagement in Everyday Conversations. eprint arXiv:cs/0410027 (2004)

Download references

Acknowledgments

This work has received funding from the European Union’s Horizon 2020 research and innovation programme (Project ARIA-VALUSPA, grant agreement no. 645378) and has been partially funded by the German Federal Ministry of Education and Research (BMBF) in the project EmpaT, research grant 16SV7229K. We thank Charamel GmbH for their continuous support and for providing us with the virtual characters Gloria and Curtis.

Author information

Authors and Affiliations

  1. Human Centered Multimedia, Augsburg University, Universitätsstraße 6a, 86159, Augsburg, Germany

    Tobias Baur, Dominik Schiller & Elisabeth André

Authors
  1. Tobias Baur
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dominik Schiller
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Elisabeth André
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tobias Baur .

Editor information

Editors and Affiliations

  1. Department of Computational Perception, Johannes Kepler University, Linz, Austria

    Marko Tkalčič

  2. Department of Computer Science, University of Bari Aldo Moro, Bari, Italy

    Berardina De Carolis

  3. Department of Computer Science, University of Bari Aldo Moro, Bari, Italy

    Marco de Gemmis

  4. Preserje, Ljubljana, Slovenia

    Ante Odić

  5. Faculty of electrical engineering, University of Ljubljana, Ljubljana, Slovenia

    Andrej Košir

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Baur, T., Schiller, D., André, E. (2016). Modeling User’s Social Attitude in a Conversational System. In: Tkalčič, M., De Carolis, B., de Gemmis, M., Odić, A., Košir, A. (eds) Emotions and Personality in Personalized Services. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-319-31413-6_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-31413-6_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-31411-2

  • Online ISBN: 978-3-319-31413-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation