skip to main content
10.1145/3383652.3423881acmconferencesArticle/Chapter ViewAbstractPublication PagesivaConference Proceedingsconference-collections
research-article

Dynamic Emotional Language Adaptation in Multiparty Interactions with Agents

Published: 19 October 2020 Publication History

Abstract

In order to achieve more believable interactions with artificial agents, there is a need to produce dialogue that is not only relevant, but also emotionally appropriate and consistent. This paper presents a comprehensive system that models the emotional state of users and an agent to dynamically adapt dialogue utterance selection. A Partially Observable Markov Decision Process (POMDP) with an online solver is used to model user reactions in real-time. The model decides the emotional content of the next utterance based on the rewards from the users and the agent. The previous approaches are extended through jointly modeling the user and agent emotions, maintaining this model over time with a memory, and enabling interactions with multiple users. A proof of concept user study is used to demonstrate that the system can deliver and maintain distinct agent personalities during multiparty interactions.

References

[1]
M. I. Ahmad, O. Mubin, S. Shahid, and J. Orlando. 2019. Robot's adaptive emotional feedback sustains children's social engagement and promotes their vocabulary learning: a long-term child-robot interaction study. Adaptive Behavior 27, 4 (2019), 243--266. https://doi.org/10.1177/1059712319844182
[2]
P. Alves-Oliveira, P. Sequeira, F. S. Melo, G. Castellano, and A. Paiva. 2019. Empathic Robot for Group Learning: A Field Study. J. Hum.-Robot Interact. 8, 1, Article 3 (March 2019), 34 pages, https://doi.org/10.1145/3300188
[3]
L. F. Barrett, R. Adolphs, S. Marsella, A. M. Martinez, and S. D. Pollak. 2019. Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest 20, 1 (2019), 1--68.
[4]
J. Bates. 1994. The role of emotion in believable agents. Commun. ACM 37, 7 (1994), 122--125.
[5]
J. Broekens and W.-P. Brinkman. 2013. AffectButton: A method for reliable and valid affective self-report. International Journal of Human-Computer Studies 71, 6 (2013), 641--667.
[6]
M. Burkitt and D. M. Romano. 2008. The mood and memory of believable adaptable socially intelligent characters. In Proceedings of the International Workshop on Intelligent Virtual Agents. Springer, 372--379.
[7]
J. Cassell, J. Sullivan, E. Churchill, and S. Prevost. 2000. Embodied Conversational Agents. MIT press.
[8]
P. Colombo, W. Witon, A. Modi, J. Kennedy, and M. Kapadia. 2019. Affect-Driven Dialog Generation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). ACL. https: //doi.org/10.18653/v1/N19-1374
[9]
D. DeVault, R. Artstein, G. Benn, T. Dey, E. Fast, A. Gainer, K. Georgila, J. Gratch, A. Hartholt, M. Lhommet, G. Lucas, S. Marsella, F. Morbini, A. Nazarian, S. Scherer, G. Stratou, A. Suri, D. Traum, R. Wood, Y. Xu, A. Rizzo, and L.-P. Morency. 2014. SimSensei Kiosk: A Virtual Human Interviewer for Healthcare Decision Support. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS '14). 1061--1068.
[10]
P. Emami, A.J. Hamlet, and C. D. Crane. 2015. POMDPy: An Extensible Framework for Implementing Partially-Observable Markov Decision Processes in Python. (2015).
[11]
A. Esuli and F. Sebastiani. 2006. Sentiwordnet: A publicly available lexical resource for opinion mining. In LREC, Vol. 6. Citeseer, 417--422.
[12]
K. Forbes-Riley and D. Litman. 2012. Adapting to Multiple Affective States in Spoken Dialogue. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue (Seoul, South Korea) (SIGDIAL '12). Association for Computational Linguistics, USA, 217--226.
[13]
P. Gebhard. 2005. ALMA: a layered model of affect. In Proceedings of the Fourth International Joint Conference on Autonomous Agents and Multi-Agent Systems. ACM, 29--36.
[14]
G. Gordon, S. Spaulding, J. K. Westlund, J. J. Lee, L. Plummer, M. Martinez, M. Das, and C. Breazeal. 2016. Affective Personalization of a Social Robot Tutor for Children's Second Language Skills. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (Phoenix, Arizona) (AAAI'16). AAAI Press, 3951--3957.
[15]
S. D. Gosling, P.J. Rentfrow, and W. B. Swann Jr. 2003. A very brief measure of the Big-Five personality domains. Journal of Research in personality 37, 6 (2003), 504--528.
[16]
C. J. Hutto and E. Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the Eighth International AAAI Conference on Weblogs and Social Media.
[17]
L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. 1998. Planning and acting in partially observable stochastic domains. Artificial Intelligence 101, 1-2 (1998), 99--134.
[18]
T. Kanda, M. Shiomi, Z. Miyashita, H. Ishiguro, and N. Hagita. 2009. An affective guide robot in a shopping mall. In Proceedings of the 4th ACM/IEEE International Conference on Human-Robot Interaction. ACM, 173--180.
[19]
Z. Kasap, M. B. Moussa, P. Chaudhuri, and N. Magnenat-Thalmann. 2009. Making them remember-Emotional virtual characters with memory. IEEE Computer Graphics and Applications 29, 2(2009), 20--29.
[20]
B. Kempe, N. Pfleger, and M. L. öckelt. 2005. Generating Verbal and Nonverbal Utterances for Virtual Characters. In Proc. Third Int'l Conf. Virtual Storytelling. 73--76. https://doi.org/10.1007/11590361_8
[21]
D. Kollias and S. Zafeiriou. 2018. Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition. arXiv preprint arXiv: 1811.07770 (2018).
[22]
I. Leite, M. McCoy, M. Lohani, D. Ullman, N. Salomons, C. Stokes, S. Rivers, and B. Scassellati. 2017. Narratives with Robots: The Impact of Interaction Context and Individual Differences on Story Recall and Emotional Understanding. Frontiers in Robotics and AI 4 (2017), 29. https://doi.org/10.3389/frobt.2017.00029
[23]
G. Loewenstein and J. S. Lerner. 2003. The role of affect in decision making. Handbook of Affective Science (2003), 619--642.
[24]
V. R. Martinez and J. Kennedy. 2020. A multiparty chat-based dialogue system with concurrent conversation tracking and memory. In CUI' 20 International Conference on Conversational User Interfaces. 1--9. https://doi.org/10.1145/3405755.3406121
[25]
G. S. Martins, H. Al Tair, L. Santos, and J. Dias. 2019. αPOMDP: POMDP-based user-adaptive decision-making for social robots. Pattern Recognition Letters 118 (2019), 94--103.
[26]
Y. Matsuyama, A. Bhardwaj, R. Zhao, O. Romeo, S. Akoju, and J. Cassell. 2016. Socially-aware animated intelligent personal assistant agent. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL '16). 224--227.
[27]
G. Matthews, I. J. Deary, and M. C. Whiteman. 2003. Personality traits. Cambridge University Press.
[28]
A. Mehrabian. 1996. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Current Psychology 14, 4 (1996), 261--292.
[29]
M. B. Moussa and N. Magnenat-Thalmann. 2013. Toward socially responsible agents: integrating attachment and learning in emotional decision-making. Computer Animation and Virtual Worlds 24, 3-4 (2013), 327--334.
[30]
A. Ortony, G. L. Clore, and A. Collins. 1988. The Cognitive Structure of Emotions. Cambridge University Press.
[31]
A. Paiva. 1999. Affective interactions: toward a new generation of computer interfaces?. In Proceedings of the International Workshop on Affective Interactions. Springer, 1--8.
[32]
H. Pfister and G. Böhm. 2008. The multiplicity of emotions: a framework of emotional functions in decision making. Judgment and Decision Making 3, 1 (2008), 5--17.
[33]
R. Read and T.Belpaeme. 2016. People Interpret Robotic Non-linguistic Utterances Categorically. Int J of Soc Robotics 8 (2016), 31--50. https://doi.org/10.1007/s12369-015-0304-0
[34]
J. A. Russell. 1980. A circumplex model of affect. Journal of personality and social psychology 39, 6 (1980), 1161.
[35]
P. Sajjadi, L. Hoffmann, P. Cimiano, and S. Kopp. 2018. On the Effect of a Personality-Driven ECA on Perceived Social Presence and Game Experience in VR. In Proceedings of the 10th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games). IEEE, 1--8.
[36]
M. Schroder, E. Bevacqua, R. Cowie, F. Eyben, H. Gunes, D. Heylen, M. ter Maat, G. McKeown, S. Pammi, M. Pantic, C. Pelachaud, B. Schuller, E. de Sevin, M. Valstar, and M. Wollmer. 2012. Building Autonomous Sensitive Artificial Listeners. IEEE Transactions on Affective Computing 3, 2 (2012), 165--183.
[37]
M. Shvo, J. Buhmann, and M. Kapadia. 2019. An Interdependent Model of Personality, Motivation, Emotion, and Mood for Intelligent Virtual Agents. In Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents. ACM, 65--72.
[38]
D. Silver and J. Veness. 2010. Monte-Carlo planning in large POMDPs. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 2164--2172.
[39]
R. Trappl and P. Petta. 1997. Creating personalities for synthetic actors: Towards autonomous personality agents. Vol. 119. Springer Science & Business Media.
[40]
G. Yannakakis and A. Paiva. 2014. Emotion in Games. In The Oxford Handbook of Affective Computing. Oxford University Press, 459--471. https://doi.org/10.1093/ oxfordhb/9780199942237.001.0001
[41]
E. Zalama, J. G. García-Bermejo, S.Marcos, S. Domínguez, R. Feliz, R. Pinillos, and J. López. 2014. Sacarino, a service robot in a hotel environment. In Proceedings of ROBOT2013: First Iberian Robotics Conference. Springer, 3--14.

Cited By

View all
  • (2024)Recommendations for designing conversational companion robots with older adults through foundation modelsFrontiers in Robotics and AI10.3389/frobt.2024.136371311Online publication date: 27-May-2024
  • (2024)Multi-modal Language Models for Human-Robot InteractionCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3638371(109-111)Online publication date: 11-Mar-2024
  • (2024)Integrating Flow Theory and Adaptive Robot Roles: A Conceptual Model of Dynamic Robot Role Adaptation for the Enhanced Flow Experience in Long-term Multi-person Human-Robot InteractionsProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634945(116-126)Online publication date: 11-Mar-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
IVA '20: Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents
October 2020
394 pages
ISBN:9781450375863
DOI:10.1145/3383652
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Emotion adaptation
  2. dialogue selection
  3. multiparty interaction

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

IVA '20
Sponsor:
IVA '20: ACM International Conference on Intelligent Virtual Agents
October 20 - 22, 2020
Scotland, Virtual Event, UK

Acceptance Rates

Overall Acceptance Rate 53 of 196 submissions, 27%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)23
  • Downloads (Last 6 weeks)0
Reflects downloads up to 12 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Recommendations for designing conversational companion robots with older adults through foundation modelsFrontiers in Robotics and AI10.3389/frobt.2024.136371311Online publication date: 27-May-2024
  • (2024)Multi-modal Language Models for Human-Robot InteractionCompanion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610978.3638371(109-111)Online publication date: 11-Mar-2024
  • (2024)Integrating Flow Theory and Adaptive Robot Roles: A Conceptual Model of Dynamic Robot Role Adaptation for the Enhanced Flow Experience in Long-term Multi-person Human-Robot InteractionsProceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction10.1145/3610977.3634945(116-126)Online publication date: 11-Mar-2024
  • (2023)Artificial Emotional Intelligence in Socially Assistive Robots for Older Adults: A Pilot StudyIEEE Transactions on Affective Computing10.1109/TAFFC.2022.314380314:3(2020-2032)Online publication date: 1-Jul-2023
  • (2021)Coffee With a Hint of Data: Towards Using Data-Driven Approaches in Personalised Long-Term InteractionsFrontiers in Robotics and AI10.3389/frobt.2021.6768148Online publication date: 28-Sep-2021

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media