skip to main content
10.1145/3411763.3451660acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
poster

Enhancing the Perceived Emotional Intelligence of Conversational Agents through Acoustic Cues

Published: 08 May 2021 Publication History

Abstract

The perceived emotional intelligence of a conversational agent (CA) can significantly impact people’s interaction with the CA. Prior research applies text-based sentiment analysis and emotional response generation to improve CAs’ emotional intelligence. However, acoustic features in speech containing rich contexts are underexploited. In this work, we designed and implemented an emotionally aware CA, called HUE (Heard yoUr Emotion) that stylized responses with emotion regulation strategies and empathetic interjections. We conducted a user study with 75 participants to evaluate their perceived emotional intelligence (PEI) of HUE by having them observe conversations between people and HUE in different emotional scenarios. Our results show that participants’ PEI was significantly higher with the acoustic features than without.

References

[1]
Samuel Albanie, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. 2018. Emotion Recognition in Speech Using Cross-Modal Transfer in the Wild. In Proceedings of the 26th ACM International Conference on Multimedia (Seoul, Republic of Korea) (MM ’18). Association for Computing Machinery, New York, NY, USA, 292–301. https://doi.org/10.1145/3240508.3240578
[2]
T. W. Bickmore, R. Fernando, L. Ring, and D. Schulman. 2010. Empathic Touch by Relational Agents. IEEE Transactions on Affective Computing 1, 1 (2010), 60–71. https://doi.org/10.1109/T-AFFC.2010.4
[3]
Ana Paula Chaves and Marco Aurélio Gerosa. 2019. How should my chatbot interact? A survey on human-chatbot interaction design. CoRR abs/1904.02743(2019). arxiv:1904.02743
[4]
Huimin Chen, Maosong Sun, Cunchao Tu, Yankai Lin, and Zhiyuan Liu. 2016. Neural sentiment classification with user and product attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 1650–1659.
[5]
Shizhe Chen, Qin Jin, Jinming Zhao, and Shuai Wang. 2017. Multimodal multi-task learning for dimensional and continuous emotion recognition. In Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge. ACM, 19–26.
[6]
Michelle Cohn, Chun-Yen Chen, and Zhou Yu. 2019. A Large-Scale User Study of an Alexa Prize Chatbot: Effect of TTS Dynamism on Perceived Quality of Social Dialog. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue. 293–306.
[7]
Michelle G Craske, Linda Street, and David H Barlow. 1989. Instructions to focus upon or distract from internal cues during exposure treatment of agoraphobic avoidance. Behaviour research and therapy 27, 6 (1989), 663–672.
[8]
Kohji Dohsaka, Ryota Asai, Ryuichiro Higashinaka, Yasuhiro Minami, and Eisaku Maeda. 2009. Effects of Conversational Agents on Human Communication in Thought-Evoking Multi-Party Dialogues. In Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue(London, United Kingdom) (SIGDIAL ’09). Association for Computational Linguistics, USA, 217–224.
[9]
Martina Drescher. 1997. French interjections and their use in discourse. The Language of Emotions(1997), 233–246.
[10]
Kathleen Kara Fitzpatrick, Alison Darcy, and Molly Vierhile. 2017. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR mental health 4, 2 (2017), e19.
[11]
Barbara L. Fredrickson. 2013. Positive Emotions Broaden and Build. Advances in Experimental Social Psychology, Vol. 47. Academic Press, 1 – 53. https://doi.org/10.1016/B978-0-12-407236-7.00001-2
[12]
Béatrice S Hasler, Gilad Hirschberger, Tal Shani-Sherman, and Doron A Friedman. 2014. Virtual peacemakers: Mimicry increases empathy in simulated contact with virtual outgroup members. Cyberpsychology, Behavior, and Social Networking 17, 12(2014), 766–771.
[13]
Tianran Hu, Anbang Xu, Zhe Liu, Quanzeng You, Yufan Guo, Vibha Sinha, Jiebo Luo, and Rama Akkiraju. 2018. Touch Your Heart: A Tone-Aware Chatbot for Customer Care on Social Media. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3173574.3173989
[14]
Jing Huang, Qi Li, Yuanyuan Xue, Taoran Cheng, Shuangqing Xu, Jia Jia, and Ling Feng. 2015. Teenchat: a chatterbot system for sensing and releasing adolescents’ stress. In International Conference on Health Information Science. Springer, 133–145.
[15]
Mirjana Ivanović, Miloš Radovanović, Zoran Budimac, Dejan Mitrović, Vladimir Kurbalija, Weihui Dai, and Weidong Zhao. 2014. Emotional Intelligence and Agents: Survey and Possible Applications. In International Conference on Web Intelligence.
[16]
Tom Johnstone and Klaus R Scherer. 1999. The effects of emotions on voice quality. In Proceedings of the XIVth international congress of phonetic sciences. Citeseer, 2029–2032.
[17]
Philipp Kanske, Janine Heissler, Sandra Schönfelder, André Bongers, and Michele Wessa. 2011. How to regulate emotion? Neural networks for reappraisal and distraction. Cerebral Cortex 21, 6 (2011), 1379–1388.
[18]
Soomin Kim, Joonhwan Lee, and Gahgene Gweon. 2019. Comparing data from chatbot and web surveys: Effects of platform and conversational style on survey response quality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
[19]
Yi-Chieh Lee, Naomi Yamashita, Yun Huang, and Wai Fu. 2020. “I Hear You, I Feel You”: Encouraging Deep Self-Disclosure through a Chatbot. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376175
[20]
Q Vera Liao, Muhammed Mas-ud Hussain, Praveen Chandar, Matthew Davis, Yasaman Khazaeni, Marco Patricio Crasso, Dakuo Wang, Michael Muller, N Sadat Shami, and Werner Geyer. 2018. All Work and No Play?. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.
[21]
Xiaojuan Ma, Emily Yang, and Pascale Fung. 2019. Exploring Perceived Emotional Intelligence of Personality-Driven Virtual Agents in Handling User Challenges. In The World Wide Web Conference. ACM, 1222–1233.
[22]
Seyedmahdad Mirsamadi, Emad Barsoum, and Cha Zhang. 2017. Automatic speech emotion recognition using recurrent neural networks with local attention. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2227–2231.
[23]
Christos N Moridis and Anastasios A Economides. 2012. Affective learning: Empathetic agents with emotional facial and tone of voice expressions. IEEE Transactions on Affective Computing 3, 3 (2012), 260–272.
[24]
Jonathan Mumm and Bilge Mutlu. 2011. Designing motivational agents: The role of praise, social comparison, and embodiment in computer feedback. Computers in Human Behavior 27, 5 (2011), 1643–1650.
[25]
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. 2017. VoxCeleb: A Large-Scale Speaker Identification Dataset. In Proc. Interspeech 2017. 2616–2620. https://doi.org/10.21437/Interspeech.2017-950
[26]
John B Nezlek, Kristof Vansteelandt, Iven Van Mechelen, and Peter Kuppens. 2008. Appraisal-emotion relationships in daily life.Emotion 8, 1 (2008), 145.
[27]
A.I. Niculescu, S.S. Ge, Elisabeth M.A.G. van Dijk, Antinus Nijholt, Haizhou Li, and Swan Lan See. 2013. Making social robots more attractive: the effects of voice pitch, humor and empathy. International journal of social robotics 5, 2 (21 4 2013), 171–191. https://doi.org/10.1007/s12369-012-0171-x eemcs-eprint-22397.
[28]
Kevin N Ochsner and James J Gross. 2005. The cognitive control of emotion. Trends in cognitive sciences 9, 5 (2005), 242–249.
[29]
Kevin N Ochsner and James J Gross. 2008. Cognitive emotion regulation: Insights from social cognitive and affective neuroscience. Current directions in psychological science 17, 2 (2008), 153–158.
[30]
Qiao Qian, Minlie Huang, Jinhao Lei, and Xiaoyan Zhu. 2017. Linguistically Regularized LSTM for Sentiment Classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 1679–1689. https://doi.org/10.18653/v1/P17-1154
[31]
Peter Salovey and John D Mayer. 1990. Emotional intelligence. Imagination, cognition and personality 9, 3 (1990), 185–211.
[32]
Peter Ed Salovey and David J Sluyter. 1997. Emotional development and emotional intelligence: Educational implications.Basic Books.
[33]
Heung-Yeung Shum, Xiao-dong He, and Di Li. 2018. From Eliza to XiaoIce: challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering 19, 1(2018), 10–26.
[34]
Zhenqiao Song, Xiaoqing Zheng, Lu Liu, Mu Xu, and Xuan-Jing Huang. 2019. Generating Responses with a Specific Emotion in Dialog. In Proceedings of the 57th Conference of the Association for Computational Linguistics. 3685–3695.
[35]
Carlos Toxtli, Andrés Monroy-Hernández, and Justin Cranshaw. 2018. Understanding chatbot-mediated task management. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–6.
[36]
Jeng-Yi Tzeng and Cheng-Te Chen. 2012. Computer praise, attributional orientations, and games: A reexamination of the CASA theory relative to children. Computers in Human Behavior 28, 6 (2012), 2420–2430.
[37]
Justin D Weisz, Mohit Jain, Narendra Nath Joshi, James Johnson, and Ingrid Lange. 2019. BigBlueBot: teaching strategies for successful human-agent interactions. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 448–459.
[38]
Wierzbicka and Anna. 1992. The semantics of interjection. Journal of Pragmatics 18, 2-3 (1992), 159–192.
[39]
Anna Wierzbicka. 1999. Emotions across Languages and Cultures: Diversity and Universals. Cambridge University Press. https://doi.org/10.1017/CBO9780511521256
[40]
Carl E Williams and Kenneth N Stevens. 1972. Emotions and speech: Some acoustical correlates. The Journal of the Acoustical Society of America 52, 4B (1972), 1238–1250.
[41]
Emily C Willroth, Jayde AM Flett, and Iris B Mauss. 2020. Depressive symptoms and deficits in stress-reactive negative, positive, and within-emotion-category differentiation: A daily diary study. Journal of personality 88, 2 (2020), 174–184.
[42]
Ziang Xiao, Michelle X. Zhou, Q. Vera Liao, Gloria Mark, Changyan Chi, Wenxi Chen, and Huahai Yang. 2020. Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-Ended Questions. ACM Trans. Comput.-Hum. Interact. 27, 3, Article 15 (June 2020), 37 pages. https://doi.org/10.1145/3381804
[43]
Xi Yang, Marco Aurisicchio, and Weston Baxter. 2019. Understanding Affective Experiences with Conversational Agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300772
[44]
Yang Yang, Xiaojuan Ma, and Pascale Fung. 2017. Perceived emotional intelligence in virtual agents. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2255–2262.
[45]
Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2018. Emotional chatting machine: Emotional conversation generation with internal and external memory. In Thirty-Second AAAI Conference on Artificial Intelligence.
[46]
Suping Zhou, Jia Jia, Qi Wang, Yufei Dong, Yufeng Yin, and Kehua Lei. 2018. Inferring emotion from conversational voice data: A semi-supervised multi-path generative neural network approach. In Thirty-Second AAAI Conference on Artificial Intelligence.

Cited By

View all
  • (2024)Do People Mirror Emotion Differently with a Human or TTS Voice? Comparing Listener Ratings and Word EmbeddingsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650757(1-10)Online publication date: 11-May-2024
  • (2024)Perceived Empathy of Technology Scale (PETS): Measuring Empathy of Systems Toward the UserProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642035(1-18)Online publication date: 11-May-2024
  • (2023)The Bot on Speaking Terms: The Effects of Conversation Architecture on Perceptions of Conversational AgentsProceedings of the 5th International Conference on Conversational User Interfaces10.1145/3571884.3597139(1-16)Online publication date: 19-Jul-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems
May 2021
2965 pages
ISBN:9781450380959
DOI:10.1145/3411763
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 May 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. chatbot
  2. conversational agent
  3. emotion
  4. emotional intelligence
  5. voice assistant

Qualifiers

  • Poster
  • Research
  • Refereed limited

Funding Sources

  • Tsinghua University-Alibaba Joint Research Laboratory for Natural Interaction Experience
  • This work was supported by the National Key Research and Development Plan

Conference

CHI '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)181
  • Downloads (Last 6 weeks)9
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Do People Mirror Emotion Differently with a Human or TTS Voice? Comparing Listener Ratings and Word EmbeddingsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650757(1-10)Online publication date: 11-May-2024
  • (2024)Perceived Empathy of Technology Scale (PETS): Measuring Empathy of Systems Toward the UserProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642035(1-18)Online publication date: 11-May-2024
  • (2023)The Bot on Speaking Terms: The Effects of Conversation Architecture on Perceptions of Conversational AgentsProceedings of the 5th International Conference on Conversational User Interfaces10.1145/3571884.3597139(1-16)Online publication date: 19-Jul-2023
  • (2023)The Acoustically Emotion-Aware Conversational Agent With Speech Emotion Recognition and Empathetic ResponsesIEEE Transactions on Affective Computing10.1109/TAFFC.2022.320591914:1(17-30)Online publication date: 1-Jan-2023
  • (2023)Exploring the Use of Colored Ambient Lights to Convey Emotional Cues With Conversational Agents: An Experimental Study2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)10.1109/RO-MAN57019.2023.10309310(99-105)Online publication date: 28-Aug-2023
  • (2022)Empathic and Empathetic Systematic Review to Standardize the Development of Reliable and Sustainable Empathic SystemsSensors10.3390/s2208304622:8(3046)Online publication date: 15-Apr-2022
  • (2022)Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion RecognitionFrontiers in Psychology10.3389/fpsyg.2022.86404713Online publication date: 28-Jun-2022
  • (2022)Emotionally Intelligent Chatbots: A Systematic Literature ReviewHuman Behavior and Emerging Technologies10.1155/2022/96016302022(1-23)Online publication date: 26-Sep-2022
  • (2022)Expressive Auditory Gestures in a Voice-Based Pedagogical AgentProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517599(1-13)Online publication date: 29-Apr-2022
  • (2022)Emotion Recognition in Conversations Using Brain and Physiological SignalsProceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511148(229-242)Online publication date: 22-Mar-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media