short-paper

Trends & Methods in Chatbot Evaluation

Authors:

Marc-Olivier Tricot,

Omar Abou Khaled,

Elena Mugellini,

Philippe Cudré-MaurouxAuthors Info & Claims

ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction

Pages 280 - 286

https://doi.org/10.1145/3395035.3425319

Published: 27 December 2020 Publication History

Abstract

Chatbots are computer programs aiming to replicate human conversational abilities through voice exchanges, textual dialogues, or both. They are becoming increasingly pervasive in many domains like customer support, e-coaching or entertainment. Yet, there is no standardised way of measuring the quality of such virtual agents. Instead, multiple individuals and groups have established their own standards either specifically for their chatbot project or have taken some inspiration from other groups. In this paper, we make a review of current techniques and trends in chatbot evaluation. We examine chatbot evaluation methodologies and assess them according to the ISO 9214 concepts of usability: Effectiveness, Efficiency and Satisfaction. We then analyse the methods used in the literature from 2016 to 2020 and compare their results. We identify a clear trend towards evaluating the efficiency of chatbots in many recent papers, which we link to the growing popularity of task-based chatbots that are currently being deployed in many business contexts.

References

[1]

Alain Abran, Adel Khelifi, Witold Suryn, and Ahmed Seffah. 2003. Usability Meanings and Interpretations in ISO Standards. Software Quality Journal, Vol. 11 (11 2003), 325--338. https://doi.org/10.1023/A:1025869312943

Digital Library

[2]

Bayan AbuShawar and Eric Atwell. 2016. Usefulness, localizability, humanness, and language-benefit: additional evaluation criteria for natural language dialogue systems. International Journal of Speech Technology, Vol. 19, 2 (2016), 373--383.

Digital Library

[3]

Martin Adam, Michael Wessel, and Alexander Benlian. 2020. AI-based chatbots in customer service and their effects on user compliance. Electronic Markets (2020), 1--19.

[4]

Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977 (2020).

[5]

Dana Al-Ghadhban and Nora Al-Twairesh. [n.d.]. Nabiha: An Arabic Dialect Chatbot. ( [n.,d.]).

[6]

Dana Abu Ali and Nizar Habash. 2016. Botta: An arabic dialect chatbot. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations. 208--212.

[7]

Leonardo Angelini, Maurizio Caon, Jacky Casas, Federica Cena, Amon Rapp, Omar Abou Khaled, and Elena Mugellini. 2018. Ubiquitous Chatbots: Workshop on Wearable and Embodied Conversational Agents. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers (Singapore, Singapore) (UbiComp '18). Association for Computing Machinery, New York, NY, USA, 1652--1655. https://doi.org/10.1145/3267305.3274146

Digital Library

[8]

Theo Araujo. 2018. Living up to the chatbot hype: The influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptions. Computers in Human Behavior, Vol. 85 (03 2018). https://doi.org/10.1016/j.chb.2018.03.051

[9]

V. Arulmangainayaki, M. Harini, M. Keerthiga, and M. Deva Priya. 2020. Intelligent Chatbot for Medical Assistance in Rural Areas. nternational Journal of Innovative Technology and Exploring Engineering, Vol. 9 (2020).

[10]

Ayah Atiyah, Shaidah Jusoh, and Firas Alghanim. 2019. Evaluation of the Naturalness of Chatbot Applications. In 2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT). IEEE, 359--365.

[11]

Eva Bittner and Omid Shoury. 2019. Designing Automated Facilitation for Design Thinking: A Chatbot for Supporting Teams in the Empathy Map Method. In HICSS .

[12]

Daniel Braun and Florian Matthes. 2019. Towards a Framework for Classifying Chatbots. In Proceedings of the 21th International Conference on Enterprise Information Systems, Vol. 1.

[13]

Joseph-Peter Yoofi Brown-Pobee. 2019. Automating Chalkboard Support Processes using a Chatbot. Bsc Thesis. Ashesi University College, Berekuso, Ghana.

[14]

Jacky Casas. 2018. The state of conversational agents: a 2017--2018 review .

[15]

Jacky Casas, Elena Mugellini, and Omar Abou Khaled. 2018. Food Diary Coaching Chatbot. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers (Singapore, Singapore) (UbiComp '18). Association for Computing Machinery, New York, NY, USA, 1676--1680. https://doi.org/10.1145/3267305.3274191

Digital Library

[16]

Chatbots.org. 2018. Consumers Say No to Chatbot Silos in US and UK Survey. https://www.chatbots.org/images/news/chatbot_survey_2018.pdf [Online; accessed 24-January-2020].

[17]

Neasa N'i Chiaráin and Ailbhe N'i Chasaide. 2016. Chatbot Technology with Synthetic Voices in the Acquisition of an Endangered Language: Motivation, Development and Evaluation of a Platform for Irish. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16). European Language Resources Association (ELRA), Portorovz, Slovenia, 3429--3435. https://www.aclweb.org/anthology/L16--1547

[18]

Sahil Chopra, Rachel Gianforte, and John Sholar. [n.d.]. Meet Percy: The CS 221 Teaching Assistant Chatbot. ( [n.,d.]).

[19]

Minjee Chung, Eunju Ko, Heerim Joung, and Sang Kim. 2018. Chatbot e-service and customer satisfaction regarding luxury brands. Journal of Business Research (11 2018). https://doi.org/10.1016/j.jbusres.2018.10.004

[20]

K. Coperich, Elizabeth A. Cudney, and H. B. Nembhard. [n.d.]. Continuous Improvement Study of Chatbot Technologies using a Human Factors Methodology.

[21]

Lei Cui, Shaohan Huang, Furu Wei, Chuanqi Tan, Chaoqun Duan, and Ming Zhou. 2017. SuperAgent: A Customer Service Chatbot for E-commerce Websites. In Proceedings of ACL 2017, System Demonstrations. Association for Computational Linguistics, Vancouver, Canada, 97--102. https://www.aclweb.org/anthology/P17--4017

[22]

Daniëlle Duijst. 2017. Can we Improve the User Experience of Chatbots with Personalisation? Ph.D. Dissertation. University of Amsterdam, Amsterdam, Holland. https://doi.org/10.13140/RG.2.2.36112.92165

[23]

Mira El Kamali, Leonardo Angelini, Maurizio Caon, Giuseppe Andreoni, Omar Abou Khaled, and Elena Mugellini. 2018. Towards the NESTORE E-Coach: A Tangible and Embodied Conversational Agent for Older Adults. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers (Singapore, Singapore) (UbiComp '18). Association for Computing Machinery, New York, NY, USA, 1656--1663. https://doi.org/10.1145/3267305.3274188

Digital Library

[24]

Endang Wahyu Pamungkas. 2019. Emotionally-Aware Chatbots: A Survey.

[25]

Erwin Van Lun. 2020. Chatbots.org - Chatbot Reviews, Community & News. chatbots.org [Online; accessed 8-January-2020].

[26]

Cheong Sik Feng, Lim Jing, Chieu Hai Leong, and Wong Jialiang Joshua. [n.d.]. QUORA QUESTION ANSWERING CHATBOT. ( [n.,d.]).

[27]

Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2014. The Rise of Social Bots. Commun. ACM, Vol. 59 (07 2014). https://doi.org/10.1145/2818717

Digital Library

[28]

Boris Galitsky and Dmitry Ilvovsky. 2017. Chatbot with a Discourse Structure-Driven Dialogue Management. In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Valencia, Spain, 87--90. https://www.aclweb.org/anthology/E17--3022

[29]

Boris Galitsky and Dmitry Ilvovsky. 2019. On a Chatbot Conducting Virtual Dialogues. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (Beijing, China) (CIKM '19). Association for Computing Machinery, New York, NY, USA, 2925--2928. https://doi.org/10.1145/3357384.3357842

Digital Library

[30]

gk_. 2017. Soul of the Machine: How Chatbots Work. http://web.archive.org/web/20190202142330/https://medium.com/@gk_/how-chat-bots-work-dfff656a35e2

[31]

Ulrich Gnewuch, Stefan Morana, Marc Adam, and Alexander Maedche. 2018. Faster is not always better: understanding the effect of dynamic response delays in human-chatbot interaction. (2018).

[32]

H. James Wilson, Paul R. Daugherty, and Nicola Morini-Bianzino. 2017. The Jobs That Artificial Intelligence Will Create. http://sloanreview.mit.edu/article/will-ai-create-as-many-jobs-as-it-eliminates/ [Online; accessed 14-January-2020].

[33]

Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. 2019. Learning from Dialogue after Deployment: Feed Yourself, Chatbot! Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/p19--1358

[34]

Mohammad Hijjawi, Zuhair Bandar, and Keeley Crockett. 2016. A general evaluation framework for text based conversational agent. International Journal of Advanced Computer Science and Applications, Vol. 7 (2016).

[35]

Vladimir Ilievski, Claudiu Musat, Andreea Hossmann, and Michael Baeriswyl. 2018. Goal-oriented chatbot dialog management bootstrapping with transfer learning. arXiv preprint arXiv:1802.00500 (2018).

Digital Library

[36]

Lia Maulia Indrayani, Rosaria Mita Amalia, and Fauzia Zahira Munirul Hakim. 2020. Emotive Expressions on Social Chatbot. Jurnal Sosioteknologi, Vol. 18, 3 (2020), 509--516.

[37]

Mahipal Jadeja and Neelanshi Varia. 2017. Perspectives for Evaluating Conversational AI. ArXiv, Vol. abs/1709.04734 (2017).

[38]

Mohit Jain, Pratyush Kumar, Ramachandra Kota, and Shwetak Patel. 2018. Evaluating and Informing the Design of Chatbots. 895--906. https://doi.org/10.1145/3196709.3196735

Digital Library

[39]

Grishma Jena, Mansi Vashisht, Abheek Basu, Lyle H. Ungar, and Jo ao Sedoc. 2017. Enterprise to Computer: Star Trek chatbot. ArXiv, Vol. abs/1708.00818 (2017).

[40]

Jordan Bryan. 2018. 4 Trends in Gartner Hype Cycle for Customer Service and Customer Engagement. https://www.gartner.com/smarterwithgartner/4-trends-gartner-hype-cycle-customer-service-customer-engagement/ [Online; accessed 8-January-2020].

[41]

Stefanie Maria Jungmann, Timo Klan, Sebastian Kuhn, and Florian Jungmann. 2019. Accuracy of a Chatbot (Ada) in the Diagnosis of Mental Disorders: Comparative Case Study With Lay and Expert Users. JMIR Formative Research, Vol. 3 (2019).

[42]

Lucia Krisnawati, Bill Butar-Butar, and Gloria Virginia. 2018. Prototyping a Chatbot for Student Supervision in a Pre-Registration Process. CommIT (Communication and Information Technology) Journal, Vol. 12 (10 2018), 87. https://doi.org/10.21512/commit.v12i2.4813

[43]

Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74--81. https://www.aclweb.org/anthology/W04--1013

[44]

Pedro Manuel Dias Braga Lino. 2018. Travel Booking Chatbot.

[45]

Qingtang Liu, Jingxiu Huang, Linjing Wu, Ke Min Zhu, and Shen Ba. 2019. CBET: design and evaluation of a domain-specific chatbot for mobile learning. Universal Access in the Information Society (2019), 1--19.

[46]

Andreas Lommatzsch and Jonas Katins. 2019. An Information Retrieval-based Approach for Building Intuitive Chatbots for Large Knowledge Bases. In LWDA .

[47]

Wari Maroengsit, Thanarath Piyakulpinyo, Korawat Phonyiam, Suporn Pongnumkul, Pimwadee Chaovalit, and Thanaruk Theeramunkong. 2019. A Survey on Evaluation Methods for Chatbots. In Proceedings of the 2019 7th International Conference on Information and Education Technology (Aizu-Wakamatsu, Japan) (ICIET 2019). Association for Computing Machinery, New York, NY, USA, 111--119. https://doi.org/10.1145/3323771.3323824

Digital Library

[48]

Mitch Mason. 2017. 3 types of business chatbots you can build. http://web.archive.org/web/20190814211438/https://www.ibm.com/blogs/watson/2017/12/3-types-of-business-chatbots-you-can-build/

[49]

Sonja Merisalo. 2018. Developing a Chatbot for Customer Service to Metropolia UAS Student Affairs Office. Master Thesis. Metropolia University of Applied Sciences, Helsinki, Finland.

[50]

Amit Mittal, Ayushi Agrawal, Ayushi Chouksey, Rachna Shriwas, and Saloni Agrawal. 2016. A comparative Study of Chatbots and Humans. Situations, Vol. 2 (2016), 2.

[51]

Jay Morgan, Adeline Paiement, Monika Seisenberger, Jane Williams, and Adam Zachary Wyner. 2018. A Chatbot Framework for the Children's Legal Centre. In JURIX .

[52]

Khalil Mrini, Marc Laperrouza, and Pierre Dillenbourg. 2018. Building a Question-Answering Chatbot using Forum Data in the Semantic Space.

[53]

Sana Mujeeb, Muhammad Hafeez Javed, and Tayyaba Arshad. 2017. Aquabot: a diagnostic chatbot for achluophobia and autism. Int J Adv Comput Sci Appl, Vol. 8 (2017).

[54]

Huyen Nguyen, David Morales, and Tessera Chin. 2017. A neural chatbot with personality.

[55]

Duvsan Okanović, Samuel Beck, Lasse Merz, Christoph Zorn, Leonel Merino, André van Hoorn, and Fabian Beck. 2020. Can a Chatbot Support Software Engineers with Load Testing? Approach and Experiences. In Proceedings of the ACM/SPEC International Conference on Performance Engineering (Edmonton AB, Canada) (ICPE '20). Association for Computing Machinery, New York, NY, USA, 120--129. https://doi.org/10.1145/3358960.3375792

Digital Library

[56]

Kulothunkan Palasundram, Nurfadhlina Sharef, Nurul Amelina Nasharuddin, Khairul Kasmiran, and Azreen Azman. 2019. Sequence to Sequence Model Performance for Education Chatbot. International Journal of Emerging Technologies in Learning (iJET), Vol. 14 (12 2019), 56. https://doi.org/10.3991/ijet.v14i24.12187

[57]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 311--318. https://doi.org/10.3115/1073083.1073135

Digital Library

[58]

Jorge Peniche-Avilés, Carlos Miranda-Palma, Lizzie Narváez-D'iaz, and Erika Llanes-Castro. 2016. AluxBot-A Chatbot that Encourages the Care for the Environment. International Journal of Computer Science Issues (IJCSI), Vol. 13, 6 (2016), 120.

[59]

Stefanus Ardhito Prasetya, Alva Erwin, and Maulahikmah Galinium. 2018. IMPLEMENTING INDONESIAN LANGUAGE CHATBOT FOR ECOMMERCE SITE USING ARTIFICIAL INTELLIGENCE MARKUP LANGUAGE (AIML).

[60]

Minghui Qiu, Feng-Lin Li, Siyu Wang, Xing Gao, Yan Chen, Weipeng Zhao, Haiqing Chen, Jun Huang, and Wei Chu. 2017. AliMe Chat: A Sequence to Sequence and Rerank based Chatbot Engine. 498--503. https://doi.org/10.18653/v1/P17--2079

[61]

Nicole Radziwill and Morgan Benton. 2017. Evaluating Quality of Chatbots and Intelligent Conversational Agents. (04 2017).

[62]

Rick Ramos. 2017. Screw the Turing test ? chatbots don't need to act human. https://venturebeat.com/2017/02/03/screw-the-turing-test-chatbots-dont-need-to-act-human/ [Online; accessed 14-January-2020].

[63]

Iulian Serban, Chinnadhurai Sankar, Mathieu Germain, Saizheng Zhang, Zhouhan Lin, Sandeep Subramanian, Taesup Kim, Michael Pieper, A. P. Sarath Chandar, Nan Rosemary Ke, Sai Mudumba, Alexandre de Brébisson, Jose Sotelo, Dendi Suhubdy, Vincent Michalski, Alexandre Nguyen, Joelle Pineau, and Yoshua Bengio. 2017. A Deep Reinforcement Learning Chatbot. ArXiv, Vol. abs/1709.02349 (2017).

[64]

Pavel Smutny and Petra Schreiberova. 2020. Chatbots for learning: A review of educational chatbots for the Facebook Messenger. Computers & Education (2020), 103862.

[65]

Giancarlo Sperl'i. 2020. A Deep Learning Based Chatbot for Cultural Heritage. In Proceedings of the 35th Annual ACM Symposium on Applied Computing (Brno, Czech Republic) (SAC ?20). Association for Computing Machinery, New York, NY, USA, 935--937. https://doi.org/10.1145/3341105.3374129

Digital Library

[66]

Timo Spring, Jacky Casas, Karl Daher, Elena Mugellini, and Omar Abou Khaled. 2019. Empathic Response Generation in Chatbots.

[67]

Ciaran Staunton. 2018. NowBot - Artificially Intelligent Chatbot. Bsc Thesis. National College of Ireland, Dublin, Ireland.

[68]

Stefan Kostarelis. 2018. Should we be making chatbots that will pass the Turing Test? https://www.fifthquadrant.com.au/cx-spotlight-news/should-we-be-making-chatbots-that-will-pass-the-turing-test [Online; accessed 14-January-2020].

[69]

Sebastian Stephan. 2017. Die Arten von Chatbots. http://web.archive.org/web/20200511161241/https://cloud-blog.arvato.com/die-arten-von-chatbots/

[70]

Tata Consultancy Services. 2017. How 13 Global Industries Use Artificial Intelligence. https://sites.tcs.com/artificial-intelligence/wp-content/uploads/TCS-GTS-how-13-global-industries-use-artificial-intelligence.pdf [Online - PDF; accessed 23-January-2020].

[71]

Navid Tavanapour and Eva Bittner. 2018. Automated Facilitation for Idea Platforms: Design and Evaluation of a Chatbot Prototype.

[72]

Carlos Toxtli, Andrés Monroy-Hernández, and Justin Cranshaw. 2018. Understanding chatbot-mediated task management. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1--6.

Digital Library

[73]

Milena Tsvetkova, Ruth García-Gavilanes, Luciano Floridi, and Taha Yasseri. 2017. Even good bots fight: The case of Wikipedia. In PloS one .

[74]

Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, Rahul Goel, Shaohua Yang, and Anirudh Raju. 2018. On Evaluating and Comparing Conversational Agents. (01 2018).

[75]

Joseph Weizenbaum. 1966. ELIZA?a Computer Program for the Study of Natural Language Communication between Man and Machine. Commun. ACM, Vol. 9, 1 (Jan. 1966), 36--45. https://doi.org/10.1145/365153.365168

Digital Library

[76]

Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, and Rama Akkiraju. 2017. A New Chatbot for Customer Service on Social Media. https://doi.org/10.1145/3025453.3025496

Digital Library

[77]

Zi Yin, Keng hao Chang, and Ruofei Zhang. 2017. DeepProbe: Information Directed Sequence Understanding and Chatbot Design via Recurrent Neural Networks. In KDD '17 .

Digital Library

[78]

Zhou Yu, Ziyu Xu, Alan W Black, and Alexander Rudnicky. 2016. Chatbot evaluation and database expansion via crowdsourcing. In Proceedings of the chatbot workshop of LREC, Vol. 63. 102.

[79]

Jennifer Zamora. 2017. I'm sorry, Dave, I'm afraid I can't do that: Chatbot perception and expectations. In Proceedings of the 5th International Conference on Human Agent Interaction. 253--260.

Digital Library

[80]

Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020. The design and implementation of xiaoice, an empathetic social chatbot. Computational Linguistics, Vol. 46, 1 (2020), 53--93.

Digital Library

Cited By

Singh SNamin A(2025)A survey on chatbots and large language models: Testing and evaluation techniquesNatural Language Processing Journal10.1016/j.nlp.2025.10012810(100128)Online publication date: Mar-2025
https://doi.org/10.1016/j.nlp.2025.100128
Xu YGao WWang YShan XLin Y(2024)Enhancing user experience and trust in advanced LLM-based conversational agentsComputing and Artificial Intelligence10.59400/cai.v2i2.14672:2(1467)Online publication date: 17-Aug-2024
https://doi.org/10.59400/cai.v2i2.1467
Qamar RZardari B(2024)Applications Areas, Possibilities, and Limitations of ChatGPTApplications, Challenges, and the Future of ChatGPT10.4018/979-8-3693-6824-4.ch004(75-90)Online publication date: 28-May-2024
https://doi.org/10.4018/979-8-3693-6824-4.ch004
Show More Cited By

Index Terms

Trends & Methods in Chatbot Evaluation

Recommendations

Understanding is a Two-Way Street: User-Initiated Repair on Agent Responses and Hearing in Conversational Interfaces
CSCW

Although methods for repairing prior turns in natural conversation are critical for enabling mutual understanding, or successful communication, these methods are seldom built into conversational user interfaces systematically. Chatbots and voice ...
Conversational Agents: Acting on the Wave of Research and Development
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

In the last five years, work on software that interacts with people via typed or spoken natural language, called chatbots, intelligent assistants, social bots, virtual companions, non-human players, and so on, increased dramatically. Chatbots burst into ...
Comparing How a Chatbot References User Utterances from Previous Chatting Sessions: An Investigation of Users' Privacy Concerns and Perceptions
HAI '23: Proceedings of the 11th International Conference on Human-Agent Interaction

Chatbots are capable of remembering and referencing previous conversations, but does this enhance user engagement or infringe on privacy? To explore this trade-off, we investigated the format of how a chatbot references previous conversations with a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction

October 2020

548 pages

ISBN:9781450380027

DOI:10.1145/3395035

General Chairs:
Khiet Truong
University of Twente, the Netherlands
,
Dirk Heylen
University of Twente, the Netherlands
,
Mary Czerwinski
Microsoft Research, USA
,
Program Chairs:
Nadia Berthouze
University College London, United Kingdom
,
Mohamed Chetouani
Sorbonne University, France
,
Mikio Nakano
C4A Research Institute, Japan

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 December 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

ICMI '20

Sponsor:

SIGCHI

ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 25 - 29, 2020

Virtual Event, Netherlands

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
2,050
Total Downloads

Downloads (Last 12 months)514
Downloads (Last 6 weeks)45

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Singh SNamin A(2025)A survey on chatbots and large language models: Testing and evaluation techniquesNatural Language Processing Journal10.1016/j.nlp.2025.10012810(100128)Online publication date: Mar-2025
https://doi.org/10.1016/j.nlp.2025.100128
Xu YGao WWang YShan XLin Y(2024)Enhancing user experience and trust in advanced LLM-based conversational agentsComputing and Artificial Intelligence10.59400/cai.v2i2.14672:2(1467)Online publication date: 17-Aug-2024
https://doi.org/10.59400/cai.v2i2.1467
Qamar RZardari B(2024)Applications Areas, Possibilities, and Limitations of ChatGPTApplications, Challenges, and the Future of ChatGPT10.4018/979-8-3693-6824-4.ch004(75-90)Online publication date: 28-May-2024
https://doi.org/10.4018/979-8-3693-6824-4.ch004
Şimşek A(2024)Optimizing Customer Service With ChatbotsIntelligent Optimization Techniques for Business Analytics10.4018/979-8-3693-1598-9.ch010(236-254)Online publication date: 26-Apr-2024
https://doi.org/10.4018/979-8-3693-1598-9.ch010
Zota RCîmpeanu IDragomir DLungu M(2024)Practical Approach for Smart and Circular Cities: Chatbots Used in Waste RecyclingApplied Sciences10.3390/app1407306014:7(3060)Online publication date: 5-Apr-2024
https://doi.org/10.3390/app14073060
Mubassira THasan MNoor JIslam A(2024)Enhancing EmoBot: An In-Depth Analysis of User Expectation and Satisfaction in an Emotion-Aware ChatbotProceedings of the 11th International Conference on Networking, Systems, and Security10.1145/3704522.3704527(65-71)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3704522.3704527
He LBraggaar ABasar EKrahmer EAntheunis MWiers R(2024)Exploring User Engagement Through an Interaction Lens: What Textual Cues Can Tell Us about Human-Chatbot InteractionsProceedings of the 6th ACM Conference on Conversational User Interfaces10.1145/3640794.3665536(1-14)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3640794.3665536
Tamime RSalminen JJung SJansen B(2024)Evaluating LLM-Generated Topics from Survey Responses: Identifying Challenges in Recruiting Participants through Crowdsourcing2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC60511.2024.00064(412-416)Online publication date: 2-Sep-2024
https://doi.org/10.1109/VL/HCC60511.2024.00064
Kuo ESu Y(2024)Assembling Fragmented Domain Knowledge: A LLM-Powered QA System for Taiwan Cinema2024 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC60901.2024.10612108(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/CEC60901.2024.10612108
Rese ATränkner P(2024)Perceived conversational ability of task-based chatbots – Which conversational elements influence the success of text-based dialogues?International Journal of Information Management: The Journal for Information Professionals10.1016/j.ijinfomgt.2023.10269974:COnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.ijinfomgt.2023.102699
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten