Abstract
Analysing and improving chatbot dialogues – so-called chatbot ‘training’ – is key to the successful implementation and maintenance of chatbots for customer service. Nevertheless, the details of this practice and what service providers may learn from the analysis of such dialogues is not investigated in current research on chatbots. As a first step towards bridging this gap in existing knowledge, we present a study of the qualitative analysis of chatbots dialogues in the context of the customer service department of a large telecom provider. In total 406 dialogues, randomly sampled from all chatbot dialogues during a four-week period, were included in the analysis. The analysis concerned the chatbot’s ability to resolve customers’ requests, the quality in the chatbot dialogues, and suggestions for improvements of the chatbot knowledge base generated through the analysis. The findings shed light on characteristics of successful and unsuccessful chatbot dialogues and the kind of improvements that may be derived from such analysis. On the basis of the findings we summarize implications for theory and practice, and suggest future research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Accenture: Chatbots in customer service. Technical report, Accenture (2016). https://www.accenture.com/t00010101T000000__w__/br-pt/_acnmedia/PDF-45/Accenture-Chatbots-Customer-Service.pdf
Chakrabarti, C., Luger, G.F.: Artificial conversations for customer service chatter bots: architecture, algorithms, and evaluation metrics. Expert Syst. Appl. 42(20), 6878–6897 (2015)
Collins, I.: 5 steps to training a chatbot (2016). https://venturebeat.com/2016/11/12/5-steps-to-training-a-chatbot/
Drift: The 2018 State of Chatbots Report. Technical report, Drift (2018), https://www.drift.com/blog/chatbots-report/
Følstad, A., Nordheim, C.B., Bjørkli, C.A.: What makes users trust a chatbot for customer service? An exploratory interview study. In: Bodrunova, S. (ed.) INSCI 2018. LNCS, vol. 11193, pp. 194–208. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01437-7_16
Følstad, A., Skjuve, M.: Chatbots for customer service: user experience and motivation. In: Proceedings of the International Conference on Conversational User Interfaces - CUI 2019. ACM, New York (2019)
Forrester: The Six Factors that Separate Hype from Hope in Your Conversational AI Journey. Technical report, Forrester (2018). https://www.forrester.com/report/The+Six+Factors+That+Separate+Hype+From+Hope+In+Your+Conversational+AI+Journey/-/E-RES143773
Forrester: Human vs. Machines: How to Stop Your Virtual Agent from Lagging Behind. Technical report, Forrester (2017). https://www.amdocs.com/blog/place-digital-talks-intelligent-minds/aia-humans-vs-machines-how-to-stop-your-chatbot-from-lagging-behind
Forrester: The State of Chatbots. Technical report, Forrester (2016). https://www.forrester.com/report/The+State+Of+Chatbots/-/E-RES136207
Gartner: Gartner Says 25 Percent of Customer Service Operations Will Use Virtual Customer Assistants by 2020. https://www.gartner.com/en/newsroom/press-releases/2018-02-19-gartner-says-25-percent-of-customer-service-operations-will-use-virtual-customer-assistants-by-2020
Gnewuch, U., Morana, S., Maedche, A.: Towards designing cooperative and social conversational agents for customer service. In: Proceedings of the International Conference on Information Systems - ICIS 2017 (2017)
Hall, E.: Conversational Design. A Book Apart, New York, NY (2018)
Hu, T., et al.: Touch your heart: a tone-aware chatbot for customer care on social media. In: Proceedings of CHI 2018, paper 415. ACM, New York (2018)
Lester, J., Branting, K., Mott, B.: Conversational agents. In: The Practical Handbook of Internet Computing. Chapman & Hall/CRC, Boca Raton (2004)
Luger, E., Sellen, A.: Like having a really bad PA: the gulf between user expectation and experience of conversational agents. In: Proceedings of CHI 2016, pp. 5286–5297. ACM (2016). https://doi.org/10.1145/2858036.2858288
Mimoun, M.S.B., Poncin, I., Garnier, M.: Case study-Embodied virtual agents: an analysis on reasons for failure. J. Retail. Consum. Serv. 19(6), 605–612 (2012)
Shevat, A.: Designing Bots: Creating Conversational Experiences. O’Reilly Media, Newton (2017)
Wilson, H.J., Daugherty, P.R.: Collaborative intelligence: humans and AI are joining forces. Harvard Bus. Rev. 96(7/8), 114–123 (2018)
Xu, A., Liu, Z., Guo, Y., Sinha, V., Akkiraju, R.: A new chatbot for customer service on social media. In: Proceedings of CHI 2017, pp. 3506–3510. ACM, New York (2017). https://doi.org/10.1145/3025453.3025496
Acknowledgements
The work of the three first authors was supported by Telenor Norway and Telenor Research. The work of the fourth author was supported by the Research Council of Norway through research grant no. 270940.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Kvale, K., Sell, O.A., Hodnebrog, S., Følstad, A. (2020). Improving Conversations: Lessons Learnt from Manual Analysis of Chatbot Dialogues. In: Følstad, A., et al. Chatbot Research and Design. CONVERSATIONS 2019. Lecture Notes in Computer Science(), vol 11970. Springer, Cham. https://doi.org/10.1007/978-3-030-39540-7_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-39540-7_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-39539-1
Online ISBN: 978-3-030-39540-7
eBook Packages: Computer ScienceComputer Science (R0)