Abstract
This paper evaluates the performance of three different chatbots: IRIS, TickTock and Joker, that have been made available to the public online. All three retrieval-based dialogue systems are chat-oriented and designed to engage the users into all types of conversations for as long as possible. They employ different approaches to provide relevant and valid responses, and constantly utilize conversational strategies to further automatically improve its own system through machine learning. The analysis of annotations of more than 2000 responses for the three chatbots allowed us to confirm the robustness, scalability and usability of the systems, as well as to detect a few areas in which response accuracy was lacking, and propose future work to further improve the three systems and annotations scheme.
Luis Fernando D’Haro—Research work done while working at Institute for Infocomm Research, Singapore.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Banchs RE, Li H (2012) IRIS: a chat-oriented dialogue system based on the vector space model. In: Proceedings of the ACL 2012 system demonstrations. Association for Computational Linguistics, pp 37–42
Duplessis D, Letard V, Ligozat AL, Rosset S (2016) Joker chatterbot re-wochat 2016-shared task chatbot description report. In: RE-WOCHAT: workshop on collecting and generating resources for chatbots and conversational agents-development and evaluation workshop programme, 28 May 2016, p 45
Yu Z, Papangelis A, Rudnicky A (2015) TickTock: a non-goal-oriented multimodal dialog system with engagement awareness. In: Proceedings of the AAAI spring symposium
Acknowledgements
We would like to thank Dr. Rafael E. Banchs from Nanyang Technological University for providing us the chatbots data used in this project, and Mr. Kester Wong from National Junior College for providing this valuable opportunity to learn about Infocomm Technology that as students would otherwise not have been able to such depths.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Kong-Vega, N., Shen, M., Wang, M., D’Haro, L.F. (2019). Subjective Annotation and Evaluation of Three Different Chatbots WOCHAT: Shared Task Report. In: D'Haro, L., Banchs, R., Li, H. (eds) 9th International Workshop on Spoken Dialogue System Technology. Lecture Notes in Electrical Engineering, vol 579. Springer, Singapore. https://doi.org/10.1007/978-981-13-9443-0_32
Download citation
DOI: https://doi.org/10.1007/978-981-13-9443-0_32
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-9442-3
Online ISBN: 978-981-13-9443-0
eBook Packages: Literature, Cultural and Media StudiesLiterature, Cultural and Media Studies (R0)