Abstract
In previous studies, top-down and bottom-up approaches have been proposed for creating taxonomies of errors in chat-oriented dialogue systems. However, the reported \(\kappa \) (kappa) value for the taxonomy based on the top-down approach is low at 0.239, and no evaluation has been conducted for that based on the bottom-up approach. In this paper, we propose to revise these taxonomies to achieve better inter-annotator agreement. The revised taxonomy based on the bottom-up approach yielded a reasonable \(\kappa \) of 0.44 (Fleiss’ \(\kappa \)), suggesting that this taxonomy can be used reliably to classify errors in chat-oriented dialogue systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Banchs RE, Li H (2012) IRIS: a chat-oriented dialogue system based on the vector space model. In: Proceedings of the ACL 2012 System Demonstrations, pp 37–42
Bernsen NO, Dybkjær H, Dybkjær L (1996) Principles for the design of cooperative spoken human-machine dialogue. In: Proceedings of the ICSLP, vol 2, pp 729–732
Bohus D, Rudnicky AI (2005) Sorry, i didn’t catch that!–an investigation of non-understanding errors and recovery strategies. In: Proceedings of the SIGDIAL, pp 128–143
Clark HH (1996) Using language. Cambridge University Press, Cambridge
Dybkjær L, Bernsen NO, Dybkjær H (1996) Grice incorporated: cooperativity in spoken dialogue. In: Proceedings of the COLING, vol 1, pp 328–333
Grice HP (1975) Logic and conversation. In: Cole P, Morgan J (eds) Syntax and semantics 3: speech acts. Academic Press, New York, pp 41–58
Higashinaka R, Funakoshi K, Araki M, Tsukahara H, Kobayashi Y, Mizukami M (2015) Towards taxonomy of errors in chat-oriented dialogue systems. In: Proceedings of the SIGDIAL, pp 87–95
Higashinaka R, Funakoshi K, Kobayashi Y, Inaba M (2016) The dialogue breakdown detection challenge: Task description, datasets, and evaluation metrics. In: Proceedings of the LREC, pp 3146–3150
Higashinaka R, Imamura K, Meguro T, Miyazaki C, Kobayashi N, Sugiyama H, Hirano T, Makino T, Matsuo Y (2014) Towards an open-domain conversational system fully based on natural language processing. In: Proceedings of the COLING, pp 928–939
Higashinaka R, Mizukami M, Funakoshi K, Araki M, Tsukahara H, Kobayashi Y (2015) Fatal or not? finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems. In: Proceedings of the EMNLP, pp 2243–2248
Martinovsky B, Traum D (2003) The error is the clue: breakdown in human-machine interaction. In: Proceedings of the ISCA Workshop on Error Handling in Spoken Dialogue Systems, pp 11–16
Möller S, Engelbrecht KP, Oulasvirta A (2007) Analysis of communication failures for spoken dialogue systems. In: Proceedings of the INTERSPEECH, pp 134–137
Onishi K, Yoshimura T (2014) Casual conversation technology achieving natural dialog with computers. NTT DOCOMO Tech J 15(4):16–21
Paek T (2003) Toward a taxonomy of communication errors. In: Proceedings of the ISCA Workshop on Error Handling in Spoken Dialogue Systems, pp 53–58
Ritter A, Cherry C, Dolan WB (2011) Data-driven response generation in social media. In: Proceedings of the EMNLP, pp 583–593. Association for Computational Linguistics
Tsukahara H, Uchiumi K (2015) System utterance generation by label propagation over association graph of words and utterance patterns for open-domain dialogue systems. In: Proceedings of the PACLIC, pp 323–331
Vinyals O, Le Q (2015) A neural conversational model. arXiv:1506.05869
Wallace RS (2009) The anatomy of alice. In: Parsing the turing test. Springer, Berlin, pp 181–210
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Higashinaka, R., Araki, M., Tsukahara, H., Mizukami, M. (2019). Improving Taxonomy of Errors in Chat-Oriented Dialogue Systems. In: D'Haro, L., Banchs, R., Li, H. (eds) 9th International Workshop on Spoken Dialogue System Technology. Lecture Notes in Electrical Engineering, vol 579. Springer, Singapore. https://doi.org/10.1007/978-981-13-9443-0_29
Download citation
DOI: https://doi.org/10.1007/978-981-13-9443-0_29
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-9442-3
Online ISBN: 978-981-13-9443-0
eBook Packages: Literature, Cultural and Media StudiesLiterature, Cultural and Media Studies (R0)