Abstract
We organized a competition entitled “the dialogue system live competition” in which the audience, consisting mainly of researchers in the dialogue community, watched and evaluated a live dialogue conducted between users and dialogue systems. The motivation behind the event was to cultivate state-of-the-art techniques in dialogue systems and enable the dialogue community to share the problems with current dialogue systems. There are two parts to the competition: preliminary selection and live event. In the preliminary selection, eleven systems were evaluated by crowd-sourcing. Three systems proceeded to the live event to perform dialogues with designated speakers and to be evaluated by the audience. This paper describes the design and procedure of the competition, the results of the preliminary selection and live event of the competition, and the problems we identified from the event.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
We used Google Form for the questionnaire and the following evaluations.
References
Abe K, Sato S, Sato T, Fujii R, Matsuda K, Suzuki M, Yamaguchi K, Akama R, Ouchi H, Suzuki J, Inui K (2018) Zunkobot: a multiple knowledge bases integrated chatbot system. In: Proceedings of JSAI SIG-SLUD, pp 112–117. (In Japanese)
Black AW, Eskenazi M (2009) The spoken dialogue challenge. In: Proceedings of SIGDIAL, pp 337–340
Higashinaka R, Funakoshi K, Araki M, Tsukahara H, Kobayashi Y, Mizukami M (2015) Towards taxonomy of errors in chat-oriented dialogue systems. In: Proceedings of SIGDIAL, pp 87–95
Higashinaka R, Funakoshi K, Kobayashi Y, Inaba M (2016) The dialogue breakdown detection challenge: task description, datasets, and evaluation metrics. In: Proceedings of LREC, pp 3146–3150
Ritter A, Cherry C, Dolan WB (2011) Data-driven response generation in social media. In: Proceedings of EMNLP, pp 583–593
Shang L, Sakai T, Lu Z, Li H, Higashinaka R, Miyao Y (2016) Overview of the NTCIR-12 short text conversation task. In: Proceedings of NTCIR
Sugiyama H, Narimatsu H, Mizukami M, Arimoto T (2018) Empirical study on domain-specific conversational dialogue system based on context-aware utterance understanding and generation. In: Proceedings of JSAI SIG-SLUD, pp 118–123. (In Japanese)
Tsunomori Y, Onishi K, Fujimoto H, Kadono K, Yoshimura T, Isoda, Y (2018) Development of a customizable open domain chat-oriented dialogue system. In: Proceedings of JSAI SIG-SLUD, pp 124–127. (In Japanese)
Williams J, Raux A, Ramachandran D, Black A (2013) The dialog state tracking challenge. In: Proceedings of SIGDIAL, pp 404–413
Acknowledgements
The authors would like to thank the Japanese society for artificial intelligence (JSAI) for financially supporting the competition. We also thank the teams who participated in the competition and those who evaluated the dialogues during the live event.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Higashinaka, R., Funakoshi, K., Inaba, M., Tsunomori, Y., Takahashi, T., Akama, R. (2021). Dialogue System Live Competition: Identifying Problems with Dialogue Systems Through Live Event. In: Marchi, E., Siniscalchi, S.M., Cumani, S., Salerno, V.M., Li, H. (eds) Increasing Naturalness and Flexibility in Spoken Dialogue Interaction. Lecture Notes in Electrical Engineering, vol 714. Springer, Singapore. https://doi.org/10.1007/978-981-15-9323-9_16
Download citation
DOI: https://doi.org/10.1007/978-981-15-9323-9_16
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-9322-2
Online ISBN: 978-981-15-9323-9
eBook Packages: EngineeringEngineering (R0)