Abstract
Dialog state tracking is one of the key sub-tasks of dialog management , which defines the representation of dialog states and updates them at each moment on a given on-going conversation. To provide a common test bed for this task, three dialog state tracking challenges have been completed. In this fourth challenge, we focused on dialog state tracking on human-human dialogs. The challenge received a total of 24 entries from 7 research groups. Most of the submitted entries outperformed the baseline tracker based on string matching with ontology contents. Moreover, further significant improvements in tracking performances were achieved by combining the results from multiple trackers. In addition to the main task, we also conducted pilot track evaluations for other core components in developing modular dialog systems using the same dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Williams, J., Raux, A., Ramachandran, D., Black, A.: The dialog state tracking challenge. In: Proceedings of the SIGDIAL 2013 Conference, pp. 404–413 (2013)
Henderson, M., Thomson, B., Williams, J.: The second dialog state tracking challenge. In: Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, p. 263 (2014)
Henderson, M., Thomson, B., Williams, J.D.: The third dialog state tracking challenge. In: Proceedings of the 2014 IEEE Spoken Language Technology Workshop (SLT), pp. 324–329. IEEE (2014)
Kim, S., D’Haro, L.F., Banchs, R.E., Williams, J., Henderson, M.: Dialog state tracking challenge 4 handbook. http://www.colips.org/workshop/dstc4/Handbook_DSTC4.pdf (2015)
Smith, R.W.: Comparative error analysis of dialog state tracking. In: Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, p. 300 (2014)
Lee, S., Eskenazi, M.: Recipe for building robust spoken dialog state trackers: dialog state tracking challenge system description. In: Proceedings of the SIGDIAL 2013 Conference, pp. 414–422 (2013)
Yeh, A.: More accurate tests for the statistical significance of result differences. In: Proceedings of the 18th Conference on Computational linguistics, vol. 2. pp. 947–953. Association for Computational Linguistics (2000)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational linguistics, pp. 311–318. Association for Computational Linguistics (2002)
Banchs, R.E., D’Haro, L.F., Li, H.: Adequacy-fluency metrics: evaluating mt in the continuous space model framework. IEEE/ACM Trans. Audio Speech Lang. Process. 23(3), 472–482 (2015)
Kim, S., D’Haro, L.F., Banchs, R.E., Williams, J., Henderson, M.: Dialog state tracking challenge 4 pilot task guidelines. http://www.colips.org/workshop/dstc4/DSTC4_pilot_tasks.pdf (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer Science+Business Media Singapore
About this chapter
Cite this chapter
Kim, S., D’Haro, L.F., Banchs, R.E., Williams, J.D., Henderson, M. (2017). The Fourth Dialog State Tracking Challenge. In: Jokinen, K., Wilcock, G. (eds) Dialogues with Social Robots. Lecture Notes in Electrical Engineering, vol 427. Springer, Singapore. https://doi.org/10.1007/978-981-10-2585-3_36
Download citation
DOI: https://doi.org/10.1007/978-981-10-2585-3_36
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-2584-6
Online ISBN: 978-981-10-2585-3
eBook Packages: EngineeringEngineering (R0)