Abstract
While example-based dialog is a popular option for the construction of dialog systems, creating example bases for a specific task or domain requires significant human effort. To reduce this human effort, in this paper, we propose an active learning framework to construct example-based dialog systems efficiently. Specifically, we propose two uncertainty sampling strategies for selecting inputs to present to human annotators who create system responses for the selected inputs. We compare performance of these proposed strategies with a random selection strategy in simulation-based evaluation on 6 different domains. Evaluation results show that the proposed strategies are good alternatives to random selection in domains where the complexity of system utterances is low.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
Source files to replicate these experiments are available: https://github.com/TakuyaHiraoka/Active-Learning-for-Example-based-Dialog-Systems.
- 3.
We use dialog logs collected from http://www.cleverbot.com/j2convbydate-page1.
- 4.
- 5.
In our research, we use previous dialog act and slot filling status [4] as semantic and discourse features.
References
Murao, H., Kawaguchi, N., Matsubara, S., Inagaki, Y.: Example-based query generation for spontaneous speech. In: Proceedings of ASRU (2001)
Murao, H., Kawaguchi, N., Matsubara, S., Yamaguchi, Y., Inagaki, Y.: Example-based spoken dialogue system using WOZ system log. In: Proceeding of SIGDIAL (2003)
Nisimura, R., Nishihara, Y., Tsurumi, R., Lee, A., Saruwatari, H., Shikano, K.: Takemaru-kun: Speech-oriented information system for real world research platform. In: Proceedings of LUAR (2003)
Lee, C., Jung, S., Kim, S., Lee, G.G.: Example-based dialog modeling for practical multi-domain dialog system. Speech Commun. (2009)
Xue, X., Jeon, J., Croft, W.B.: Retrieval models for question and answer archives. In: Proceeding of SIGIR (2008)
Banchs, R.E., Li, H.: Iris: a chat-oriented dialogue system based on the vector space model. In: Proceedings of ACL (2012)
Nio, L., Sakti, S., Neubig, G., Toda, T., Nakamura., S.: Improving the robustness of example-based dialog retrieval using recursive neural network paraphrase identification. In: Proceedings of SLT (2014)
Nio, L., Sakti, S., Neubig, G., Toda, T., Nakamura, S.: Utilizing human-to-human conversation examples for a multi domain chat-oriented dialog system. Trans. IEICE (2014)
Settles, B.: Active learning literature survey. Computer Sciences Technical Report, vol. 1648, University of Wisconsin-Madison (2011)
Ghigi, F., Tamarit, V., Martinez-Hinarejos, C.D., Benedi, J.M.: Active learning for dialogue act labelling. Lect. Notes Comput. Sci. (2011)
Gotab, P., Bechet, F., Damnati, G.: Active learning for rule-based and corpus-based spoken language understanding models. In: Proceedings of ASRU (2009)
Hakkani-Tür, D., Riccardi, G., Tür, G.: An active approach to spoken language processing. ACM Trans. Speech Lang. Process. (2015)
Tür, G., Hakkani-Tür, D., Schapire, R.E.: Combining active and semi-supervised learning for spoken language understanding. Speech Commun. (2005)
Tür, G., Schapire, R.E., Hakkani-Tür, D.: Active learning for spoken language understanding. In: Proceedings of ICASSP (2003)
Williams, J.D., Niraula, N.B., Dasigi, P., Lakshmiratan, A., Suarez, C.G.J., Reddy, M., Zweig, G.: Rapidly scaling dialog systems with interactive learning. In: Proceedings of IWSDS (2015)
Riccardi, G., Hakkani-Tür, D.: Active learning: theory and applications to automatic speech recognition. Speech Audio Process. (2005)
Yu, D., Varadarajan, B., Deng, L., Acero, A.: Active learning and semi-supervised learning for speech recognition: a unified framework using the global entropy reduction maximization criterion. Comput. Speech Lang. (2010)
Mairesse, F., Gašić, M., Jurčíček, F., Keizer, S., Thomson, B., Yu, K., Young, S.: Phrase-based statistical language generation using graphical models and active learning. In: Proceedings of ACL (2010)
Gašić, M., Young, S.: Gaussian processes for POMDP-based dialogue manager optimisation. Audio Speech Lang. Process. (2014)
Leskovec, J., Rajaraman, A., Ullman, J.D.: Mining of Massive Datasets. Cambridge University Press (2014)
Ritter, A., Cherry, C., Dolan, W.B.: Data-driven response generation in social media. In: Proceedings of EMNLP (2011)
Shin, A., Sasano, R., Takamura, H., Okumura, M.: Context-dependent automatic response generation using statistical machine translation techniques. In: Proceedings of NAACL (2015)
Sordoni, A., Galle, M., Auli, M., Brockett, C., Mitchell, Y.M., Nie, J.Y., Gao, J., Dolan, B.: A neural network approach to context-sensitive generation of conversational responses. In: Proceedings of NAACL (2015)
Lewis, D.D., Gale, W.A.: A sequential algorithm for training text classifiers. In: Proceedings of SIGIR (1994)
Williams, J., Raux, A., Ramachandran, D., Black, A.: The dialog state tracking challenge. In: Proceedings of SIGDIAL (2013)
Henderson, M., Thomson, B., Williams, J.: The second dialog state tracking challenge. In: Proceedings of SIGDIAL (2014)
Henderson, M., Thomson, B., Williams, J.: The third dialog state tracking challenge. In: Proceedings of SLT (2014)
Higashinaka, R., Funakoshi, K., Araki, M., Tsukahara, H., Kobayashi, Y., Mizukami, M.: Towards taxonomy of errors in chat-oriented dialogue systems. In: Proceedings of SIGDIAL (2015)
Nio, L., Sakti, S., Neubig, G., Toda, T., Nakamura., S.: Conversation dialog corpora from television and movie scripts. In: Proceedings of O-COCOSDA (2014)
Acknowledgements
Part of this research was supported by JSPS KAKENHI Grant Number 24240032, and the Commissioned Research of National Institute of Information and Communications Technology (NICT), Japan.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer Science+Business Media Singapore
About this chapter
Cite this chapter
Hiraoka, T., Neubig, G., Yoshino, K., Toda, T., Nakamura, S. (2017). Active Learning for Example-Based Dialog Systems. In: Jokinen, K., Wilcock, G. (eds) Dialogues with Social Robots. Lecture Notes in Electrical Engineering, vol 427. Springer, Singapore. https://doi.org/10.1007/978-981-10-2585-3_5
Download citation
DOI: https://doi.org/10.1007/978-981-10-2585-3_5
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-2584-6
Online ISBN: 978-981-10-2585-3
eBook Packages: EngineeringEngineering (R0)