Milica Gašić
Abstract
Effective dialogue management is critically dependent on the information that is encoded in the dialogue state. In order to deploy reinforcement learning for policy optimization, dialogue must be modeled as a Markov Decision Process. This requires that the dialogue state must encode all relevent information obtained during the dialogue prior to that state. This can be achieved by combining the user goal, the dialogue history, and the last user action to form the dialogue state. In addition, to gain robustness to input errors, dialogue must be modeled as a Partially Observable Markov Decision Process (POMDP) and hence, a distribution over all possible states must be maintained at every dialogue turn. This poses a potential computational limitation since there can be a very large number of dialogue states. The Hidden Information State model provides a principled way of ensuring tractability in a POMDP-based dialogue model. The key feature of this model is the grouping of user goals into partitions that are dynamically built during the dialogue. In this article, we extend this model further to incorporate the notion of complements. This allows for a more complex user goal to be represented, and it enables an effective pruning technique to be implemented that preserves the overall system performance within a limited computational resource more effectively than existing approaches.
- Kim, K., Lee, C., Jung, S., and Lee, G. G. 2008. A frame-based probabilistic framework for spoken dialog management using dialog examples. In Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue. Association for Computational Linguistics, Morristown, NJ, 120--127. Google Scholar
Digital Library
- Levin, E., Pieraccini, R., and Eckert, W. 1998. Using Markov decision processes for learning dialogue strategies. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing.Google Scholar
- Sutton, R. and Barto, A. 1998. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA. Google ScholarDigital Library
- Thomson, B. 2009. Statistical methods for spoken dialogue management. Ph.D. thesis, University of Cambridge.Google Scholar
- Thomson, B. and Young, S. 2010. Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems. Comput. Speech Lang. 24, 562--568. Google ScholarDigital Library
- Thomson, B., Yu, K., Gašić, M., Keizer, S., Mairesse, F., Schatzmann, J., and Young, S. 2008. Evaluating semantic-level confidence scores with multiple hypotheses. In Proceedings of Interspeech.Google Scholar
- Williams, J. 2010. Incremental partition recombiantion for efficient tracking of multiple dialogue states. In Proceedings of the International Conference on Acoustics Speech and Signal Processing.Google Scholar
- Williams, J., Poupart, P., and Young, S. 2005. Factored partially observable Markov decision processes for dialogue management. In Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialogue Systems.Google Scholar
- Young, S., Gašić, M., Keizer, S., Mairesse, F., Schatzmann, J., Thomson, B., and Yu, K. 2010. The Hidden Information State Model: A practical framework for POMDP-based spoken dialogue management. Comput. Speech Lang. 24, 2, 150--174. Google ScholarDigital Library
Index Terms
- Effective handling of dialogue state in the hidden information state POMDP-based dialogue manager
Recommendations
Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems
This paper describes a statistically motivated framework for performing real-time dialogue state updates and policy learning in a spoken dialogue system. The framework is based on the partially observable Markov decision process (POMDP), which provides ...
Reinforcement learning for parameter estimation in statistical spoken dialogue systems
Reinforcement techniques have been successfully used to maximise the expected cumulative reward of statistical dialogue systems. Typically, reinforcement learning is used to estimate the parameters of a dialogue policy which selects the system's ...
A Reinforcement Learning approach to evaluating state representations in spoken dialogue systems
Although dialogue systems have been an area of research for decades, finding accurate ways of evaluating different systems is still a very active subfield since many leading methods, such as task completion rate or user satisfaction, capture different ...
Comments