Abstract
Spoken dialogue systems are man-machine interfaces which use speech as the medium of interaction. In recent years, dialogue optimization using reinforcement learning has evolved to be a state-of-the-art technique. The primary focus of research in the dialogue domain is to learn some optimal policy with regard to the task description (reward function) and the user simulation being employed. However, in case of human-human interaction, the parties involved in the dialogue conversation mutually evolve over the period of interaction. This very ability of humans to coadapt attributes largely towards increasing the naturalness of the dialogue. This paper outlines a novel framework for coadaptation in spoken dialogue systems, where the dialogue manager and user simulation evolve over a period of time; they incrementally and mutually optimize their respective behaviors.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abbeel, P., Ng, A.: Apprenticeship learning via inverse reinforcement learning. In: Proceedings of ICML, Banff, Alberta (2004)
Astrom, K.J.: Optimal control of markov decision processes with incomplete state estimation. J. Math. Anal. Appl. 10, 174–205 (1965)
Bellman, R.: A markovian decision process. J. Math. Mech. 6, 679–684 (1957)
Chandramohan, S., Geist, M., Lefèvre, F., Pietquin, O.: User simulation in dialogue systems using inverse reinforcement learning. In: Proceedings of Interspeech 2011, Florence (2011)
Daubigney, L., Gasic, M., Chandramohan, S., Geist, M., Pietquin, O., Young, S.: Uncertainty management for on-line optimisation of a POMDP-based large-scale spoken dialogue system. In: Proceedings of Interspeech 2011, Florence pp. 1301–1304 (2011)
Eckert, W., Levin, E., Pieraccini, R.: User modeling for spoken dialogue system evaluation. In: Proceedings of ASRU, pp. 80–87 (1997)
Frampton, M., Lemon, O.: Recent research advances in reinforcement learning in spoken dialogue systems. Knowl. Eng. Rev. 24(4), 375–408 (2009)
Gasic, M., Jurcicek, F., Thomson, B., Yu, K., Young, S.: On-line policy optimisation of spoken dialogue systems via live interaction with human subjects”. In: Proceedings of ASRU 2011, Hawaii (2011)
Georgila, K., Henderson, J., Lemon, O.: Learning user simulations for information state update dialogue systems. In: Proceedings of Eurospeech, Lisbon (2005)
Lagoudakis, M.G., Parr, R.: Least-squares policy iteration. J. Mach. Lear. Res. 4, 1107–1149 (2003)
Lemon, O., Georgila, K., Henderson, J., Stuttle, M.: An ISU dialogue system exhibiting reinforcement learning of dialogue policies: generic slot-filling in the TALK in-car system. In: Proceedings of EACL’06, Morristown (2006)
Lemon, O., Pietquin, O.: Machine learning for spoken dialogue systems. In: Proceedings of InterSpeech’07, Belgium (2007)
Levin, E., Pieraccini, R.: Using markov decision process for learning dialogue strategies. In: Proceedings ICASSP’98, Seattle (1998)
Ng, A.Y., Russell, S.: Algorithms for inverse reinforcement learning. In: Proceedings of ICML, Stanford (2000)
Pietquin, O.: Consistent goal-directed user model for realistic man-machine task-oriented spoken dialogue simulation. In: Proceedings of ICME’06, Toronto, pp. 425–428 (2006)
Pietquin, O., Dutoit, T.: A probabilistic framework for dialog simulation and optimal strategy learning. IEEE Trans. Audio Speech Lang. Process. 14(2), 589–599 (2006)
Pietquin, O., Geist, M., Chandramohan, S., Frezza-Buet, H.: Sample-efficient batch reinforcement learning for dialogue management optimization. ACM Trans. Speech Lang. Process. 7(3), 7:1–7:21 (2011)
Pietquin, O., Rossignol, S., Ianotto, M.: Training bayesian networks for realistic man-machine spoken dialogue simulation. In: Proceedings of IWSDS 2009, Irsee (2009)
Schatzmann, J., Stuttle, M.N., Weilhammer, K., Young, S.: Effects of the user model on simulation-based learning of dialogue strategies. In: Proceedings of ASRU, Puerto Rico (2005)
Schatzmann, J., Thomson, B., Weilhammer, K., Ye, H., Young., S.: Agenda-based user simulation for bootstrapping a POMDP dialogue system. In: Proceedings of HLT NAACL, Rochester (2007)
Singh, S., Kearns, M., Litman, D., Walker, M.: Reinforcement learning for spoken dialogue systems. In: Proceedings of NIPS, Denver (1999)
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction, 3rd edn. MIT, Cambridge (1998)
Williams, J.D., Young, S.: Partially observable markov decision processes for spoken dialog systems. Comput. Speech Lang. 21(2), 393–422 (2007). DOI: http://dx.doi.org/10.1016/j.csl.2006.06.008
Acknowledgements
This research was partly funded by the EU INTERREG IVa project ALLEGRO and by the Règion Lorraine (France).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer Science+Business Media New York
About this paper
Cite this paper
Chandramohan, S., Geist, M., Lefèvre, F., Pietquin, O. (2014). Co-adaptation in Spoken Dialogue Systems. In: Mariani, J., Rosset, S., Garnier-Rizet, M., Devillers, L. (eds) Natural Interaction with Robots, Knowbots and Smartphones. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-8280-2_31
Download citation
DOI: https://doi.org/10.1007/978-1-4614-8280-2_31
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-8279-6
Online ISBN: 978-1-4614-8280-2
eBook Packages: EngineeringEngineering (R0)