Abstract
This article presents the design of a generic negotiation dialogue game between two or more players. The goal is to reach an agreement, each player having his own preferences over a shared set of options. Several simulated users have been implemented. An MDP policy has been optimised individually with Fitted Q-Iteration for several user instances. Then, the learnt policies have been cross evaluated with other users. Results show strong disparity of inter-user performances. This illustrates the importance of user adaptation in negotiation-based dialogue systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, vol. 1. MIT Press Cambridge (1998)
Levin, E., Pieraccini, R.: A stochastic model of computer-human interaction for learning dialogue strategies. In: Proceedings of the 5th European Conference on Speech Communication and Technology (Eurospeech) (1997)
Laroche, R., Putois, G., Bretier, P., Bouchon-Meunier, B.: Hybridisation of expertise and reinforcement learning in dialogue systems. In: Proceedings of the 9th Annual Conference of the International Speech Communication Association (Interspeech), pp. 2479–2482 (2009)
Lemon, O., Pietquin, O.: Data-Driven Methods for Adaptive Spoken Dialogue Systems: Computational Learning for Conversational Interfaces. Springer (2012)
English, M.S., Heeman, P.A.: Learning mixed initiative dialogue strategies by using reinforcement learning on both conversants. In: Proceedings of the Conference on Human Language Technology (HLT) (2005)
Georgila, K., Traum, D.R.: Reinforcement learning of argumentation dialogue policies in negotiation. In: Proceedings of the 11th Annual Conference of the International Speech Communication Association (Interspeech), pp. 2073–2076 (2011)
Bowling, M., Veloso, M.: Multiagent learning using a variable learning rate. Artif. Intell. 136(2), 215–250 (2002)
Georgila, K., Nelson, C., Traum, D.: Single-agent vs. multi-agent techniques for concurrent reinforcement learning of negotiation dialogue policies. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL) (2014)
Shapley, L.S.: Stochastic games. Proc. Natl. Acad. Sci. U.S.A. 39(10), 1095 (1953)
Barlier, M., Perolat, J., Laroche, R., Pietquin, O.: Human-machine dialogue as a stochastic game. In: Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue (Sigdial) (2015)
Efstathiou, I., Lemon, O.: Learning non-cooperative dialogue behaviours. In: Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (Sigdial)
Putois, G., Laroche, R., Bretier, P.: Online reinforcement learning for spoken dialogue systems: the story of a commercial deployment success. In: Proceedings of the 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 185–192. Citeseer (2010)
Laroche, R., Putois, G., Bretier, P., Aranguren, M., Velkovska, J., Hastie, H., Keizer, S., Yu, K., Jurcicek, F., Lemon, O., Young, S.: D6.4: final evaluation of classic towninfo and appointment scheduling systems. Report D6, 4 (2011)
El Asri, L., Lemonnier, R., Laroche, R., Pietquin, O., Khouzaimi, H.: Nastia: negotiating appointment setting interface. In: Proceedings of the 9th Edition of Language Resources and Evaluation Conference (LREC) (2014)
Genevay, A., Laroche, R.: Transfer learning for user adaptation in spoken dialogue systems. In: Proceedings of the 15th International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS). International Foundation for Autonomous Agents and Multiagent Systems (2016)
Chandramohan, S., Geist, M., Lefèvre, F., Pietquin, O.: Co-adaptation in spoken dialogue systems. In: Proceedings of the 4th International Workshop on Spoken Dialogue Systems (IWSDS), p. 1. Paris, France (Nov 2012)
Casanueva, I., Hain, T., Christensen, H., Marxer, R., Green, P.: Knowledge transfer between speakers for personalised dialogue management. In: Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue (Sigdial) (2015)
Khouzaimi, H., Laroche, R., Lefevre, F.: Optimising turn-taking strategies with reinforcement learning. In: Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue (Sigdial) (2015)
Gordon, G.J.: Stable function approximation in dynamic programming. In: Proceedings of the 12th International Conference on Machine Learning (ICML) (1995)
Chandramohan, S., Geist, M., Pietquin, O.: Optimizing spoken dialogue management with fitted value iteration. In: Proceedings of the 10th Annual Conference of the International Speech Communication Association (Interspeech) (2010)
Ng, A.Y., Russell, S.: Algorithms for inverse reinforcement learning. In: Proceedings of the 17th International Conference on Machine Learning (ICML), pp. 663–670. Morgan Kaufmann (2000)
El Asri, L., Piot, B., Geist, M., Laroche, R., Pietquin, O.: Score-based inverse reinforcement learning. In: Proceedings of the 15th International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS). International Foundation for Autonomous Agents and Multiagent Systems (2016)
Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 10, 1633–1685 (2009)
Lazaric, A.: Transfer in reinforcement learning: a framework and a survey. In: Reinforcement Learning, pp. 143–173. Springer (2012)
Gašic, M., Breslin, C., Henderson, M., Kim, D., Szummer, M., Thomson, B., Tsiakoulis, P., Young, S.: POMDP-based dialogue manager adaptation to extended domains. In: Proceedings of the 14th Annual Meeting of the Special Interest Group on Discourse and Dialogue (Sigdial) (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer Science+Business Media Singapore
About this chapter
Cite this chapter
Laroche, R., Genevay, A. (2017). The Negotiation Dialogue Game. In: Jokinen, K., Wilcock, G. (eds) Dialogues with Social Robots. Lecture Notes in Electrical Engineering, vol 427. Springer, Singapore. https://doi.org/10.1007/978-981-10-2585-3_33
Download citation
DOI: https://doi.org/10.1007/978-981-10-2585-3_33
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-10-2584-6
Online ISBN: 978-981-10-2585-3
eBook Packages: EngineeringEngineering (R0)