Abstract
Multi-agent reinforcement learning (MARL) is an emerging area of research. However, it lacks two important elements: a coherent view on MARL, and a well-defined problem objective. We demonstrate these points by introducing three phenomena, social norms, teaching, and bounded rationality, which are inadequately addressed by the previous research. Based on the ideas of bounded rationality, we define a very broad class of MARL problems that are equivalent to learning in partially observable Markov decision processes (POMDPs). We show that this perspective on MARL accounts for the three missing phenomena, but also provides a well-defined objective for a learner, since POMDPs have a well-defined notion of optimality. We illustrate the concept in an empirical study, and discuss its implications for future research.
Thanks to Nicholas Kushmerick for helpful discussions and valuable comments. This research was supported by grant SFI/01/F.1/C015 from Science Foundation Ireland, and grant N00014-03-1-0274 from the US Office of Naval Research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Shoham, Y., Grenager, T., Powers, R.: Multi-agent reinforcement learning: A critical survey. Tech.rep., Stanford University (2003)
Simon, H.: Models of Man. Social and Rational. John Wiley and Sons, Chichester (1957)
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Filar, J., Vrieze, K.: Competitive Markov Decision Processes. Springer, Heidelberg (1997)
Osborne, M., Rubinstein, A.: A Course in Game Theory. MIT Press, Cambridge (1999)
Littman, M.: Markov games as a framework for multi-agent reinforcement learning. In: Proc. of the 11th Intl. Conf. on Machine Learning (1994)
Hu, J., Wellman, M.P.: Nash Q-learning for general-sum stochastic games. Journal of Machine Learning Research 4 (2003)
Claus, C., Boutilier, C.: The dynamics of reinforcement learning in cooperative multiagent systems. In: Proc. of the 15th AAAI Conf. (1998)
Carmel, D., Markovitch, S.: Learning models of intelligent agents. In: Proc. of the 13th AAAI Conf. (1996)
Bowling, M.: Multiagent Learning in the Presence of Agents with Limitations. PhD thesis, Carnegie Mellon University (2003)
Dutta, P.K.: A folk theorem for stochastic games. J. of Economic Theory 66 (1995)
Rubinstein, A.: Equilibrium in supergames. In: Essays in Game Theory, Springer, Heidelberg (1994)
Axelrod, R.: The Evolution of Cooperation. Basic Books, New York (1984)
Fudenberg, D., Levine, D.K.: The Theory of Learning in Games. MIT Press, Cambridge (1998)
Chang, Y., Kaelbling, L.P.: Playing is believing: The role of beliefs in multi-agent learning. In: Advances in Neural Information Processing Systems, vol. 14, The MIT Press, Cambridge (2001)
Harsanyi, J., Selton, R.: A General Theory of Equilibrium Selection in Games. MIT Press, Cambridge (1988)
Conitzer, V., Sandholm, T.: Complexity results about Nash equilibria. In: Proc. of the 18th Intl. Joint Conf. on AI (2003)
Singh, S., Jaakkola, T., Jordan, M.: Learning without state-estimation in partially observable Markovian decision processes. In: Proc. of the 11th Intl. Conf. on Machine Learning (1994)
Peshkin, L., Meuleau, N., Kim, K.E., Kaelbling, L.: Learning to cooperate via policy search. In: Proc. of the 16th Conf. on Uncertainty in AI (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Khoussainov, R. (2004). Towards Well-Defined Multi-agent Reinforcement Learning. In: Bussler, C., Fensel, D. (eds) Artificial Intelligence: Methodology, Systems, and Applications. AIMSA 2004. Lecture Notes in Computer Science(), vol 3192. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30106-6_41
Download citation
DOI: https://doi.org/10.1007/978-3-540-30106-6_41
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-22959-9
Online ISBN: 978-3-540-30106-6
eBook Packages: Springer Book Archive