Abstract
A challenge in applying reinforcement learning to large problems is how to manage the explosive increase in storage and time complexity. This is especially problematic in multi-agent systems, where the state space grows exponentially in the number of agents. Function approximation based on simple supervised learning is unlikely to scale to complex domains on its own, but structural abstraction that exploits system properties and problem representations shows more promise. In this paper, we investigate several classes of known abstractions: 1) symmetry, 2) decomposition into multiple agents, 3) hierarchical decomposition, and 4) sequential execution. We compare memory requirements, learning time, and solution quality empirically in two problem variations. Our results indicate that the most effective solutions come from combinations of structural abstractions, and encourage development of methods for automatic discovery in novel problem formulations.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Clark, A., Thornton, C.: Trading spaces: Computation, representation, and the limits of uninformed learning. Behavioral and Brain Sciences 20, 57–66 (1997)
Utgoff, P.E., Stracuzzi, D.J.: Many-layered learning. In: Neural Computation. MIT Press Journals, Cambridge (2002)
Ashby, R.: Introduction to Cybernetics. Chapman & Hall, London (1956)
Dietterich, T.G.: Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research 13, 227–303 (2000)
Watkins, C.J.C.H.: Learning from Delayed Rewards. PhD thesis, King’s College (1989)
Ravindran, B., Barto, A.G.: SMDP homomorphisms: An algebraic approach to abstraction in semi markov decision processes. In: Proc. of the Eighteenth International Joint Conference on Artificial Intelligence (IJCAI 2003), pp. 1011–1018 (2003)
Ravindran, B., Barto, A.G.: Model minimization in hierarchical reinforcement learning. In: Koenig, S., Holte, R.C. (eds.) SARA 2002. LNCS (LNAI), vol. 2371, pp. 196–211. Springer, Heidelberg (2002)
Dean, T., Givan, R.: Model minimization in markov decision processes. In: AAAI/IAAI, 106–111 (1997)
Givan, R., Leach, S.M., Dean, T.: Bounded-parameter markov decision processes. Artificial Intelligence 122, 71–109 (2000)
Crites, R.H., Barto, A.G.: Elevator group control using multiple reinforcement learning agents. Machine Learning 33, 235–262 (1998)
Wolpert, D., Tumer, K.: An introduction to collective intelligence. Technical Report NASA-ARC-IC-99-63, NASA Ames Research Center, CA (1999)
Braess, D.: Über ein Paradoxon der Verkehrsplanung. Unternehmensforschung 12, 258–268 (1968)
Rohanimanesh, K., Mahadevan, S.: Learning to take concurrent actions. In: NIPS, pp. 1619–1626 (2002)
Hengst, B.: Discovering hierarchy in reinforcement learning with HEXQ. In: Sammut, C., Hoffmann, A. (eds.) Proceedings of the Nineteenth International Conference on Machine Learning, pp. 243–250. Morgan Kaufmann, San Francisco (2002)
Kaelbling, L.P.: Hierarchical learning in stochastic domains: Preliminary results. In: Machine Learning Proceedings of the Tenth International Conference, San Mateo, CA, pp. 167–173. Morgan Kaufmann, San Francisco (1993)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Fitch, R., Hengst, B., Šuc, D., Calbert, G., Scholz, J. (2005). Structural Abstraction Experiments in Reinforcement Learning. In: Zhang, S., Jarvis, R. (eds) AI 2005: Advances in Artificial Intelligence. AI 2005. Lecture Notes in Computer Science(), vol 3809. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11589990_19
Download citation
DOI: https://doi.org/10.1007/11589990_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-30462-3
Online ISBN: 978-3-540-31652-7
eBook Packages: Computer ScienceComputer Science (R0)