Abstract
Reinforcement Learning studies the problem of learning through interaction with the unknown environment. Learning efficiently in large scale problems and complex tasks demands a decomposition of the original complex task to simple and smaller subtasks. In this paper a local graph clustering algorithm is represented for discovering subgoals. The main advantage of the proposed algorithm is that only the local information of the graph is considered to cluster the agent state space. Subgoals discovered by the algorithm are then used to generate skills. Experimental results show that the proposed subgoal discovery algorithm has a dramatic effect on the learning performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Kaelbling, L.P., Littman, M.L.: Reinforcement Learning: A Survey. J. Artificial Intelligence Research 4 (1996)
Bertsekas, D.B., Tsitsiklis, J.N.: Neuro-dynamic programming. Athena Scientific (1995)
Parr, R., Russell, S.: Reinforcement learning with hierarchies of machines. In: Proc. the 1997 Conference on Advances in Neural Information Processing Systems, Cambridge, MA, USA, pp. 1043–1049 (1997)
Sutton, R., Precup, D., Singh, S.: Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. J. Artificial Intelligence 112, 181–211 (1999)
Dietterich, T.G.: Hierarchical reinforcement learning with the MAXQ value function decomposition. J. Artificial Intelligence 13, 227–303 (2000)
Barto, A.G., Mahadevan, S.: Recent Advances in Hierarchical Reinforcement Learning. Discrete Event Dynamic Systems 13, 341–379 (2003)
Şimşek, Ö., Barto, A.G.: Learning Skills in Reinforcement Learning Using Relative Novelty, pp. 367–374 (2005)
Digney, B.L.: Learning hierarchical control structures for multiple tasks and changing environments. In: Proc. the Fifth International Conference on Simulation of Adaptive Behavior on From Animals to Animals 5, Univ. of Zurich, Zurich, Switzerland (1998)
McGovern, A., Barto, A.G.: Automatic Discovery of Subgoals in Reinforcement Learning using Diverse Density. In: Proc. the Eighteenth International Conference on Machine Learning, pp. 361–368 (2001)
Menache, I., Manno, S., Shimkin, N.: Q-Cut - Dynamic Discovery of Sub-goals in Reinforcement Learning. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) ECML 2002. LNCS (LNAI), vol. 2430, p. 295. Springer, Heidelberg (2002)
Mannor, S., Menache, I., Hoze, A., Klein, U.: Dynamic abstraction in reinforcement learning via clustering. In: Proc. the Twenty-First International Conference on Machine Learning, Banff, Alberta, Canada (2004)
Şimşek, Ö., Wolfe, A.P., Barto, A.G.: Identifying useful subgoals in reinforcement learning by local graph partitioning. In: Proc. The 22nd International Conference on Machine Learning, Bonn, Germany (2005)
Jing, S., Guochang, G., Haibo, L.: Automatic option generation in hierarchical reinforcement learning via immune clustering. In: 1st International Symposium on Systems and Control in Aerospace and Astronautics, ISSCAA 2006, p. 4, p. 500 (2006)
Şimşek, Ö., Barto, A.G.: Skill Characterization Based on Betweenness. In: Advances in Neural Information Processing Systems, vol. 21, pp. 1497–1504 (2009)
Jonsson, A., Barto, A.G.: Automated state abstraction for options using the u-tree algorithm. In: Advances in Neural Information Processing Systems: Proceedings of the 2000 Conference, pp. 1054–1060 (2001)
Elfwing, S., Uchibe, E., Doya, K.: An Evolutionary Approach to Automatic Construction of the Structure in Hierarchical Reinforcement Learning. In: Genetic and Evolutionary Computation, pp. 198–198 (2003)
Jonsson, A., Barto, A.: A causal approach to hierarchical decomposition of factored MDPs. In: Proc. the 22nd International Conference on Machine Learning, Bonn, Germany ( 2005)
Jonsson, A., Barto, A.: Causal Graph Based Decomposition of Factored MDPs. J. Machine Learning, Res. 7, 2259–2301 (2006)
Mehta, N., Ray, S., Tadepalli, P., Dietterich, T.G.: Automatic discovery and transfer of MAXQ hierarchies. In: Proc. of the 25th International Conference on Machine Learning, Helsinki, Finland (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Entezari, N., Shiri, M.E., Moradi, P. (2010). A Local Graph Clustering Algorithm for Discovering Subgoals in Reinforcement Learning. In: Kim, Th., Vasilakos, T., Sakurai, K., Xiao, Y., Zhao, G., Ślęzak, D. (eds) Communication and Networking. FGCN 2010. Communications in Computer and Information Science, vol 120. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-17604-3_5
Download citation
DOI: https://doi.org/10.1007/978-3-642-17604-3_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-17603-6
Online ISBN: 978-3-642-17604-3
eBook Packages: Computer ScienceComputer Science (R0)