Abstract
In reinforcement learning, when dimensionality of the state space increases, making use of state abstraction seems inevitable. Among the methods proposed to solve this problem, decision tree based methods could be useful as they provide automatic state abstraction. But existing methods use univariate, therefore axis-aligned, splits in decision nodes, imposing hyper-rectangular partitioning of the state space. In some applications, multivariate splits can generate smaller and more accurate trees. In this paper, we use oblique decision trees as an instance of multivariate trees to implement state abstraction for reinforcement learning agents. Simulation results on mountain car and puddle world tasks show significant improvement in the average received rewards, average number of steps to finish the task, and size of the trees both in learning and test phases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Tesauro, G.: Temporal difference learning and td-gammon. Communications of the ACM 38(3), 58–68 (1995)
Boyan, J.A., Moore, A.W.: Generalization in reinforcement learning: Safely approximating the value function. In: Tesauro, G., Touretsky, D.S., Leen, T.K. (eds.) Advances in Neural Information Processing Systems, Cambridge, MA, vol. 7 (1995)
Moore, A.W.: The parti-game algorithm for variable resolution reinforcement learning in multidimensional state-spaces. In: Cowan, J.D., Tesauro, G., Alspector, J. (eds.) Advances in Neural Information Processing Systems, vol. 6, pp. 711–718. Morgan Kaufmann Publishers, Inc., San Francisco (1994)
Chapman, D., Kaelbling, L.P.: Input generalization in delayed reinforcement learning: An algorithm and performance comparisons. In: Proceedings of the Twelfth International Joint Conference on Artificial Intelligence, pp. 726–731 (1991)
McCallum, A.: Reinforcement with Selective Perception and Hidden State. PhD thesis, Computer Science Dept., University of Rochester (1995)
Uther, W.R., Veloso, M.M.: Tree Based Discretization for Continuous State Space Reinforcement Learning. In: Proceedings of the National Conference on Artificial Intelligence, pp. 769–775 (1998)
Asadpour, M.: Behavior Design in Microrobots: Hierarchical Reinforcement Learning under Resource Constraints. PHD thesis, EPFL, Lausanne, Switzerland (2006)
Pyeatt, L.D., Howe, A.E.: Decision tree function approximation in reinforcement learning. In: Proceedings of the Third International Symposium on Adaptive Systems: Evolutionary Computation and Probabilistic Graphical Models, pp. 70–77 (2001)
Brodley, C.E., Utgoff, P.E.: Multivariate decision trees. Machine Learning 19, 45–77 (1995)
Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and regression trees. Technical report, Wadsworth International, Monterey, CA (1984)
Murthy, S.K.: On Growing Better Decision Trees from Data, Ph.D. dissertation, Johns Hopkins Univ., Dept. Comput. Sci., Baltimore, MD (1995)
Heath, D., Kasif, S., Salzberg, S.: Learning Oblique Decision Trees. In: Proceedings of the 13th International Joint Conference on Artificial Intelligence, Chambery, France, pp. 1002–1007. Morgan Kaufmann, San Francisco (1993)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Tanner, B., White, A.: RL-Glue: Language-Independent Software for Reinforcement-Learning Experiments. Journal of Machine Learning Research, 2133–2136 (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Saghezchi, H.B., Asadpour, M. (2010). Multivariate Decision Tree Function Approximation for Reinforcement Learning. In: Wong, K.W., Mendis, B.S.U., Bouzerdoum, A. (eds) Neural Information Processing. Theory and Algorithms. ICONIP 2010. Lecture Notes in Computer Science, vol 6443. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-17537-4_83
Download citation
DOI: https://doi.org/10.1007/978-3-642-17537-4_83
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-17536-7
Online ISBN: 978-3-642-17537-4
eBook Packages: Computer ScienceComputer Science (R0)