Abstract
In this paper, we present a novel feature sparsification approach for a class of kernel-based approximate policy iteration algorithms called KLSPI. We firstly introduce the relative approximation error in the sparsification process based on the approximate linear dependence (ALD) analysis. The relative approximation error is used as the criterion for selecting the kernel-based features. An improved KLSPI algorithm is also proposed by integrating the new sparsification method with KLSPI. Experimental results on the Inverted Pendulum problem demonstrate that the proposed sparsification method can obtain a smaller size of kernel dictionary than the previous ALD method. Furthermore, by using the more representative samples as the kernel dictionary, the precision of value function approximation has been increased. The improved KLSPI algorithm can also achieve better learning efficiency and policy quality than the original one. The feasibility and validity of the new method are proven.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Sutton, R., Barto, A.: Reinforcement learning:an introduction. MIT Press, Cambridge (1998)
Boyan, J.: Technical update: Least-squares temporal difference learning. Mach. Learn. 49(2-3), 233–246 (2002)
Xu, X., et al.: Kernel-based Least-Squares Policy Iteration for Reinforcement Learning. IEEE Trans. on Neural Networks 18, 973–992 (2007)
Xu, X., et al.: Efficient Reinforcement Learning Using Recursive Least-Squares Methods. Journal of Artificial Intelligence Research 16, 259–292 (2002)
Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines. Cambridge Univ. Press, Cambridge (2000)
Kaelbling, L.P., et al.: Reinforcement learning: A survey. J. Artif. Intell. Res. 4, 237–285 (1996)
Lagoudakis, M.G., Parr, R.: Least-squares policy iteration. J.Mach. Learn. Res. 4, 1107–1149 (2003)
Ormoneit, S.: Kernel-based reinforcement learning. Machine Learning 49(2), 161–178 (2002)
Schölkopf, B., Smola, A.: Learning With Kernels. MIT Press, Cambridge (2002)
Tsitsiklis, J.N., Roy, B.V.: An analysis of temporal difference learning with function approximation. IEEE Trans. Autom. Control. 42(5), 674–690 (1997)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Huang, Z., Liu, C., Xu, X., Lian, C., Wu, J. (2012). A Novel Feature Sparsification Method for Kernel-Based Approximate Policy Iteration. In: Wang, J., Yen, G.G., Polycarpou, M.M. (eds) Advances in Neural Networks – ISNN 2012. ISNN 2012. Lecture Notes in Computer Science, vol 7367. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31346-2_28
Download citation
DOI: https://doi.org/10.1007/978-3-642-31346-2_28
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-31345-5
Online ISBN: 978-3-642-31346-2
eBook Packages: Computer ScienceComputer Science (R0)