Abstract
Function approximators are commonly in use for reinforcement learning systems to cover the untrained situations, when dealing with large problems. By doing so however, theoretical proves on the convergence criteria are still lacking, and practical researches have both positive and negative results. In a recent work [3] with neural networks, the authors reported that the final results did not reach the quality of a Q-table in which no approximation ability was used. In this paper, we continue this research with grid based function approximators. In addition, we consider the required number of state transitions and apply ideas from the field of active learning to reduce this number. We expect the learning process of a similar problem in a real world system to be significantly shorter because state transitions, which represent an object’s actual movements, require much more time than basic computational processes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Baird, L.: Residual Algorithms: Reinforcement Learning with Function Approximation. In: Machine Learning: Twelfth International Conference, SF, USA (1995)
Boyan, J.A., Moore, A.W.: Generalization in Reinforcement Learning: Safely Approximating the Value Function. In: Advances in Neural Information Processing Systems 7. Morgan Kaufmann, San Mateo (1995)
Carreras, M., Ridao, P., El-Fakdi, A.: Semi-Online Neural-Q_learning for Real-time Robot Learning. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (2003)
Cohn, D., Atlas, L., Ladner, R.: Improving Generalization with Active Learning. Machine Learning 15, 201–221 (1994)
Cohn, D.A., Ghahramani, Z., Jordan, M.I.: Active Learning with Statistical Models. Journal of Artificial Intelligence Research 4, 129–145 (1996)
Hasenjäger, M., Ritter, H.: Active Learning in Neural Networks. In: New learning paradigms in soft computing. Physica-Verlag Studies In Fuzziness And Soft Computing Series, pp. 137–169 (2002)
Kretchmar, R.M., Anderson, C.W.: Comparison of CMACs and Radial Basis Functions for Local Function Approximators in Reinforcement Learning. In: International Conference on Neural Networks, Houston, Texas, USA (1997)
Merke, A., Schoknecht, R.: A Necessary Condition of Convergence for Reinforcement Learning with Function Approximation. In: ICML, pp. 411–418 (2002)
Poland, J., Zell, A.: Different Criteria for Active Learning in Neural Networks: A Comparative Study. In: Verleysen, M. (ed.) Proceedings of the 10th European Symposium on Artificial Neural Networks, pp. 119–124 (2002)
Riedmiller, M.: Concepts and Facilities of a neural reinforcement learning control architecture for technical process control (draft version). Neural Computation and Application Journal 8, 323–338 (1999)
Sung, K.K., Niyogi, P.: Active learning for function approximation. In: Tesauro, G., Touretzky, D., Leen, T.K. (eds.) Advances in Neural Processing Systems, vol. 7, pp. 593–600. MIT Press, Cambridge (1998)
Sutton, R.S.: Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding. In: Hasselmo, M.E., Touretzky, D.S., Mozer, M.C. (eds.) Advances in Neural Information Processing Systems, 8. MIT Press, Cambridge (1996)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Watkins, C.J.C.H., Dayan, P.: Q-learning. Machine learning 8, 279–292 (1992)
Weaver, S., Baird, L., Polycarpou, M.: An Analytical Framework for Local Feedforward Networks. IEEE Transactions on Neural Networks 9(3) (1999)
Zheng, Z., Padmanabhan, B.: On Active Learning for Data Acquisition. In: 2002 IEEE International Conference on Data Mining, Maebashi City, Japan, p. 562 (2002)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Sung, A., Merke, A., Riedmiller, M. (2005). Reinforcement Learning Using a Grid Based Function Approximator. In: Wermter, S., Palm, G., Elshaw, M. (eds) Biomimetic Neural Learning for Intelligent Robots. Lecture Notes in Computer Science(), vol 3575. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11521082_14
Download citation
DOI: https://doi.org/10.1007/11521082_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-27440-7
Online ISBN: 978-3-540-31896-5
eBook Packages: Computer ScienceComputer Science (R0)