Abstract
In this paper, we regard the sequence of returns as outputs from a parametric compound source. The coding rate of the source shows the amount of information on the return, so the information gain concerning future information is given by the sum of the discounted coding rates. We accordingly formulate a temporal difference learning for estimating the expected information gain, and give a convergence proof of the information gain under certain conditions. As an example of applications, we propose the ratio w of return loss to information gain to be used in probabilistic action selection strategies. We found in experiments that our w-based strategy performs well compared with the conventional Q-based strategy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Sutton, R.S., Barto, A.G.: Reinforcement Learning:An Introduction. Adaptive Computation and Machine Learning. MIT Press, Cambridge (1998)
Zhang, W., Dietterich, T.G.: A reinforcement learning approach to job-stop scheduling. In: Mellish, C.S. (ed.) Proceedings of the 14th International Joint Conference on Artificial Intelligence, Montreal, Canada, pp. 1114–1120. Morgan Kaufmann, San Mateo (1995)
Likas, A.: A reinforcement learning approach to on-line clustering. Neural Computation 11, 1915–1932 (1999)
Sato, M., Kobayashi, S.: Variance-penalized reinforcement learning for risk-averse asset allocation. In: Leung, K.-S., Chan, L., Meng, H. (eds.) IDEAL 2000. LNCS, vol. 1983, pp. 244–249. Springer, Heidelberg (2000)
Billingsley, P.: Probability and Measure, 3rd edn. Wiley Series in Probability and Mathematical Statistics. JohnWiley & Sons, NewYork (1995)
Rissanen, J.: Stochastic complexity and modeling. The Annals of Statistics 14, 1080–1100 (1986)
Han, T.S., Kobayashi, K.: Mathematics of Information and Coding. In: Translations of Mathematical Monographs, vol. 203, American Mathematical Society, Providence (2002) (Translated by Joe Suzuki)
Watkins, C.J.C.H., Dayan, P.: Technical note: Q-learning. Machine Learning 8, 279–292 (1992)
Kushner, H.J., Yin, G.G.: Exercises in Computer Systems Analysis. Apprications of Mathematics, vol. 35. Springer, NewYork (1997)
Sato, M., Kobayashi, S.: Average-reward reinforcement learning for variance penalized markov decision problems. In: Brodley, C.E., Danyluk, A.P. (eds.) Proceedings of the 18th International Conference on Machine Learning, Williams College, pp. 473–480. Morgan Kaufmann Publishers, San Francisco (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Iwata, K., Ikeda, K. (2003). Temporal Difference Coding in Reinforcement Learning. In: Liu, J., Cheung, Ym., Yin, H. (eds) Intelligent Data Engineering and Automated Learning. IDEAL 2003. Lecture Notes in Computer Science, vol 2690. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-45080-1_30
Download citation
DOI: https://doi.org/10.1007/978-3-540-45080-1_30
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40550-4
Online ISBN: 978-3-540-45080-1
eBook Packages: Springer Book Archive