Abstract
When encountering continuous, or very large domains, using a compact representation of the state space is preferable for practical reinforcement learning (RL). This approach can reduce the size of the state space and enable generalization by relating similar or neighboring states. However, many state abstraction techniques cannot achieve satisfactory approximation quality in the presence of limited memory resources, while expert state space shaping can be costly and usually does not scale well. We have investigated the principle of Sparse Distributed Memories (SDMs) and applied it as a function approximator to learn good policies for RL. This paper describes a new approach, adaptive adjacency in SDMs, that is capable of representing very large continuous state spaces with a very small collection of prototype states. This algorithm enhances an SDMs architecture to allow on-line, dynamically-adjusting generalization to assigned memory resources to provide high-quality approximation. The memory size and memory allocation no longer need to be manually assigned before and during RL. Based on our results, this approach performs well both in terms of approximation quality and memory usage. The superior performance of this approach over existing SDMs and tile coding (CMACs) is demonstrated through a comprehensive simulation study in two classic domains, Mountain Car with 2 dimensions and Hunter-Prey with 5 dimensions. Our empirical evaluations demonstrate that the adaptive adjacency approach can be used to efficiently approximate value functions with limited memories, and that the approach scales well across tested domains with continuous, large-scale state spaces.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Allen, M., Fritzsche, P.: Reinforcement learning with adaptive Kanerva coding for Xpilot game AI. In: 2011 IEEE Congress of Evolutionary Computation (CEC), pp. 1521–1528. IEEE (2011). https://doi.org/10.1109/CEC.2011.5949796
Chernova, S., Veloso, M.: Tree-based policy learning in continuous domains through teaching by demonstration. In: Proceedings of Workshop on Modeling Others from Observations (MOO 2006) (2006)
Chiariotti, F., D’Aronco, S., Toni, L., Frossard, P.: Online learning adaptation strategy for dash clients. In: Proceedings of the 7th International Conference on Multimedia Systems, p. 8. ACM (2016). https://doi.org/10.1145/2910017.2910603
Forbes, J.R.N.: Reinforcement Learning for Autonomous Vehicles. University of California, Berkeley (2002)
Frommberger, L.: Qualitative Spatial Abstraction in Reinforcement Learning. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-16590-0
Geist, M., Pietquin, O.: Algorithmic survey of parametric value function approximation. IEEE Trans. Neural Netw. Learn. Syst. 24(6), 845–867 (2013). https://doi.org/10.1109/TNNLS.2013.2247418
Hausknecht, M., Khandelwal, P., Miikkulainen, R., Stone, P.: HyperNEAT-GGP: a hyperNEAT-based atari general game player. In: Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation, pp. 217–224. ACM (2012). https://doi.org/10.1145/2330163.2330195
Kanerva, P.: Sparse distributed memory and related models, vol. 92. NASA Ames Research Center, Research Institute for Advanced Computer Science (1992)
Keller, P.W., Mannor, S., Precup, D.: Automatic basis function construction for approximate dynamic programming and reinforcement learning. In: Proceedings of International Conference on Machine Learning (2006). https://doi.org/10.1145/1143844.1143901
Li, L., Baker, T.E., White, S.R., Burke, K.: Pure density functional for strong correlations and the thermodynamic limit from machine learning. Phys. Rev. B 94(24), 245129 (2016)
Li, W., Zhou, F., Meleis, W., Chowdhury, K.: Learning-based and data-driven TCP design for memory-constrained iot. In: 2016 International Conference on Distributed Computing in Sensor Systems (DCOSS), pp. 199–205. IEEE (2016). https://doi.org/10.1109/DCOSS.2016.8
Lin, S., Wright, R.: Evolutionary tile coding: an automated state abstraction algorithm for reinforcement learning. In: Abstraction, Reformulation, and Approximation (2010)
Mao, H., Netravali, R., Alizadeh, M.: Neural adaptive video streaming with pensieve. In: Proceedings of the Conference of the ACM Special Interest Group on Data Communication, pp. 197–210. ACM (2017). https://doi.org/10.1145/3098822.3098843
Munos, R., Moore, A.: Variable resolution discretization in optimal control. Mach. Learn. 49(2), 291–323 (2002)
Ratitch, B., Precup, D.: Sparse distributed memories for on-line value-based reinforcement learning. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS (LNAI), vol. 3201, pp. 347–358. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-30115-8_33
Santamaría, J.C., Sutton, R.S., Ram, A.: Experiments with reinforcement learning in problems with continuous state and action spaces. Adapt. Behav. 6(2), 163–217 (1997)
Smart, W.D., Kaelbling, L.P.: Practical reinforcement learning in continuous spaces. In: ICML, pp. 903–910 (2000)
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. Bradford Books (1998)
Whiteson, S., Taylor, M.E., Stone, P., et al.: Adaptive tile coding for value function approximation. University of Texas at Austin, Computer Science Department (2007)
Wu, C., Li, W., Meleis, W.: Rough sets-based prototype optimization in Kanerva-based function approximation. In: 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), vol. 2, pp. 283–291. IEEE (2015). https://doi.org/10.1109/WI-IAT.2015.179
Wu, C., Meleis, W.: Adaptive Kanerva-based function approximation for multi-agent systems. In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, vol. 3. pp. 1361–1364. International Foundation for Autonomous Agents and Multiagent Systems (2008). https://doi.org/10.1145/1402821.1402872
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Li, W., Meleis, W. (2018). Adaptive Adjacency Kanerva Coding for Memory-Constrained Reinforcement Learning. In: Perner, P. (eds) Machine Learning and Data Mining in Pattern Recognition. MLDM 2018. Lecture Notes in Computer Science(), vol 10934. Springer, Cham. https://doi.org/10.1007/978-3-319-96136-1_16
Download citation
DOI: https://doi.org/10.1007/978-3-319-96136-1_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-96135-4
Online ISBN: 978-3-319-96136-1
eBook Packages: Computer ScienceComputer Science (R0)