Abstract
In the Reinforcement Learning paradigm, environments that change between (or during) episodes are known as non-stationary. This property poses a challenge for traditional Reinforcement Learning approaches due to the fact that such methods typically rely on information that has been learned in previous episodes. However, in a non-stationary task, such information is usually an obstacle to learning that can lead to worse-than-random performance because it misleads the agent. Although an active area of research, most existing RL approaches also suffer from poor sample efficiency and struggle when the size of the state space increases. This work introduces a novel Reinforcement Learning approach that performs well in non-stationary environments, irrespective of the size of the state space. The approach proposed is termed Idea-based Reinforcement Learning, a symbolic method that can be applied to any problem that is a fully observable Markov Decision Process. IbRL is proven to perform statistically significantly above chance-level in all experiments and performs better than PPO2 in non-stationary problems where the state space is large.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mnih, V., Kavukcuoglu, K., Silver, D. et al.: Playing atari with deep reinforcement learning (19 Dec 2013). arXiv:1312.5602. https://doi.org/10.48550/arXiv.1312.5602
Silver, D., Huang, A., Maddison, C., et al.: Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016). https://doi.org/10.1038/nature16961
Silver, D., Hubert, T., Schrittwieser, J., et al.: Mastering chess and shogi by self-play with a general reinforcement learning algorithm (5 Dec 2017). arXiv:1712.01815. https://doi.org/10.48550/arXiv.1712.01815
Vinyals, O., Babushkin, I., Czarnecki, W., et al.: Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575(7782), 350–354 (2019). https://doi.org/10.1038/s41586-019-1724-z
Schrittwieser, J., Antonoglou, I., Hubert, T., et al.: Mastering atari, go, chess and shogi by planning with a learned model. Nature 588(7839), 604–609 (2020). https://doi.org/10.1038/s41586-020-03051-4
Thrun, S.: Efficient Exploration in Reinforcement Learning. Carnegie Mellon University (1992). https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.45.2894
Watkins, C.: Learning from Delayed Rewards. PhD thesis, University of Cambridge, Cambridge, England (1989). https://www.academia.edu/3294050/Learning_from_delayed_rewards?from=cover_page
Garnelo, M., Arulkumaran, K., Shanahan, M.: Towards deep symbolic reinforcement learning (18 Sep 2016). arXiv:1609.05518. https://doi.org/10.48550/arXiv.1609.05518
Asai, M., Kajino, H., Fukunaga, A., et al.: Classical planning in deep latent space (30 Jun 2021). arXiv:2107.00110. https://doi.org/10.48550/arXiv.2107.00110
Doersch, C.: Tutorial on variational autoencoders (19 Jun 2016). arXiv:1606.05908. https://doi.org/10.48550/arXiv.1606.05908
Asperti, A., Trentin, M.: Balancing reconstruction error and Kullback-Leibler divergence in variational autoencoders. IEEE Access 8, 199440–199448 (2020). https://doi.org/10.48550/arXiv.2002.07514
Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995). https://doi.org/10.1007/BF00994018
OpenAI, Gym Documentation (2022). https://www.gymlibrary.dev/content/api/
Hill, A., Raffin, A., Ernestus, M., et al.: Stable Baselines (2018). https://stable-baselines.readthedocs.io/en/master/
Wang, Z., Bapst, V., Heess, N., et al.: Sample efficient actor-critic with experience replay (3 Nov 2016). arXiv:1611.01224. https://doi.org/10.48550/arXiv.1611.01224
Schulman, J., Wolski, F., Dhariwal, P., et al.: Proximal policy optimization algorithms (20 Jul 2017). arXiv:1707.06347. https://doi.org/10.48550/arXiv.1707.06347
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Toé, S.G.D., Tiddeman, B., Mac Parthaláin, N. (2024). Towards Reinforcement Learning for Non-stationary Environments. In: Naik, N., Jenkins, P., Grace, P., Yang, L., Prajapat, S. (eds) Advances in Computational Intelligence Systems. UKCI 2023. Advances in Intelligent Systems and Computing, vol 1453. Springer, Cham. https://doi.org/10.1007/978-3-031-47508-5_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-47508-5_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47507-8
Online ISBN: 978-3-031-47508-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)