Abstract
Over the last years, the field of artificial intelligence (AI) has continuously evolved to great success. As a subset of AI, Reinforcement Learning (RL) has gained significant popularity as well and a variety of RL algorithms and extensions have been developed for various use cases. Although RL is applicable to a wide range of problems today, the amount of options is overwhelming and identifying the advantages and disadvantages of methods for selecting the most suitable algorithms is difficult. Sources use conflicting terminology, imply improvements to alternative algorithms without mathematical or empirical proof, or provide incomplete information. As a result, there is the chance for engineers and researchers to miss alternatives or perfect-fit algorithms for their specific problems. In this paper, we identify and explain essential properties of RL problems and algorithms. Our discussion of these concepts can be used to select, optimize, and compare RL algorithms and their extensions with respect to particular problems, as well as reason about their performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
[25] present an overview over PG variants. They use terms like \(\sum _{t=0}^{\infty } r_t\) to indicate general correctness for infinite horizon problems. Still, in practice, those sums reflect the Monte Carlo PG variants, which operate on finite horizons and break down to finite sums.
- 2.
Notably, in policy optimization for uncountable action sets, the learned distribution does not have to accurately reflect the distribution of the true Q-values, as they are not used to estimate state-values.
- 3.
As an example, assume a Q-values estimator wrongly predicts all values as 0. This would always make the result of the gradient calculations 0 as well.
- 4.
See the Spinning Up article: https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html.
- 5.
See proof for exact Q-values in Spinning Up documentation: https://spinningup.openai.com/en/latest/spinningup/extra_pg_proof2.html.
- 6.
See this discussion on Stack Exchange: https://ai.stackexchange.com/questions/11679.
References
Achiam, J.: Spinning up in deep reinforcement learning (2018). https://spinningup.openai.com/
Doya, K.: Reinforcement learning in continuous time and space. Neural Comput. 12(1), 219–245 (2000)
Eysenbach, B., Levine, S.: Maximum entropy RL (provably) solves some robust RL problems. arXiv preprint arXiv:2103.06257 (2021)
Fakoor, R., Chaudhari, P., Smola, A.J.: P3O: policy-on policy-off policy optimization. In: Uncertainty in Artificial Intelligence, pp. 1017–1027. PMLR (2020)
Fujimoto, S., et al.: Addressing function approximation error in actor-critic methods. In: ICML, pp. 1587–1596. PMLR (2018)
Géron, A.: Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. O’Reilly Media, Inc., Sebastopol (2019)
Haarnoja, T., et al.: Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 (2018)
Haarnoja, T., et al.: Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: ICML, pp. 1861–1870. PMLR (2018)
Hasselt, H.: Double q-learning. Adv. Neural Inf. Process. Syst. 23 (2010)
Henderson, P., et al.: Deep reinforcement learning that matters. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1 (2018). https://doi.org/10.1609/aaai.v32i1.11694, https://ojs.aaai.org/index.php/AAAI/article/view/11694
Hussenot, L., et al.: Hyperparameter selection for imitation learning. In: ICML, pp. 4511–4522. PMLR (2021)
Kirk, D.E.: Optimal Control Theory: An Introduction. Courier Corporation, Chelmsford (2004)
Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114(13), 3521–3526 (2017)
Liessner, R., et al.: Hyperparameter optimization for deep reinforcement learning in vehicle energy management. In: ICAART (2), pp. 134–144 (2019)
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
Mahmood, A.R., et al.: Benchmarking reinforcement learning algorithms on real-world robots. In: Conference on Robot Learning, pp. 561–591. PMLR (2018)
Mankowitz, D.J., et al.: Robust reinforcement learning for continuous control with model misspecification. arXiv preprint arXiv:1906.07516 (2019)
McFarlane, R.: A survey of exploration strategies in reinforcement learning. McGill University (2018)
Minsky, M.: Steps toward artificial intelligence. Proc. IRE 49(1), 8–30 (1961)
Mnih, V., et al.: Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
Nikishin, E., et al.: Improving stability in deep reinforcement learning with weight averaging. In: Uncertainty in Artificial Intelligence Workshop on Uncertainty in Deep Learning (2018)
Ren, Z., Zhu, G., Hu, H., Han, B., Chen, J., Zhang, C.: On the estimation bias in double q-learning. Adv. Neural Inf. Process. Syst. 34 (2021)
Schaul, T., et al.: Prioritized experience replay. arXiv preprint arXiv:1511.05952 (2015)
Schröder, K., Kastius, A., Schlosser, R.: Welcome to the jungle: a conceptual comparison of reinforcement learning algorithms. In: Proceedings of the 12th International Conference on Operations Research and Enterprise Systems - Volume 1: ICORES, pp. 143–150 (2023)
Schulman, J., et al.: High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 (2015)
Schulman, J., et al.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)
Silver, D.: Lectures on reinforcement learning (2015). https://www.davidsilver.uk/teaching/
Silver, D., et al.: Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815 (2017)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018). https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf
Sutton, R.S., McAllester, D., Singh, S., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. Adv. Neural Inf. Process. Syst. 12 (1999)
Weaver, L., Tao, N.: The optimal reward baseline for gradient-based reinforcement learning. arXiv preprint arXiv:1301.2315 (2013)
Weng, J., et al.: Tianshou: a highly modularized deep reinforcement learning library. arXiv preprint arXiv:2107.14171 (2021)
Weng, L.: Policy gradient algorithms (2018). lilianweng.github.io, https://lilianweng.github.io/posts/2018-04-08-policy-gradient/
Weng, L.: Exploration strategies in deep reinforcement learning (2020). https://lilianweng.github.io/
Yildiz, C., et al.: Continuous-time model-based reinforcement learning. In: ICML, pp. 12009–12018. PMLR (2021). https://youtu.be/PIouASLg_-g
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Schröder, K., Kastius, A., Schlosser, R. (2024). Reinforcement Learning Algorithms: Categorization and Structural Properties. In: Liberatore, F., Wesolkowski, S., Demange, M., Parlier, G.H. (eds) Operations Research and Enterprise Systems. ICORES ICORES 2022 2023. Communications in Computer and Information Science, vol 1985. Springer, Cham. https://doi.org/10.1007/978-3-031-49662-2_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-49662-2_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-49661-5
Online ISBN: 978-3-031-49662-2
eBook Packages: Computer ScienceComputer Science (R0)