Abstract
Reinforcement learning is one of the research hotspots in the field of artificial intelligence in recent years. In the past few years, deep reinforcement learning has been widely used to solve various decision-making problems. However, due to the characteristics of neural networks, it is very easy to fall into local minima when facing small scale discrete space path planning problems. Traditional reinforcement learning uses continuous updating of a single agent when algorithm executes, which leads to a slow convergence speed. Although some scholars have done some improvement work to solve these problems, there are still many shortcomings to be overcome. In order to solve the above problems, we proposed a new asynchronous tabular reinforcement learning algorithms framework in this paper, and present four new variants of asynchronous reinforcement learning algorithms. We apply these algorithms on the standard reinforcement learning environments: frozen lake problem, cliff walking problem and windy gridworld problem, and the simulation results show that these methods can solve discrete space path planning problems efficiently and well balance the exploration and exploitation.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Zhang C, Bi J, Xu S et al (2019) Multi-Imbalance: An open-source software for multi-class imbalance learning. Knowl-Based Syst. https://doi.org/10.1016/j.knosys.2019.03.001
Fujita H, Cimr D (2019) Computer Aided detection for fibrillations and flutters using deep convolutional neural network. Inf Sci 486:231–239
Zhang C, Liu C, Zhang X et al (2017) An up-to-date comparison of state-of-the-art classification algorithms. Expert Syst Appl 82:128–150
Xiao Q, Dai J, Luo J et al (2019) Multi-view manifold regularized learning-based method for prioritizing candidate disease miRNAs. Knowl-Based Syst. https://doi.org/10.1016/j.knosys.2019.03.023
Sutton R, Barto A (1998) Reinforcement learning: An introduction. MIT press, Cambridge
Silver D, Schrittwieser J, Simonyan K et al (2017) Mastering the game of Go without human knowledge. Nature 550(7676):354–359
Silver D, Huang A, Maddison M et al (2016) Mastering the game of Go with deep neural networks and tree search. Nature 529(7587):484–489
Mnih V, Kavukcuoglu K, Silver D et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533
Mnih V, Kavukcuoglu K, Silver D et al (2013) Playing atari with deep reinforcement learning. Proceedings of Workshops at the 26th Neural Information Processing Systems, Lake Tahoe, pp 201–220
Levine S, Pastor P, Krizhevsky A et al (2016) Learning Hand-Eye Coordination for Robotic Grasping with Large-Scale Data Collection. International Symposium on Experimental Robotics. Springer, Cham, 173–184
Lenz I, Knepper R, Saxena A (2015) Deepmpc: learning deep latent features for model predictive control. In: Proceedings of the Robotics Science and Systems, Rome, pp 201–209
Satija H, Pineau J (2016) Simultaneous machine translation using deep reinforcement learning. Proceedings of the Workshops of International Conference on Machine Learning, New York, pp 110–119
Guo H (2015) Generating text with deep reinforcement learning. Proceedings of the Workshops of Advances in Neural Information Processing Systems, Montreal, pp 1–9
Li J, Monroe W, Ritter A et al (2016) Deep reinforcement learning for dialogue generation. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Austin, pp 1192–1202
Caicedo J, Lazebnik S (2015) Active Object Localization with Deep Reinforcement Learning. IEEE International Conference on Computer Vision. IEEE, 2488–2496
Sutton R (1988) Learning to predict by the methods of temporal differences. Mach Learn 3(1):9–44
Watkins C (1989) Learning from delayed rewards. King's College, Cambridge
Rummery G, Niranjan M (1994) On-line Q-learning using connectionist systems. University of Cambridge, Department of Engineering, Cambridge
Singh S, Sutton R (1996) Reinforcement learning with replacing eligibility traces. Recent Advances in Reinforcement Learning, 123–158
Tsitsiklis J (1994) Asynchronous stochastic approximation and Q-learning. Mach Learn 16(3):185–202
Mnih V, Badia A, Mirza M et al (2016) Asynchronous methods for deep reinforcement learning. International Conference on Machine Learning, 1928–1937
Zhao X, Ding S, An Y et al (2018) Asynchronous Reinforcement Learning Algorithms for Solving Discrete Space Path Planning Problems. Appl Intell 48(12):4889–4904
Zhao X, Ding S, An Y et al (2019) Applications of asynchronous deep reinforcement learning based on dynamic updating weights. Appl Intell 49(2):581–591
Zhao X, Ding S, An Y (2018) A new asynchronous architecture for tabular reinforcement learning algorithms. Proceedings of the Eighth International Conference on Extreme Learning Machines, 172–180
Nair A, Srinivasan P, Blackwell S et al (2015) Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.04296
van Hasselt H (2010) Double Q-learning. Adv Neural Inf Proces Syst 23:2613–2621
Wang Y-H, Li T-H, Lin C-J (2013) Backward Q-learning: The combination of Sarsa algorithm and Q-learning. Eng Appl Artif Intell 26(9):2184–2193
Tokic M, Palm G (2011) Value-difference based exploration: adaptive control between epsilon-greedy and softmax. Annual Conference on Artificial Intelligence, Berlin, pp 335–346
Acknowledgements
This work is supported by the Fundamental Research Funds for the Central Universities (No.2017XKZD03).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Ding, S., Zhao, X., Xu, X. et al. An effective asynchronous framework for small scale reinforcement learning problems. Appl Intell 49, 4303–4318 (2019). https://doi.org/10.1007/s10489-019-01501-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-019-01501-9