Abstract
Conventional collaborative robots can solve complex problems through programming approaches. But the current tasks are different and nonrepetitive, many problems cannot be solved by conventional programming methods. Deep reinforcement learning provides a framework for solving robotic control tasks using machine learning techniques. However, the existing model-free deep reinforcement learning algorithms lack unified framework for comparing sample efficiency with final performance. In this paper, a hybrid deep reinforcement learning framework and its application in robot control are proposed based on the existing model-free deep reinforcement learning algorithms. In the acting process, the distributed actors acting with the environment are used to acquire the data, while prior actors are used to solve the cold boot problem of the algorithm. In the learning process, prioritized experience replay and multi-step learning are designed for the improvement on the final performance. Simulations are represented to show the practicality and potential of the proposed algorithm. Results show that the hybrid deep reinforcement learning algorithm in this paper has a significant improvement on the final performance and sample efficiency while it can ensure the stability and convergence.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Vecerik, M., Hester, T., Scholz, J., et al.: Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 (2017)
Tai, L., Zhang, J., Liu, M., et al.: A survey of deep network solutions for learning control in robotics: from reinforcement to imitation. arXiv preprint arXiv:1612.07139 (2016)
Barto, G., Sutton, S., Anderson, W.: Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. 5, 834–846 (1984)
Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Playing Atari with deep reinforcement learning. In: Neural Information Processing Systems (2013)
Lillicrap, P., Hunt, J., Pritzel, A., et al.: Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015)
Mnih, V., Badia, P., Mirza, M., et al.: Asynchronous methods for deep reinforcement learning. arXiv preprint arXiv:1602.01783 (2016)
Wu, Y., Mansimov, E., Grosse, B., et al.: Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation. In: Neural Information Processing Systems, pp. 5279–5288 (2017)
Schulman, J., Levine, S., Abbeel, P., et al.: Trust region policy optimization. In: 32nd International Conference on Machine Learning (ICML 2015), pp. 1889–1897 (2015)
Schulman, J., Wolski, F., Dhariwal, P., et al.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)
Goodfellow, I., Pouget, J., Mirza, M., et al.: Generative adversarial nets. In: Neural Information Processing Systems, pp. 2672–2680 (2014)
Dean, J., Corrado, G., Monga, R., et al.: Large scale distributed deep networks. In: 25th International Conference on Neural Information Processing Systems (2012)
Nair, A., Srinivasan, P., Blackwell, S., et al.: Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.04296 (2015)
Horgan, D., Quan, J., Budden, D., et al.: Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933 (2018)
Hessel, M., Modayil, J., Hasselt, H., et al.: Rainbow: combining improvements in deep reinforcement learning. arXiv preprint arXiv:1710.02298 (2017)
Babaeizadeh, M., Frosio, I., Tyree, S., et al.: Reinforcement learning through asynchronous advantage actor-critic on a gpu. arXiv preprint arXiv:1611.06256 (2016)
Google. https://github.com/openai/gym/tree/master/gym/envs/mujoco. Accessed 06 May 2019
Li, J., Liu, L., Wang, Y., et al.: Adaptive hybrid impedance control of robot manipulators with robustness against environment’s uncertainties. In: 2015 IEEE International Conference on Mechatronics and Automation, pp. 1846–1851. IEEE (2015)
Acknowledgment
This research was supported, in part, by the National Natural Science Foundation of China (No. 51875393) and by the China Advance Research for Manned Space Project (No. 030601).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Ma, C., Li, J., Bai, J., Wang, Y., Liu, B., Sun, J. (2019). A Hybrid Deep Reinforcement Learning Algorithm for Intelligent Manipulation. In: Yu, H., Liu, J., Liu, L., Ju, Z., Liu, Y., Zhou, D. (eds) Intelligent Robotics and Applications. ICIRA 2019. Lecture Notes in Computer Science(), vol 11743. Springer, Cham. https://doi.org/10.1007/978-3-030-27538-9_31
Download citation
DOI: https://doi.org/10.1007/978-3-030-27538-9_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-27537-2
Online ISBN: 978-3-030-27538-9
eBook Packages: Computer ScienceComputer Science (R0)