Yanpeng Huo
ABSTRACT
Artificial intelligence now has different applications in various industrial fields. Reinforcement learning (RL) is one of the hot topics in the artificial intelligence, also in robotics. It is an important learning method in the field of robotic manipulation. The training policies of reinforcement learning can be divided into online learning policy and offline learning policy. Besides, the reinforcement learning algorithm of offline policy has great potential in transforming large data sets into powerful decision engine. To solve the problem that most of robot applications involve collecting data from scratch for each new task, offline learning combined with online learning is to make the training more efficient and convenient. The aim of this paper is to clearly introduce the application of offline reinforcement learning in the field of robotic manipulation. The basic formulation of reinforcement learning includes two points: First, it introduces Markov Decision Process and one of method of solution – policy gradients. Then through analyzing an application of offline learning in the field of robotic manipulation - COG algorithm, this paper analyzes the process of offline learning combining the prior data to learn new robotic skills and uses this method to solve specific tasks of robotic, such as the problems of sample efficiency. The results show that the offline learning policy has important research value in the field of robotic manipulation by reducing training time and make process efficient, and it fully embodies its advantages in solving the problems of robotic sample efficiency.
- Yinlei Wen, Huaguang Zhang, Hanguang Su, He Ren 2020 Optimal tracking control for non-zero-sum games of linear discrete-time systems via off-policy reinforcement learning. Optimal Control Applications and MethodsGoogle Scholar
- Yoshida Hiroyuki 2019 Deep Learning and AlphaGo. Brain and nerve = Shinkei kenkyu no shinpoGoogle Scholar
- Priya Shukla, Hitesh Kumar;G C Nandi 2021 Robotic grasp manipulation using evolutionary computing and deep reinforcement learning. Intelligent Service RoboticsGoogle Scholar
- Michelle A. Lee, Carlos Florensa, Jonathan Tremblay, Nathan Ratliff, Animesh Garg, Fabio Ramos, Dieter Fox 2020 Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning. arXiv preprint arXiv:2005.10872v2Google Scholar
- Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, Ken Goldberg 2010 Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones. arXiv preprint arXiv:2010.15920v1Google Scholar
- Fei Guo, Xiaowei Zhou, Jiahuan Liu, Yun Zhang, Dequn Li, Huamin Zhou 2019 A reinforcement learning decision model for online process parameters optimization from offline data in injection molding. Applied Soft Computing JournalGoogle Scholar
- Berkenkamp, F., Turchetta, M., Schoellig, A., and Krause, A 2017 Safe model-based reinforcement learning with stability guarantees. In Advances in neural information processing systems, p 908–918Google Scholar
- Di Cao, Junbo Zhao, Guozhou Zhang, Bin Zhang, Zhou Liu, Zhe Chen, Frede Blaabjerg 2020 Reinforcement Learning and Its Applications: A Review. Journal of Modern Power Systems and Clean EnergyGoogle ScholarCross Ref
- Junzi Zhang, Jongho Kim, Brendan O'Donoghue, Stephen Boyd 2010 Sample Efficient Reinforcement Learning with REINFORCE. arXiv preprint arXiv:2010.11364v2Google Scholar
- J. Fu, A. Kumar, M. Soh, S. Levine 2019 Diagnosing bottlenecks in deep Q-learning algorithms. arXiv preprint arXiv:1902.10250Google Scholar
- S. Fujimoto, D. Meger, D. Precup 2018 Off-policy deep reinforcement learning without exploration. arXiv preprint arXiv:1812.02900Google Scholar
- Cabi, S., Colmenarejo, S. G., Novikov, A., Konyushkova, K., Reed, S., Jeong, R., Zołna, K., Aytar, ˙ Y., Budden, D., Vecerik, M., 2019 A framework for data-driven robotics. arXiv preprint arXiv:1909.12200Google Scholar
- Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, Sergey Levine 2020 COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning. arXiv preprint arXiv:2010.14500v1Google Scholar
- Jun Jin, Daniel Graves, Cameron Haigh, Jun Luo, Martin Jagersand 2020 Offline Learning of Counterfactual Perception as Prediction for Real-World Robotic Reinforcement Learning. arXiv preprint arXiv:2011.05857v1Google Scholar
- Sergey Levine, Aviral Kumar, George Tucker, Justin Fu 2020 Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv preprint arXiv:2005.01643v3Google Scholar
- T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, V. K. Jie Tan, H. Zhu, A. Gupta, P. Abbeel, S. Levine 2018 Soft actor-critic algorithms and applications. Technical reportGoogle Scholar
- A. Kumar, J. Fu, M. Soh, G. Tucker, S. Levine 2019 Stabilizing off-policy q-learning via bootstrapping error reduction. In NeurIPSGoogle Scholar
- A. Kumar, A. Zhou, G. Tucker, and S. Levine. 2020 Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779Google Scholar
Recommendations
Swarm Deep Reinforcement Learning for Robotic Manipulation
AbstractDeep reinforcement learning scheme, which combines both deep learning and reinforcement learning, enables robots to learn from exploration and flexibly performance in a range of different operational tasks under highly dynamic and complex ...
Reinforcement learning for appearance based visual servoing in robotic manipulation
ROCOM'08: Proceedings of the 8th WSEAS International Conference on Robotics, Control and Manufacturing TechnologyThe objective of this paper is to develop a new appearance based visual servoing method that needs no prior structuring of the environment and also eliminates the correspondence problem associated with conventional visual servoing methods. Detailed ...
Reinforcement learning for robotic manipulation using simulated locomotion demonstrations
AbstractMastering robotic manipulation skills through reinforcement learning (RL) typically requires the design of shaped reward functions. Recent developments in this area have demonstrated that using sparse rewards, i.e. rewarding the agent only when ...
Comments