Long-Horizon Manipulation by a Single-arm Robot via Sub-goal Network based Hierarchical Reinforcement Learning
Authors: 
Jin Gyun Jeong, Ji-Heon Oh, Hwanseok Jung, Jin Hyuk Lee, Ismael Nicolas Espinoza Jaramillo, Channabasava Chola, Won Hee Lee, Tae-Seong KimAuthors Info & Claims
Jin Gyun Jeong, Ji-Heon Oh, Hwanseok Jung, Jin Hyuk Lee, Ismael Nicolas Espinoza Jaramillo, Channabasava Chola, Won Hee Lee, Tae-Seong KimAuthors Info & ClaimsPages 88 - 92
Abstract
In this work, we present an approach of long-horizon intelligence that utilizes Sub-goal network based hierarchical reinforcement learning (HRL) for long-horizon tasks by a single-arm robot. Long-horizon (LH) tasks are complicated due to their longer complex sequences and the large number of environmental variables. We attempt to solve the LH learning problem by the Sub-goal network based HRL. The proposed approach is tested in both simulation and hardware environments by a LH task of opening a drawer, grasping and relocating an object, and closing a drawer. Our Sub-goal network based HRL achieves a success rate of 90.3% in completing the LH tasks. Whereas the conventional deep reinforcement learning solution could not complete the LH task.
References
[1]
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. 2018. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Robotics: Science and Systems XIV, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA, June 26-30, 2018.
[2]
Sham Kakade. 2001. A natural policy gradient. In Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic (NIPS'01). MIT Press, Cambridge, MA, USA, 1531–1538.
[3]
Nachum, O., Gu, S., Lee, H., and Levine, S. 2018. Data-efficient hierarchical reinforcement learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS '18). Curran Associates Inc., Red Hook, NY, USA, 3307–3317.
[4]
Beyret, B., Shafti, A., and Faisal, A. A. 2019. Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulation. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE Press, 5014–5019. https://doi.org/10.1109/IROS40897.2019.8968488
[5]
Zhang, J., Yu, H., and Xu, W. 2021. Hierarchical Reinforcement Learning by Discovering Intrinsic Options. In International Conference on Learning Representations.
[6]
Scheiderer, C., Mosbach, M., Posada-Moreno, A. F., and Meisen, T. 2020. Transfer of Hierarchical Reinforcement Learning Structures for Robotic Manipulation Tasks. In 2020 International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, NV, USA, 504–509. https://doi.org/10.1109/CSCI51800.2020.00091.
[7]
Kulkarni, T. D., Narasimhan, K., Saeedi, A., and Tenenbaum, J. 2016. Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation. In NeurIPS 2016.
[8]
Sutton, R. S., Precup, D., and Singh, S. P. 1999. Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artificial Intelligence.
[9]
Li, B., Li, J., Lu, T., Cai, Y., and Wang, S. 2021. Hierarchical Learning from Demonstrations for Long-Horizon Tasks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi'an, China, 4545–4551. https://doi.org/10.1109/ICRA48506.2021.9561408.
[10]
Yang, X., 2022. Hierarchical Reinforcement Learning with Universal Policies for Multistep Robotic Manipulation. IEEE Transactions on Neural Networks and Learning Systems 33, 9 (2022), 4727–4741. https://doi.org/10.1109/TNNLS.2021.3059912.
[11]
Rosete-Beas, E., Mees, O., Kalweit, G., Boedecker, J., and Burgard, W. 2022. Latent Plans for Task-Agnostic Offline Reinforcement Learning. In 6th Annual Conference on Robot Learning.
[12]
Amari, S. 1998. Natural Gradient Works Efficiently in Learning. Neural Computation 10, 2 (Feb. 1998), 251–276. https://doi.org/10.1162/089976698300017746.
[13]
Todorov, E., Erez, T., and Tassa, Y. 2012. MuJoCo: A Physics Engine for Model-Based Control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, Portugal, 5026–5033.
[14]
Universal Robots. UR3 Robot. [Online]. Available: https://www.universal-robots.com/products/ur3-robot/.
[15]
Kumar, V., Xu, Z., and Todorov, E. 2013. Fast, Strong and Compliant Pneumatic Actuation for Dexterous Tendon-Driven Hands. In 2013 IEEE International Conference on Robotics and Automation, Karlsruhe, Germany, 1512–1519.
[16]
QB Robotics. QB Hand. [Online]. Available: https://qbrobotics.com/.
[17]
Intel. Intel RealSense Depth Camera D415. [Online]. Available: https://www.intelrealsense.com/depth-camera-d415/.
[18]
Ren, S., He, K., Girshick, R., and Sun, J. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Neural Information Processing Systems, Montreal, Canada, 91–99.
Index Terms
- Long-Horizon Manipulation by a Single-arm Robot via Sub-goal Network based Hierarchical Reinforcement Learning
Recommendations
Hierarchical Reinforcement Learning: A Comprehensive Survey
Hierarchical Reinforcement Learning (HRL) enables autonomous decomposition of challenging long-horizon decision-making tasks into simpler subtasks. During the past years, the landscape of HRL research has grown profoundly, resulting in copious ...
Multi-robot Cooperation Based on Hierarchical Reinforcement Learning
ICCS '07: Proceedings of the 7th international conference on Computational Science, Part III: ICCS 2007Multi-agent reinforcement learning for multi-robot systems is a challenging issue in both robotics and artificial intelligence. But multi-agent reinforcement learning is bedeviled by the curse of dimensionality. In this paper, a novel hierarchical ...
Transfer in variable-reward hierarchical reinforcement learning
Transfer learning seeks to leverage previously learned tasks to achieve faster learning in a new task. In this paper, we consider transfer learning in the context of related but distinct Reinforcement Learning (RL) problems. In particular, our RL ...
Comments
Information & Contributors
Information
Published In
June 2023
271 pages
ISBN:9798400707438
DOI:10.1145/3620679
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 19 December 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ICBET 2023
ICBET 2023: 2023 13th International Conference on Biomedical Engineering and Technology
June 15 - 18, 2023
Tokyo, Japan
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 57Total Downloads
- Downloads (Last 12 months)40
- Downloads (Last 6 weeks)2
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format