skip to main content
10.1145/3573910.3573914acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicraiConference Proceedingsconference-collections
research-article

Autonomous Navigation and Obstacle Avoidance using Self-Guided and Self-Regularized Actor-Critic

Published: 20 January 2023 Publication History

Abstract

This paper presents a deep reinforcement learning (DRL) algorithm for autonomous navigation and obstacle avoidance of a mobile robot. Deep Reinforcement learning algorithms (DRL) have achieved great success in sequential decision-making problems and control tasks. One of the controlling elements of deep reinforcement learning (DRL) is the target network which alleviates the divergence when learning the Q function. But, target networks slow down the learning process due to delayed learning updates. So, this makes learning unstable and induces poor performance in high-dimensional domains. As a result, it is challenging to train and deploy robots in real-time using existing Deep Reinforcement learning algorithms, which require a large number of training examples to converge or perform better in various scenarios. To tackle this issue we have used an algorithm called self-guided and self-regularized actor-critic (GRAC) which doesn't require target networks for learning state-action values in high dimensional continuous state space and continuous actions. Where we have developed an experimental test bed in the ROS-Gazebo simulator by implementing self-guided self-regularized actor-critic (GRAC) algorithm for goal-directed navigation and obstacle avoidance tasks using the Turtlebot3 robot. The simulated experiments were conducted in two different environments. In comparison to existing algorithms such as DDPG, the simulation results demonstrate that the proposed algorithm exhibits improved performance and faster convergence in navigation and obstacle avoidance of a mobile robot in high dimensional continuous state space and continuous actions.

References

[1]
Inhwan Kim, Sarvar Hussain Nengroo, Dongsoo Har, “Reinforcement Learning for Navigation of Mobile Robot with LiDAR,” arXiv preprint arXiv:2112.02954, 2021.
[2]
M. G. Sarwar Murshed, James J. Carroll, Nazar Khan, Faraz Hussain, springer “Efficient Deployment of Deep Learning Models on Autonomous Robots in the ROS Environment”, springer: 13 November 2021, Volume 3 pp 215–243.
[3]
Daniel Zhang, Colleen P. Bailey, “Obstacle Avoidance and Navigation Utilizing Reinforcement Learning with Reward Shaping,” arXiv preprint arXiv:2003.12863, 2020.
[4]
Ryan K. Cosner, Ivan D. Jimenez Rodriguez, Tamas G. Molnar, Wyatt Ubellacker, Yisong Yue, Aaron D. Ames, Katherine L. Bouman, Self-Supervised Online Learning for Safety-Critical Control using Stereo Vision, arXiv preprint arXiv: 2203.01404, ICRA 2022.
[5]
Marco Cimdins, Sven Ole Schmidt, Peter Bartmann, Horst Hellbrück, Exploiting Ultra-Wideband Channel Impulse Responses for Device-Free Localization”, Sensors 2022.
[6]
Qingfeng Yao, Zeyu Zheng ZHENG, Liang Qi, Haitao Yuan, Xiwang Guo, Ming Zhao, Zhi Liu, and Tianji Yang, “Path Planning Method With Improved Artificial Potential Field—A Reinforcement Learning Perspective,” IEEE Access, Vol.8, pp-135513 – 135523, 2020.
[7]
Rafal Szczepanski, Tomasz Tarczewski, Krystian Erwinski, “Energy Efficient Local Path Planning Algorithm Based on Predictive Artificial Potential Field”, IEEE Vehicular Technology Society Section, Vol: 10, Page(s): 39729 – 39742, IEEE, 2022.
[8]
Tareq M. Shami, Ayman A. El-Saleh, Mohammed Alswaitti, Qasem Al-Tashi, Mhd Amen Summakieh ” Particle Swarm Optimization: A Comprehensive Survey”, Vol: 10, pages:10031 – 10061, IEEE 2022.
[9]
L. Tai, G. Paolo, and M. Liu, “Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation,” in International Conference on Intelligent Robots and Systems, 2017.
[10]
Qiang Fang, Xin Xu, Xitong Wang, Yujun Zeng, ” Target‐driven visual navigation in indoor scenes using reinforcement learning and imitation learning “, CAAI Transactions on Intelligence Technology, Vol 7, Issue: 2, Pages 167-176, June 2022.
[11]
Felipe Codevilla, Matthias Mülle,; Antonio López, Vladlen Koltun, Alexey Dosovitskiy, “End-to-End Driving Via Conditional Imitation Learning”, IEEE – ICRA, May 2018.
[12]
Charles Sun, Jedrzej Orbik, Coline Manon Devin, Brian H. Yang, Abhishek Gupta, Glen Berseth, Sergey Levine, “Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation”, Proceedings of the 5th Conference on Robot Learning, PMLR 164:308-319, 2022.
[13]
Shixiang Gu, Ethan Holly, Timothy Lillicrap, Sergey Levine, “Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates”, arXiv preprint arXiv:1610.00633, 2016.
[14]
Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, Ben Coppin, “Deep Reinforcement Learning in Large Discrete Action Spaces”, arXiv preprint arXiv:1512.07679, 2015.
[15]
T.P. Lillicrap, J.J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver and D. Wierstra, “Continuous control with deep reinforcement learning”, arXiv preprint arXiv:1509.02971, 2015.
[16]
Lin Shao, Yifan You, Mengyuan Yan, Qingyun Sun, Jeannette Bohg, “GRAC: Self-Guided and Self-Regularized Actor-Critic”, arXiv preprint arXiv:2009.08973, 2020
[17]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller, “Playing Atari with Deep Reinforcement Learning”, arXiv preprint arXiv:1312.5602, 2013.
[18]
Tai Lei, Liu Ming, “Towards cognitive exploration through deep reinforcement learning for mobile robots”. arXiv preprint arXiv:1610.01733, 2016.
[19]
L. Qiang, D. Nanxun, L. Huican and W. Heng, “A model-free mapless navigation method for mobile robot using reinforcement learning, In 2018 Chinese Control And Decision Conference (CCDC)”, pages 3410–3415. IEEE, 2018.
[20]
L. Tai, G. Paolo, and M. Liu, “Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation,” in Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ International Conference on. IEEE, 2017, pp. 31–36.
[21]
Daniel Zhanyg, Colleen P. Baile, “Obstacle Avoidance and Navigation Utilizing Reinforcement Learning with Reward Shaping”, arXiv preprint arXiv:2003.12863, 2020.
[22]
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, Pieter Abbeel, “Trust Region Policy Optimization”, arXiv preprint arXiv:1502.05477, 2015.
[23]
Scott Fujimoto, Herke van Hoof, David Meger, “Addressing Function Approximation Error in Actor-Critic Methods”, arXiv preprint arXiv:1802.09477, 2018.
[24]
John Schulman, Filip Wolski, Prafulla Dhariwal, “Alec Radford, Oleg Klimov, Proximal Policy Optimization Algorithms”, arXiv preprint arXiv:1707.06347, 2017.
[25]
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”, arXiv preprint arXiv:1801.01290, 2018.
[26]
Elad Sarafian, Aviv Tamar, Sarit Kraus, “Constrained Policy Improvement for Safe and Efficient Reinforcement Learning”, arXiv preprint arXiv:1805.07805, 2018.
[27]
Qingfeng Lan, Yangchen Pan, Alona Fyshe, Martha White, “Maxmin Q-learning: Controlling the Estimation Bias of Q-learning”, arXiv preprint arXiv:2002.06487, 2021.
[28]
Nanxun Duo; Qinzhao Wang; Qiang Lv; Heng Wei; Pei Zhang, “A Deep Reinforcement Learning Based Mapless Navigation Algorithm Using Continuous Actions, IEEE – ICRIS, May 2018.
[29]
Junior C. Jesus; Jair A. Bottega; Marco A. S. L. Cuadros; Daniel F. T. Gamarra, “Deep Deterministic Policy Gradient for Navigation of Mobile Robots in Simulated Environments”, IEEE – ICAR, 2019.
[30]
Enrico Marchesini; Alessandro Farinelli, “Discrete Deep Reinforcement Learning for Mapless Navigation”, IEEE – ICRA, 2020.
[31]
Morgan Quigley, Brian Gerkey, Ken Conley, Josh Faust, Tully Foote, Jeremy Leibs, Eric Berger, Rob Wheeler, Andrew Ng, “ROS: an open-source Robot Operating System”, ICRA, 2009.
[32]
Corey Williams, Adam Schroeder, “Utilizing ROS 1 and the Turtlebot3 in a Multi-Robot System”, arXiv arXiv:2011.10488, 2020.
[33]
Marius Marian, Florin Stinga, Marian-Terxinius Georgescu, Horatiu Roibu, Dorin Popescu, Florin Manta, “A ROS-based Control Application for a Robotic Platform Using the Gazebo 3D Simulator”, IEEE- ICCC, 2020
[34]
N. Koenig; A. Howard, “Design and use paradigms for Gazebo, an open-source multi-robot simulator”, IEEE-IROS, 2004.
[35]
Erico Guizzo; Evan Ackerman, “The TurtleBot3 Teacher [Resources_Hands On]”, IEEE Spectrum ( Volume: 54, Issue: 8, 2017)
[36]
Guimaraes, R.L.; de Oliveira, A.S.; Fabro, J.A.; Becker, T.; Brenner, V.A, “ROS navigation: Concepts and tutorial. In Robot Operating System (ROS)”, Springer: Berlin, Germany, 2016; pp. 121–160.

Cited By

View all
  • (2024)OpenBot-Fleet: A System for Collective Learning with Real Robots2024 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA57147.2024.10610960(4758-4765)Online publication date: 13-May-2024
  • (2024)Analysis on 2D Mapping for Mobile Robotonthesharped Edge Area2024 9th International Conference on Mechatronics Engineering (ICOM)10.1109/ICOM61675.2024.10652363(255-263)Online publication date: 13-Aug-2024
  • (2023)Autonomous Navigation of Robots: Optimization with DQNApplied Sciences10.3390/app1312720213:12(7202)Online publication date: 16-Jun-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ICRAI '22: Proceedings of the 8th International Conference on Robotics and Artificial Intelligence
November 2022
89 pages
ISBN:9781450397544
DOI:10.1145/3573910
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 January 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Navigation
  2. Obstacle Avoidance
  3. Reinforcement learning
  4. Self-Guided Self-Regularized Actor-Critic (GRAC)

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ICRAI 2022

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)21
  • Downloads (Last 6 weeks)2
Reflects downloads up to 25 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)OpenBot-Fleet: A System for Collective Learning with Real Robots2024 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA57147.2024.10610960(4758-4765)Online publication date: 13-May-2024
  • (2024)Analysis on 2D Mapping for Mobile Robotonthesharped Edge Area2024 9th International Conference on Mechatronics Engineering (ICOM)10.1109/ICOM61675.2024.10652363(255-263)Online publication date: 13-Aug-2024
  • (2023)Autonomous Navigation of Robots: Optimization with DQNApplied Sciences10.3390/app1312720213:12(7202)Online publication date: 16-Jun-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media