Skip to main content

Advertisement

Springer Nature Link
Log in

R3T*-MOSafeRL(\(\lambda \)): path planning of mobile robots in unknown dynamic environments

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Path planning has a wide range of applications in many fields of engineering. One of the widely used approaches to solve this problem is sampling-based algorithms. However, a significant limitation of these algorithms is their inability to function effectively in the presence of unknown obstacles within the environment. A feasible way of addressing this issue is by using reinforcement learning to interact with the environment. In the context of path planning, safety means choosing safe and non-optimal paths over the optimal ones and having as few collisions as possible during both the training phase and the test phase. In this paper, a novel two-stage algorithm called R3T*-MOSafeRL(\(\lambda \)) is proposed capable of path planning safely in an environment with unknown dynamic obstacles. In the first stage, the Roadmap Multi-Tree RRT* (R3T*) algorithm is presented which uses an initial map of the environment and an expert’s information regarding the important regions of the environment to generate a roadmap. The roadmap acts as discretization of the continuous environment to practically deploy tabular reinforcement learning algorithms on top of it. In the next stage, the algorithm uses a novel eligibility trace-based multi-objective safe reinforcement learning (MOSafeRL(\(\lambda \))) to perform safe path planning on the roadmap generated by R3T*. Moreover, a heatmap algorithm based on the roadmap and the weights learned by MOSafeRL(\(\lambda \)) is presented which provides an interpretable method to gain information about the regions of the map with high activity of the unknown dynamic obstacles. Hence, the proposed algorithm provides a powerful method for path planning in unknown dynamic environments. Finally, to illustrate the efficiency of the proposed algorithm and to verify it, some case studies are considered and the computational results are compared and discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Fig. 1
Algorithm 2
Fig. 2
Fig. 3
Algorithm 3
Algorithm 4
Algorithm 5
Algorithm 6
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Orozco-Rosas U, Montiel O, Sepúlveda R (2019) Mobile robot path planning using membrane evolutionary artificial potential field. Appl Soft Comput 77:236–251

    Article  Google Scholar 

  2. Faridi AQ, Sharma S, Shukla A, Tiwari R, Dhar J (2018) Multi-robot multi-target dynamic path planning using artificial bee colony and evolutionary programming in unknown environment. Intel Serv Robot 11:171–186

    Article  Google Scholar 

  3. Wu L, Huang X, Cui J, Liu C, Xiao W (2023) Modified adaptive ant colony optimization algorithm and its application for solving path planning of mobile robot. Expert Syst Appl 215:119410

    Article  Google Scholar 

  4. Montiel O, Orozco-Rosas U, Sepúlveda R (2015) Path planning for mobile robots using bacterial potential field for avoiding static and dynamic obstacles. Expert Syst Appl 42(12):5177–5191

    Article  Google Scholar 

  5. Ajeil FH, Ibraheem IK, Sahib MA, Humaidi AJ (2020) Multi-objective path planning of an autonomous mobile robot using hybrid pso-mfb optimization algorithm. Appl Soft Comput 89:106076

    Article  Google Scholar 

  6. LaValle SM, Kuffner JJ, Donald B et al (2001) Rapidly-exploring random trees: progress and prospects. Algorithm Comput Robot New Dir 5:293–308

    Google Scholar 

  7. Karaman S, Frazzoli E (2011) Sampling-based algorithms for optimal motion planning. Int J Robot Res 30(7):846–894

    Article  Google Scholar 

  8. Kuffner JJ, LaValle SM (2000) Rrt-connect: an efficient approach to single-query path planning. In: Proceedings 2000 ICRA. Millennium conference. IEEE international conference on robotics and automation. Symposia proceedings (Cat. No. 00CH37065). IEEE, vol. 2, pp 995–1001

  9. Jordan M, Perez A (2013) Optimal bidirectional rapidly-exploring random trees

  10. Martin SR, Wright SE, Sheppard JW (2007) Offline and online evolutionary bi-directional rrt algorithms for efficient re-planning in dynamic environments. In: 2007 IEEE international conference on automation science and engineering. IEEE, pp 1131–1136

  11. Zammit C, Kampen E-J (2023) Real-time 3d uav path planning in dynamic environments with uncertainty. Unmanned Syst 11(03):203–219

    Article  Google Scholar 

  12. Qi J, Yang H, Sun H (2020) Mod-rrt*: a sampling-based algorithm for robot path planning in dynamic environment. IEEE Trans Ind Electron 68(8):7244–7251

    Article  Google Scholar 

  13. Eshtehardian S, Khodaygan S (2023) A continuous rrt*-based path planning method for non-holonomic mobile robots using b-spline curves. J Ambient Intell Humaniz Comput 14(7):8693–8702

    Article  Google Scholar 

  14. Yao Q, Zheng Z, Qi L, Yuan H, Guo X, Zhao M, Liu Z, Yang T (2020) Path planning method with improved artificial potential field-a reinforcement learning perspective. IEEE Access 8:135513–135523

    Article  Google Scholar 

  15. Sabzekar S, Samadzad M, Mehditabrizi A, Tak AN (2023) A deep reinforcement learning approach for uav path planning incorporating vehicle dynamics with acceleration control. Unmanned Syst 12:477

    Article  Google Scholar 

  16. Sombolestan S, Rasooli A, Khodaygan S (2019) Optimal path-planning for mobile robots to find a hidden target in an unknown environment based on machine learning. J Ambient Intell Humaniz Comput 10:1841–1850

    Article  Google Scholar 

  17. Qu C, Gai W, Zhong M, Zhang J (2020) A novel reinforcement learning based Grey Wolf optimizer algorithm for unmanned aerial vehicles (uavs) path planning. Appl Soft Comput 89:106099

    Article  Google Scholar 

  18. Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT press, Cambridge

    Google Scholar 

  19. Lei X, Zhang Z, Dong P (2018) Dynamic path planning of unknown environment based on deep reinforcement learning. J Robot 2018:5781591

    Google Scholar 

  20. Fan T, Long P, Liu W, Pan J (2020) Distributed multi-robot collision avoidance via deep reinforcement learning for navigation in complex scenarios. Int J Robot Res 39(7):856–892

    Article  Google Scholar 

  21. Cimurs R, Suh IH, Lee JH (2021) Goal-driven autonomous exploration through deep reinforcement learning. IEEE Robot Autom Lett 7(2):730–737

    Article  Google Scholar 

  22. Tao W, Huang H (2023) Fast and robust training and deployment of deep reinforcement learning based navigation policy. In: 2023 IEEE international conference on unmanned systems (ICUS). IEEE, pp 1581–1586

  23. Wen S, Zhao Y, Yuan X, Wang Z, Zhang D, Manfredi L (2020) Path planning for active slam based on deep reinforcement learning under unknown environments. Intel Serv Robot 13:263–272

    Article  Google Scholar 

  24. Qin H, Qiao B, Wu W, Deng Y (2022) A path planning algorithm based on deep reinforcement learning for mobile robots in unknown environment. In: 2022 IEEE 5th advanced information management, communicates, electronic and automation control conference (IMCEC). IEEE, vol. 5, pp 1661–1666

  25. Trott A, Zheng S, Xiong C, Socher R (2019) Keeping your distance: solving sparse reward tasks using self-balancing shaped rewards. Adv Neural Inf Process Syst 32:10376

  26. Otte M, Frazzoli E (2015) \({\text{RRT}}^{\text{ X }}\): real-time motion planning/replanning for environments with unpredictable obstacles. In: Algorithmic foundations of robotics XI: selected contributions of the eleventh international workshop on the algorithmic foundations of robotics, pp 461–478. Springer

  27. Adiyatov O, Varol HA (2017) A novel rrt-based algorithm for motion planning in dynamic environments. In: 2017 IEEE international conference on mechatronics and automation (ICMA). IEEE, pp. 1416–1421

  28. Garcıa J, Fernández F (2015) A comprehensive survey on safe reinforcement learning. J Mach Learn Res 16(1):1437–1480

    MathSciNet  Google Scholar 

  29. Horie N, Matsui T, Moriyama K, Mutoh A, Inuzuka N (2019) Multi-objective safe reinforcement learning: the relationship between multi-objective reinforcement learning and safe reinforcement learning. Artif Life Robot 24:352–359

    Article  Google Scholar 

  30. Aissani N, Beldjilali B, Trentesaux D (2008) Efficient and effective reactive scheduling of manufacturing system using sarsa-multi-objective agents. In: MOSIM’08: 7th conference internationale de modelisation et simulation, pp 698–707

  31. Takeyama D, Kanoh M, Matsui T, Nakamura T (2015) Obtaining robot’s behavior to avoid danger by using probability based reinforcement learning. J Jpn Soc Fuzzy Theory Intell Inform 27(6):877–884

    Google Scholar 

  32. Horie N, Matsui T, Moriyama K, Mutoh ANI (2016) Reinforcement learning based on action values combined with success probability and profit. In: Proceedings of the 30th annual conference of the Japanese society for artifcial intelligence

  33. Devaurs D, Siméon T, Cortés J (2014) A multi-tree extension of the transition-based rrt: application to ordering-and-pathfinding problems in continuous cost spaces. In: 2014 IEEE/RSJ international conference on intelligent robots and systems. IEEE, pp 2991–2996

  34. Kavraki LE, Svestka P, Latombe J-C, Overmars MH (1996) Probabilistic roadmaps for path planning in high-dimensional configuration spaces. IEEE Trans Robot Autom 12(4):566–580

    Article  Google Scholar 

  35. Thakar S (2021) Planning for mobile manipulation. PhD thesis, University of Southern California

  36. Tsardoulias EG, Iliakopoulou A, Kargakos A, Petrou L (2016) A review of global path planning methods for occupancy grid maps regardless of obstacle density. J Intell Robot Syst 84:829–858

    Article  Google Scholar 

  37. Ravankar A, Ravankar AA, Kobayashi Y, Hoshino Y, Peng C-C (2018) Path smoothing techniques in robot navigation: state-of-the-art, current and future challenges. Sensors 18(9):3170

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering, Sharif University of Technology, Azadi Ave., Tehran, Iran

    Homayoun Honari & Saeed Khodaygan

  2. Department of Mechanical Engineering, University of Victoria, 3800 Finnerty Rd., Victoria, Canada

    Homayoun Honari

Authors
  1. Homayoun Honari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Saeed Khodaygan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saeed Khodaygan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Honari, H., Khodaygan, S. R3T*-MOSafeRL(\(\lambda \)): path planning of mobile robots in unknown dynamic environments. Intel Serv Robotics 17, 1175–1188 (2024). https://doi.org/10.1007/s11370-024-00566-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-024-00566-x

Keywords