Skip to main content

Advertisement

Mapless autonomous navigation for UGV in cluttered off-road environment with the guidance of wayshowers using deep reinforcement learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Navigating unmanned ground vehicle (UGV) through off-road environments is critical for various tasks like exploration and rescue. Unlike scenarios allowing offline global planning based on prior knowledge, online navigation becomes essential due to the dynamic nature of these tasks. Although deep reinforcement learning (DRL) offers promise for mapless autonomous navigation due to its end-to-end advantages, existing approaches often rely solely on goal positions. This neglects the complex distribution of obstacles along the path, leading to inefficient interactions with the environment during training. To address this challenge, a deep reinforcement learning framework is proposed for autonomous navigation guided by wayshowers. Initially, a new metric is developed based on multilevel analysis to generate elevation maps, aiding in the identification of optimal wayshowers. Upon integrating wayshower information with other inputs, a multi-head attention (MHA) module is incorporated into DRL network, which includes a length attention mechanism to enhance focus on recent historical observation sequences to promote model convergence. Furthermore, the reward function is reshaped to offer dense reward signals, thereby resolving the sparse reward problem inherent in goal-driven methods. To validate the proposed approach, experiments are conducted on several off-road maps in both the Carla and Gazebo simulators. The results demonstrate the superiority of our method not only in simple environments but also in more challenging scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

References

  1. Kiran BR, Sobh I, Talpaert V et al (2021) Deep reinforcement learning for autonomous driving: A survey. IEEE Trans Intell Transp Syst 23(6):4909–4926. https://doi.org/10.1109/TITS.2021.3054625

    Article  MATH  Google Scholar 

  2. Thoresen M, Nielsen NH, Mathiassen K et al (2021) Path planning for ugvs based on traversability hybrid a. IEEE Robot Autom Lett 6(2):1216–1223. https://doi.org/10.1109/LRA.2021.3056028

    Article  MATH  Google Scholar 

  3. Qi Y, He B, Wang R et al (2022) Hierarchical motion planning for autonomous vehicles in unstructured dynamic environments. IEEE Robot Autom Lett 8(2):496–503. https://doi.org/10.1109/LRA.2022.3228159

    Article  MATH  Google Scholar 

  4. Ye L, Wu F, Zou X et al (2023) Path planning for mobile robots in unstructured orchard environments: An improved kinematically constrained bi-directional rrt approach. Comput Electron Agric 215:108453. https://doi.org/10.1016/j.compag.2023.108453

    Article  Google Scholar 

  5. Roriz R, Cabral J, Gomes T (2022) Automotive lidar technology: A survey. IEEE Trans Intell Transp Syst 23(7):6282–6297. https://doi.org/10.1109/TITS.2021.3086804

    Article  MATH  Google Scholar 

  6. Kong D, Li X, Xu Q et al (2024) Sc_lpr: Semantically consistent lidar place recognition based on chained cascade network in long-term dynamic environments. IEEE Trans Image Process 33(1):2145–2157. https://doi.org/10.1109/TIP.2024.3364511

    Article  MATH  Google Scholar 

  7. Chib PS, Singh P (2023) Recent advancements in end-to-end autonomous driving using deep learning: A survey. IEEE Trans Intell Veh 9(1):103–118. https://doi.org/10.1109/TIV.2023.3318070

    Article  MATH  Google Scholar 

  8. Li F, Guo C, Luo B et al (2021) Multi goals and multi scenes visual mapless navigation in indoor using meta-learning and scene priors. Neurocomputing 449:368–377. https://doi.org/10.1016/j.neucom.2021.03.084

    Article  MATH  Google Scholar 

  9. Jiang H, Esfahani MA, Wu K et al (2022) itd3-cln: Learn to navigate in dynamic scene through deep reinforcement learning. Neurocomputing 503:118–128. https://doi.org/10.1016/j.neucom.2022.06.102

    Article  Google Scholar 

  10. Staroverov A, Yudin DA, Belkin I et al (2020) Real-time object navigation with deep neural networks and hierarchical reinforcement learning. IEEE Access 8:195608–195621. https://doi.org/10.1109/ACCESS.2020.3034524

    Article  Google Scholar 

  11. Li Z, Zhou A (2023) Rddrl: a recurrent deduction deep reinforcement learning model for multimodal vision-robot navigation. Appl Intell 53(20):23244–23270. https://doi.org/10.1007/s10489-023-04754-7

    Article  Google Scholar 

  12. Rais MS, Boudour R, Zouaidia K et al (2023) Decision making for autonomous vehicles in highway scenarios using harmonic sk deep sarsa. Appl Intell 53(3):2488–2505. https://doi.org/10.1007/s10489-022-03357-y

    Article  Google Scholar 

  13. Hu J, Li X, Hu W et al (2024) A cooperative control methodology considering dynamic interaction for multiple connected and automated vehicles in the merging zone. IEEE Trans Intell Transp Syst 25(9):12669–12681. https://doi.org/10.1109/TITS.2024.3386200

    Article  MATH  Google Scholar 

  14. Hu J, Li X, Cen Y et al (2022) A roadside decision-making methodology based on deep reinforcement learning to simultaneously improve the safety and efficiency of merging zone. IEEE Trans Intell Transp Syst 23(10):18620–18631. https://doi.org/10.1109/TITS.2022.3157910

    Article  Google Scholar 

  15. Kahn G, Abbeel P, Levine S (2021) Badgr: An autonomous self-supervised learning-based navigation system. IEEE Robot Autom Lett 6(2):1312–1319. https://doi.org/10.1109/LRA.2021.3057023

    Article  Google Scholar 

  16. Hu H, Zhang K, Tan AH et al (2021) A sim-to-real pipeline for deep reinforcement learning for autonomous robot navigation in cluttered rough terrain. IEEE Robot Autom Lett 6(4):6569–6576. https://doi.org/10.1109/LRA.2021.3093551

    Article  MATH  Google Scholar 

  17. Josef S, Degani A (2020) Deep reinforcement learning for safe local planning of a ground vehicle in unknown rough terrain. IEEE Robot Autom Lett 5(4):6748–6755. https://doi.org/10.1109/LRA.2020.3011912

    Article  MATH  Google Scholar 

  18. Zhu W, Hayashibe M (2023) A hierarchical deep reinforcement learning framework with high efficiency and generalization for fast and safe navigation. IEEE Trans Ind Electron 70(5):4962–4971. https://doi.org/10.1109/TIE.2022.3190850

    Article  MATH  Google Scholar 

  19. Zhang K, Niroui F, Ficocelli M et al (2018) Robot navigation of environments with unknown rough terrain using deep reinforcement learning. In: 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), IEEE, pp 1–7. https://doi.org/10.1109/SSRR.2018.8468643

  20. Cimurs R, Suh IH, Lee JH (2022) Goal-driven autonomous exploration through deep reinforcement learning. IEEE Robot Autom Lett 7(2):730–737. https://doi.org/10.1109/LRA.2021.3133591

    Article  Google Scholar 

  21. Weerakoon K, Sathyamoorthy AJ, Patel U et al (2022) Terp: Reliable planning in uneven outdoor environments using deep reinforcement learning. In: 2022 International Conference on Robotics and Automation (ICRA), pp 9447–9453. https://doi.org/10.1109/ICRA46639.2022.9812238

  22. Samsani SS, Mutahira H, Muhammad MS (2022) Memory-based crowd-aware robot navigation using deep reinforcement learning. Complex Intell Syst 9:2147–2158. https://doi.org/10.1007/s40747-022-00906-3

    Article  Google Scholar 

  23. Huang W, Zhou Y, He X et al (2024) Goal-guided transformer-enabled reinforcement learning for efficient autonomous navigation. IEEE Trans Intell Transp Syst 25(2):1832–1845. https://doi.org/10.1109/TITS.2023.3312453

    Article  MATH  Google Scholar 

  24. Zhou K, Guo C, Zhang H et al (2023) Optimal graph transformer viterbi knowledge inference network for more successful visual navigation. Adv Eng Inf 55:101889. https://doi.org/10.1016/j.aei.2023.101889

    Article  MATH  Google Scholar 

  25. Yang Y, Jiang J, Zhang J et al (2023) St\(^{2}\): Spatial-temporal state transformer for crowd-aware autonomous navigation. IEEE Robot Autom Lett 8(2):912–919. https://doi.org/10.1109/LRA.2023.3234815

    Article  MATH  Google Scholar 

  26. Liu H, Huang Z, Mo X et al (2024) Augmenting reinforcement learning with transformer-based scene representation learning for decision-making of autonomous driving. IEEE Trans Intell Veh 9(3):4405–4421. https://doi.org/10.1109/TIV.2024.3372625

    Article  MATH  Google Scholar 

  27. Miranda VRF, Neto AA, Freitas GM et al (2024) Generalization in deep reinforcement learning for robotic navigation by reward shaping. IEEE Trans Ind Electron 71(6):6013–6020. https://doi.org/10.1109/TIE.2023.3290244

    Article  MATH  Google Scholar 

  28. Wu J, Zhou Y, Yang H et al (2023) Human-guided reinforcement learning with sim-to-real transfer for autonomous navigation. IEEE Trans Pattern Anal Mach Intell 45(12):14745–14759. https://doi.org/10.1109/TPAMI.2023.3314762

    Article  MATH  Google Scholar 

  29. Zhang L, Peng J, Yi W et al (2024) A state-decomposition ddpg algorithm for uav autonomous navigation in 3-d complex environments. IEEE Internet Things J 11(6):10778–10790. https://doi.org/10.1109/JIOT.2023.3327753

    Article  MATH  Google Scholar 

  30. Lim H, Oh M, Myung H (2021) Patchwork: Concentric zone-based region-wise ground segmentation with ground likelihood estimation using a 3d lidar sensor. IEEE Robot Autom Lett 6(4):6458–6465. https://doi.org/10.1109/LRA.2021.3093009

    Article  MATH  Google Scholar 

  31. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. In: Guyon I, Luxburg UV, Bengio S et al (eds) Advances in Neural Information Processing Systems, Long Beach, CA, USA

  32. Han SY, Liang T (2022) Reinforcement-learning-based vibration control for a vehicle semi-active suspension system via the ppo approach. Appl Sci 12(6):3078. https://doi.org/10.3390/app12063078

    Article  MATH  Google Scholar 

  33. Huang X, Deng H, Zhang W et al (2021) Towards multi-modal perception-based navigation: A deep reinforcement learning method. IEEE Robot Autom Lett 6(3):4986–4993. https://doi.org/10.1109/LRA.2021.3064461

    Article  MATH  Google Scholar 

  34. Raffin A, Hill A, Gleave A et al (2021) Stable-baselines3: Reliable reinforcement learning implementations. J Mach Learn Res 22(268):1–8. http://jmlr.org/papers/v22/20-1364.html

  35. Teng S, Hu X, Deng P et al (2023) Motion planning for autonomous driving: The state of the art and future perspectives. IEEE Trans Intell Veh 8(6):3692–3711. https://doi.org/10.1109/TIV.2023.3274536

    Article  MATH  Google Scholar 

  36. Dosovitskiy A, Ros G, Codevilla F et al (2017) CARLA: An open urban driving simulator. In: Proceedings of the 1st annual conference on robot learning, proceedings of machine learning research, vol 78. PMLR, pp 1–16. https://proceedings.mlr.press/v78/dosovitskiy17a.html

  37. Yang F, Cao C, Zhu H et al (2022) Far planner: Fast, attemptable route planner using dynamic visibility update. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 9–16. https://doi.org/10.1109/IROS47612.2022.9981574

  38. Tordesillas J, Lopez BT, Everett M et al (2022) Faster: Fast and safe trajectory planner for navigation in unknown environments. IEEE Trans Robot 38(2):922–938. https://doi.org/10.1109/TRO.2021.3100142

    Article  MATH  Google Scholar 

  39. Benatti S, Young A, Elmquist A et al (2022) End-to-end learning for off-road terrain navigation using the chrono open-source simulation platform. Multibody Syst Dyn 54(4):399–414. https://doi.org/10.1007/s11044-022-09816-1

    Article  MATH  Google Scholar 

  40. Wang Y, Wang J, Yang Y et al (2022) An end-to-end deep reinforcement learning model based on proximal policy optimization algorithm for autonomous driving of off-road vehicle. In: International conference on autonomous unmanned systems. Springer, pp 2692–2704. https://doi.org/10.1007/978-981-99-0479-2_248

  41. Abbaszadeh Shahri A, Chunling S, Larsson S (2024) A hybrid ensemble-based automated deep learning approach to generate 3d geo-models and uncertainty analysis. Eng Comput 40:1501–1516. https://doi.org/10.1007/s00366-023-01852-5

    Article  Google Scholar 

  42. Abbaszadeh Shahri A, Shan C, Larsson S (2022) A novel approach to uncertainty quantification in groundwater table modeling by automated predictive deep learning. Nat Resour Res 31:1351–1373. https://doi.org/10.1007/s11053-022-10051-w

    Article  Google Scholar 

  43. Gu Y, Cheng Y, Chen CLP et al (2022) Proximal policy optimization with policy feedback. IEEE Trans Syst Man Cybern Syst 52(7):4600–4610. https://doi.org/10.1109/TSMC.2021.3098451

    Article  MATH  Google Scholar 

  44. Hiraoka T, Imagawa T, Hashimoto T et al (2022) Dropout q-functions for doubly efficient reinforcement learning. In: International conference on learning representations. https://openreview.net/forum?id=xCVJMsPv3RT

  45. Bhatt A, Palenicek D, Belousov B et al (2024) Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In: The twelfth international conference on learning representations, Vienna, Austria. https://openreview.net/forum?id=PczQtTsTIX

  46. Fujimoto S, Chang WD, Smith E et al (2023) For sale: State-action representation learning for deep reinforcement learning. In: Advances in neural information processing systems, New Orleans, LA, USA, pp 61573–61624. https://proceedings.neurips.cc/paper_files/paper/2023/file/c20ac0df6c213db6d3a930fe9c7296c8-Paper-Conference.pdf

  47. Abbaszadeh Shahri A, Shan C, Larsson S et al (2024) Normalizing large scale sensor-based mwd data: An automated method toward a unified database. Sensors 24(4):1209. https://doi.org/10.3390/s24041209

  48. Yan X, Yang J, Zhu X et al (2024) Denoising framework based on multiframe continuous point clouds for autonomous driving lidar in snowy weather. IEEE Sens J 24(7):10515–10527. https://doi.org/10.1109/JSEN.2024.3358341

    Article  Google Scholar 

  49. Wu J, Huang Z, Huang W et al (2024) Prioritized experience-based reinforcement learning with human guidance for autonomous driving. IEEE Trans Neural Netw Learn Syst 35(1):855–869. https://doi.org/10.1109/TNNLS.2022.3177685

    Article  MATH  Google Scholar 

  50. Wu J, Huang Z, Hu Z et al (2023) Toward human-in-the-loop ai: Enhancing deep reinforcement learning via real-time human guidance for autonomous driving. Engineering 21:75–91. https://doi.org/10.1016/j.eng.2022.05.017

    Article  MATH  Google Scholar 

Download references

Acknowledgements

This work is co-funded by the National Science and Technology Innovation 2030 of China-New Generation Artificial Intelligence (Grant number: 2022ZD0115603), the Key R&D program of Jiangsu Province (Grant number:BE2022053-5).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xu Li.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Z., Li, X., Hu, J. et al. Mapless autonomous navigation for UGV in cluttered off-road environment with the guidance of wayshowers using deep reinforcement learning. Appl Intell 55, 254 (2025). https://doi.org/10.1007/s10489-024-06054-0

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-06054-0

Keywords