Abstract
Navigating unmanned ground vehicle (UGV) through off-road environments is critical for various tasks like exploration and rescue. Unlike scenarios allowing offline global planning based on prior knowledge, online navigation becomes essential due to the dynamic nature of these tasks. Although deep reinforcement learning (DRL) offers promise for mapless autonomous navigation due to its end-to-end advantages, existing approaches often rely solely on goal positions. This neglects the complex distribution of obstacles along the path, leading to inefficient interactions with the environment during training. To address this challenge, a deep reinforcement learning framework is proposed for autonomous navigation guided by wayshowers. Initially, a new metric is developed based on multilevel analysis to generate elevation maps, aiding in the identification of optimal wayshowers. Upon integrating wayshower information with other inputs, a multi-head attention (MHA) module is incorporated into DRL network, which includes a length attention mechanism to enhance focus on recent historical observation sequences to promote model convergence. Furthermore, the reward function is reshaped to offer dense reward signals, thereby resolving the sparse reward problem inherent in goal-driven methods. To validate the proposed approach, experiments are conducted on several off-road maps in both the Carla and Gazebo simulators. The results demonstrate the superiority of our method not only in simple environments but also in more challenging scenarios.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability
Data sharing is not applicable to this article as no new data were created or analyzed in this study.
References
Kiran BR, Sobh I, Talpaert V et al (2021) Deep reinforcement learning for autonomous driving: A survey. IEEE Trans Intell Transp Syst 23(6):4909–4926. https://doi.org/10.1109/TITS.2021.3054625
Thoresen M, Nielsen NH, Mathiassen K et al (2021) Path planning for ugvs based on traversability hybrid a. IEEE Robot Autom Lett 6(2):1216–1223. https://doi.org/10.1109/LRA.2021.3056028
Qi Y, He B, Wang R et al (2022) Hierarchical motion planning for autonomous vehicles in unstructured dynamic environments. IEEE Robot Autom Lett 8(2):496–503. https://doi.org/10.1109/LRA.2022.3228159
Ye L, Wu F, Zou X et al (2023) Path planning for mobile robots in unstructured orchard environments: An improved kinematically constrained bi-directional rrt approach. Comput Electron Agric 215:108453. https://doi.org/10.1016/j.compag.2023.108453
Roriz R, Cabral J, Gomes T (2022) Automotive lidar technology: A survey. IEEE Trans Intell Transp Syst 23(7):6282–6297. https://doi.org/10.1109/TITS.2021.3086804
Kong D, Li X, Xu Q et al (2024) Sc_lpr: Semantically consistent lidar place recognition based on chained cascade network in long-term dynamic environments. IEEE Trans Image Process 33(1):2145–2157. https://doi.org/10.1109/TIP.2024.3364511
Chib PS, Singh P (2023) Recent advancements in end-to-end autonomous driving using deep learning: A survey. IEEE Trans Intell Veh 9(1):103–118. https://doi.org/10.1109/TIV.2023.3318070
Li F, Guo C, Luo B et al (2021) Multi goals and multi scenes visual mapless navigation in indoor using meta-learning and scene priors. Neurocomputing 449:368–377. https://doi.org/10.1016/j.neucom.2021.03.084
Jiang H, Esfahani MA, Wu K et al (2022) itd3-cln: Learn to navigate in dynamic scene through deep reinforcement learning. Neurocomputing 503:118–128. https://doi.org/10.1016/j.neucom.2022.06.102
Staroverov A, Yudin DA, Belkin I et al (2020) Real-time object navigation with deep neural networks and hierarchical reinforcement learning. IEEE Access 8:195608–195621. https://doi.org/10.1109/ACCESS.2020.3034524
Li Z, Zhou A (2023) Rddrl: a recurrent deduction deep reinforcement learning model for multimodal vision-robot navigation. Appl Intell 53(20):23244–23270. https://doi.org/10.1007/s10489-023-04754-7
Rais MS, Boudour R, Zouaidia K et al (2023) Decision making for autonomous vehicles in highway scenarios using harmonic sk deep sarsa. Appl Intell 53(3):2488–2505. https://doi.org/10.1007/s10489-022-03357-y
Hu J, Li X, Hu W et al (2024) A cooperative control methodology considering dynamic interaction for multiple connected and automated vehicles in the merging zone. IEEE Trans Intell Transp Syst 25(9):12669–12681. https://doi.org/10.1109/TITS.2024.3386200
Hu J, Li X, Cen Y et al (2022) A roadside decision-making methodology based on deep reinforcement learning to simultaneously improve the safety and efficiency of merging zone. IEEE Trans Intell Transp Syst 23(10):18620–18631. https://doi.org/10.1109/TITS.2022.3157910
Kahn G, Abbeel P, Levine S (2021) Badgr: An autonomous self-supervised learning-based navigation system. IEEE Robot Autom Lett 6(2):1312–1319. https://doi.org/10.1109/LRA.2021.3057023
Hu H, Zhang K, Tan AH et al (2021) A sim-to-real pipeline for deep reinforcement learning for autonomous robot navigation in cluttered rough terrain. IEEE Robot Autom Lett 6(4):6569–6576. https://doi.org/10.1109/LRA.2021.3093551
Josef S, Degani A (2020) Deep reinforcement learning for safe local planning of a ground vehicle in unknown rough terrain. IEEE Robot Autom Lett 5(4):6748–6755. https://doi.org/10.1109/LRA.2020.3011912
Zhu W, Hayashibe M (2023) A hierarchical deep reinforcement learning framework with high efficiency and generalization for fast and safe navigation. IEEE Trans Ind Electron 70(5):4962–4971. https://doi.org/10.1109/TIE.2022.3190850
Zhang K, Niroui F, Ficocelli M et al (2018) Robot navigation of environments with unknown rough terrain using deep reinforcement learning. In: 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), IEEE, pp 1–7. https://doi.org/10.1109/SSRR.2018.8468643
Cimurs R, Suh IH, Lee JH (2022) Goal-driven autonomous exploration through deep reinforcement learning. IEEE Robot Autom Lett 7(2):730–737. https://doi.org/10.1109/LRA.2021.3133591
Weerakoon K, Sathyamoorthy AJ, Patel U et al (2022) Terp: Reliable planning in uneven outdoor environments using deep reinforcement learning. In: 2022 International Conference on Robotics and Automation (ICRA), pp 9447–9453. https://doi.org/10.1109/ICRA46639.2022.9812238
Samsani SS, Mutahira H, Muhammad MS (2022) Memory-based crowd-aware robot navigation using deep reinforcement learning. Complex Intell Syst 9:2147–2158. https://doi.org/10.1007/s40747-022-00906-3
Huang W, Zhou Y, He X et al (2024) Goal-guided transformer-enabled reinforcement learning for efficient autonomous navigation. IEEE Trans Intell Transp Syst 25(2):1832–1845. https://doi.org/10.1109/TITS.2023.3312453
Zhou K, Guo C, Zhang H et al (2023) Optimal graph transformer viterbi knowledge inference network for more successful visual navigation. Adv Eng Inf 55:101889. https://doi.org/10.1016/j.aei.2023.101889
Yang Y, Jiang J, Zhang J et al (2023) St\(^{2}\): Spatial-temporal state transformer for crowd-aware autonomous navigation. IEEE Robot Autom Lett 8(2):912–919. https://doi.org/10.1109/LRA.2023.3234815
Liu H, Huang Z, Mo X et al (2024) Augmenting reinforcement learning with transformer-based scene representation learning for decision-making of autonomous driving. IEEE Trans Intell Veh 9(3):4405–4421. https://doi.org/10.1109/TIV.2024.3372625
Miranda VRF, Neto AA, Freitas GM et al (2024) Generalization in deep reinforcement learning for robotic navigation by reward shaping. IEEE Trans Ind Electron 71(6):6013–6020. https://doi.org/10.1109/TIE.2023.3290244
Wu J, Zhou Y, Yang H et al (2023) Human-guided reinforcement learning with sim-to-real transfer for autonomous navigation. IEEE Trans Pattern Anal Mach Intell 45(12):14745–14759. https://doi.org/10.1109/TPAMI.2023.3314762
Zhang L, Peng J, Yi W et al (2024) A state-decomposition ddpg algorithm for uav autonomous navigation in 3-d complex environments. IEEE Internet Things J 11(6):10778–10790. https://doi.org/10.1109/JIOT.2023.3327753
Lim H, Oh M, Myung H (2021) Patchwork: Concentric zone-based region-wise ground segmentation with ground likelihood estimation using a 3d lidar sensor. IEEE Robot Autom Lett 6(4):6458–6465. https://doi.org/10.1109/LRA.2021.3093009
Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. In: Guyon I, Luxburg UV, Bengio S et al (eds) Advances in Neural Information Processing Systems, Long Beach, CA, USA
Han SY, Liang T (2022) Reinforcement-learning-based vibration control for a vehicle semi-active suspension system via the ppo approach. Appl Sci 12(6):3078. https://doi.org/10.3390/app12063078
Huang X, Deng H, Zhang W et al (2021) Towards multi-modal perception-based navigation: A deep reinforcement learning method. IEEE Robot Autom Lett 6(3):4986–4993. https://doi.org/10.1109/LRA.2021.3064461
Raffin A, Hill A, Gleave A et al (2021) Stable-baselines3: Reliable reinforcement learning implementations. J Mach Learn Res 22(268):1–8. http://jmlr.org/papers/v22/20-1364.html
Teng S, Hu X, Deng P et al (2023) Motion planning for autonomous driving: The state of the art and future perspectives. IEEE Trans Intell Veh 8(6):3692–3711. https://doi.org/10.1109/TIV.2023.3274536
Dosovitskiy A, Ros G, Codevilla F et al (2017) CARLA: An open urban driving simulator. In: Proceedings of the 1st annual conference on robot learning, proceedings of machine learning research, vol 78. PMLR, pp 1–16. https://proceedings.mlr.press/v78/dosovitskiy17a.html
Yang F, Cao C, Zhu H et al (2022) Far planner: Fast, attemptable route planner using dynamic visibility update. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 9–16. https://doi.org/10.1109/IROS47612.2022.9981574
Tordesillas J, Lopez BT, Everett M et al (2022) Faster: Fast and safe trajectory planner for navigation in unknown environments. IEEE Trans Robot 38(2):922–938. https://doi.org/10.1109/TRO.2021.3100142
Benatti S, Young A, Elmquist A et al (2022) End-to-end learning for off-road terrain navigation using the chrono open-source simulation platform. Multibody Syst Dyn 54(4):399–414. https://doi.org/10.1007/s11044-022-09816-1
Wang Y, Wang J, Yang Y et al (2022) An end-to-end deep reinforcement learning model based on proximal policy optimization algorithm for autonomous driving of off-road vehicle. In: International conference on autonomous unmanned systems. Springer, pp 2692–2704. https://doi.org/10.1007/978-981-99-0479-2_248
Abbaszadeh Shahri A, Chunling S, Larsson S (2024) A hybrid ensemble-based automated deep learning approach to generate 3d geo-models and uncertainty analysis. Eng Comput 40:1501–1516. https://doi.org/10.1007/s00366-023-01852-5
Abbaszadeh Shahri A, Shan C, Larsson S (2022) A novel approach to uncertainty quantification in groundwater table modeling by automated predictive deep learning. Nat Resour Res 31:1351–1373. https://doi.org/10.1007/s11053-022-10051-w
Gu Y, Cheng Y, Chen CLP et al (2022) Proximal policy optimization with policy feedback. IEEE Trans Syst Man Cybern Syst 52(7):4600–4610. https://doi.org/10.1109/TSMC.2021.3098451
Hiraoka T, Imagawa T, Hashimoto T et al (2022) Dropout q-functions for doubly efficient reinforcement learning. In: International conference on learning representations. https://openreview.net/forum?id=xCVJMsPv3RT
Bhatt A, Palenicek D, Belousov B et al (2024) Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In: The twelfth international conference on learning representations, Vienna, Austria. https://openreview.net/forum?id=PczQtTsTIX
Fujimoto S, Chang WD, Smith E et al (2023) For sale: State-action representation learning for deep reinforcement learning. In: Advances in neural information processing systems, New Orleans, LA, USA, pp 61573–61624. https://proceedings.neurips.cc/paper_files/paper/2023/file/c20ac0df6c213db6d3a930fe9c7296c8-Paper-Conference.pdf
Abbaszadeh Shahri A, Shan C, Larsson S et al (2024) Normalizing large scale sensor-based mwd data: An automated method toward a unified database. Sensors 24(4):1209. https://doi.org/10.3390/s24041209
Yan X, Yang J, Zhu X et al (2024) Denoising framework based on multiframe continuous point clouds for autonomous driving lidar in snowy weather. IEEE Sens J 24(7):10515–10527. https://doi.org/10.1109/JSEN.2024.3358341
Wu J, Huang Z, Huang W et al (2024) Prioritized experience-based reinforcement learning with human guidance for autonomous driving. IEEE Trans Neural Netw Learn Syst 35(1):855–869. https://doi.org/10.1109/TNNLS.2022.3177685
Wu J, Huang Z, Hu Z et al (2023) Toward human-in-the-loop ai: Enhancing deep reinforcement learning via real-time human guidance for autonomous driving. Engineering 21:75–91. https://doi.org/10.1016/j.eng.2022.05.017
Acknowledgements
This work is co-funded by the National Science and Technology Innovation 2030 of China-New Generation Artificial Intelligence (Grant number: 2022ZD0115603), the Key R&D program of Jiangsu Province (Grant number:BE2022053-5).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Li, Z., Li, X., Hu, J. et al. Mapless autonomous navigation for UGV in cluttered off-road environment with the guidance of wayshowers using deep reinforcement learning. Appl Intell 55, 254 (2025). https://doi.org/10.1007/s10489-024-06054-0
Accepted:
Published:
DOI: https://doi.org/10.1007/s10489-024-06054-0