Abstract
This article introduces a novel Deep Reinforcement Learning (DRL)-based approach for mapless navigation in Industrial Autonomous Mobile Robots, emphasizing advancements in generalization through Potential Risk State Augmentation (PRSA) and an adaptive safety optimization reward function. Traditional LiDAR-based state representations often fail to capture environmental intricacies, leading to suboptimal performance. PRSA addresses this by improving the representation of high-dimensional LiDAR data, focusing on essential risk-related information to reduce redundancy and enhance the DRL agent’s generalization across various industrial settings. The adaptive reward function integrated with intrinsic reward mitigates the issue of sparse rewards in complex tasks, promoting faster learning and optimal policy convergence. Extensive experiments demonstrate that our method maintains a high success rate (over 90%) and low collision risk in narrow and dynamic environments compared to existing DRL-based methods. Meanwhile, compared with the classic navigation baseline, the proposed method improves the success rate by about 33% and reduces the mean navigation time by about 48% in real-world navigation tasks. The direct transfer of policies trained in simulations to real-world environments has demonstrated significant potential for enhancing both the efficacy and reliability of autonomous navigation.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability Statement
Data cannot be shared openly but are available on request from authors: Data sets generated during the current study are available from the corresponding author on reasonable request.
References
Cadena C, Carlone L, Carrillo H et al (2016) Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Trans Robot 32(6):1309–1332
Yang H, Xu X, Hong J (2022) Automatic parking path planning of tracked vehicle based on improved a* and dwa algorithms. IEEE Trans Transp Electrif 9(1):283–292
Liu J, Ji J, Ren Y et al (2021) Path planning for vehicle active collision avoidance based on virtual flow field. Int J Automot Technol 22:1557–1567
Zhu K, Zhang T (2021) Deep reinforcement learning based mobile robot navigation: a review. Tsinghua Sci Technol 26(5):674–691
Tai L, Paolo G, Liu M (2017) Virtual-to-real deep reinforcement learning: continuous control of mobile robots for mapless navigation. In: 2017 IEEE/RSJ International conference on intelligent robots and systems (IROS), IEEE, pp 31–36
Shi H, Shi L, Xu M et al (2019) End-to-end navigation strategy with deep reinforcement learning for mobile robots. IEEE Trans Ind Inform 16(4):2393–2402
Wu K, Wang H, Esfahani MA et al (2021) Learn to navigate autonomously through deep reinforcement learning. IEEE Trans Ind Electron 69(5):5342–5352
Luong M, Pham C (2021) Incremental learning for autonomous navigation of mobile robots based on deep reinforcement learning. J Intell Robot Syst 101(1):1
Zhang W, Zhang Y, Liu N et al (2022) Ipaprec: A promising tool for learning high-performance mapless navigation skills with deep reinforcement learning. IEEE/ASME Trans Mechatron 27(6):5451–5461
Wang C, Wang J, Shen Y et al (2019) Autonomous navigation of uavs in large-scale complex environments: a deep reinforcement learning approach. IEEE Trans Veh Technol 68(3):2124–2136
Xie Z, Dames P (2023) Drl-vo: Learning to navigate through crowded dynamic scenes using velocity obstacles. IEEE Trans Robot
De Ryck M, Versteyhe M, Debrouwere F (2020) Automated guided vehicle systems, state-of-the-art control algorithms and techniques. J Manuf Syst 54:152–173
Sprunk C, Lau B, Pfaff P et al (2017) An accurate and efficient navigation system for omnidirectional robots in industrial environments. Auton Robots 41:473–493
Liu X, Wang W, Li X et al (2022) Mpc-based high-speed trajectory tracking for 4wis robot. ISA Trans 123:413–424
Rasekhipour Y, Khajepour A, Chen SK et al (2016) A potential field-based model predictive path-planning controller for autonomous road vehicles. IEEE Trans Intell Transp Syst 18(5):1255–1267
Yang H, Wang Z, Xia Y et al (2023) Empc with adaptive apf of obstacle avoidance and trajectory tracking for autonomous electric vehicles. ISA Trans 135:438–448
Xiao X, Liu B, Warnell G et al (2022) Motion planning and control for mobile robot navigation using machine learning: a survey. Auton Robots 46(5):569–597
Zhu Y, Mottaghi R, Kolve E et al (2017) Target-driven visual navigation in indoor scenes using deep reinforcement learning. In: 2017 IEEE international conference on robotics and automation (ICRA), IEEE, pp 3357–3364
Yokoyama K, Morioka K (2020) Autonomous mobile robot with simple navigation system based on deep reinforcement learning and a monocular camera. In: 2020 IEEE/SICE International Symposium on System Integration (SII), IEEE, pp 525–530
Zhou Z, Zhu P, Zeng Z et al (2022) Robot navigation in a crowd by integrating deep reinforcement learning and online planning. Appl Intell 52(13):15600–15616
Chen Y, Liu C, Shi BE et al (2020) Robot navigation in crowds by graph convolutional networks with attention learned from human gaze. IEEE Robot Autom Lett 5(2):2754–2761
Sun X, Zhang Q, Wei Y et al (2023) Risk-aware deep reinforcement learning for robot crowd navigation. Electronics 12(23):4744
Liu L, Dugas D, Cesari G, et al (2020) Robot navigation in crowded environments using deep reinforcement learning. In: 2020 IEEE/RSJ International conference on intelligent robots and systems (IROS), IEEE, pp 5671–5677
Pfeiffer M, Schaeuble M, Nieto J et al (2017) From perception to decision: A data-driven approach to end-to-end motion planning for autonomous ground robots. In: 2017 IEEE international conference on robotics and automation (icra), IEEE, pp 1527–1533
Francis A, Faust A, Chiang HTL et al (2020) Long-range indoor navigation with prm-rl. IEEE Trans Robot 36(4):1115–1134
Pfeiffer M, Shukla S, Turchetta M et al (2018) Reinforced imitation: sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations. IEEE Robot Autom Lett 3(4):4423–4430
Li W, Yue M, Shangguan J et al (2023) Navigation of mobile robots based on deep reinforcement learning: Reward function optimization and knowledge transfer. Int J Control Autom Syst 21(2):563–574
Guo H, Ren Z, Lai J et al (2023) Optimal navigation for agvs: a soft actor-critic-based reinforcement learning approach with composite auxiliary rewards. Eng Appl Artif Intell 124:106613
Martinez-Baselga D, Riazuelo L, Montano L (2023) Improving robot navigation in crowded environments using intrinsic rewards. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp 9428–9434
Jiang H, Esfahani MA, Wu K et al (2022) itd3-cln: Learn to navigate in dynamic scene through deep reinforcement learning. Neurocomputing 503:118–128
Jang Y, Baek J, Han S (2021) Hindsight intermediate targets for mapless navigation with deep reinforcement learning. IEEE Trans Ind Electron 69(11):11816–11825
Zhu W, Hayashibe M (2022) A hierarchical deep reinforcement learning framework with high efficiency and generalization for fast and safe navigation. IEEE Trans Ind Electron 70(5):4962–4971
Miranda VR, Neto AA, Freitas GM, et al (2023) Generalization in deep reinforcement learning for robotic navigation by reward shaping. IEEE Trans Ind Electron
Yan C, Qin J, Liu Q et al (2022) Mapless navigation with safety-enhanced imitation learning. IEEE Trans Ind Electron 70(7):7073–7081
Chang L, Shan L, Zhang W et al (2023) Hierarchical multi-robot navigation and formation in unknown environments via deep reinforcement learning and distributed optimization. Robot Comput-Integr Manuf 83:102570
Lim J, Ha S, Choi J (2020) Prediction of reward functions for deep reinforcement learning via gaussian process regression. IEEE/ASME Trans Mechatron 25(4):1739–1746. https://doi.org/10.1109/TMECH.2020.2993564
Zhang W, Liu N, Zhang Y (2021) Learn to navigate maplessly with varied lidar configurations: a support point-based approach. IEEE Robot Autom Lett 6(2):1918–1925. https://doi.org/10.1109/LRA.2021.3061305
Haarnoja T, Zhou A, Abbeel P, et al (2018) Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International conference on machine learning, PMLR, pp 1861–1870
Yang J, Lu S, Han M et al (2023) Mapless navigation for uavs via reinforcement learning from demonstrations. Sci China Technol Sci 66(5):1263–1270
Huang W, Zhou Y, He X, et al (2023) Goal-guided transformer-enabled reinforcement learning for efficient autonomous navigation. IEEE Trans Intell Transp Syst
Gao X, Yan L, Li Z et al (2023) Improved deep deterministic policy gradient for dynamic obstacle avoidance of mobile robot. IEEE Trans Syst, Man, Cybern Syst 53(6):3675–3682
Pathak D, Agrawal P, Efros AA et al (2017) Curiosity-driven exploration by self-supervised prediction. In: International conference on machine learning, PMLR, pp 2778–2787
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Xu, D., Chen, P., Zhou, X. et al. Deep reinforcement learning based mapless navigation for industrial AMRs: advancements in generalization via potential risk state augmentation. Appl Intell 54, 9295–9312 (2024). https://doi.org/10.1007/s10489-024-05679-5
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-024-05679-5