Skip to main content

Advertisement

SpringerLink
Log in

An Efficient Deep Reinforcement Learning Algorithm for Mapless Navigation with Gap-Guided Switching Strategy

  • Short Paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Deep reinforcement learning (DRL) has recently received a lot of attention due to its better performance compared to traditional algorithms in mapless navigation tasks. However, random exploration and sparse rewards may lead to inefficient DRL training in complex scenarios. In this paper, we first propose a gap-guided controller switching strategy to significantly improve the training efficiency of DRL by introducing human-designed priori knowledge. Specifically, we design a reward-based controller switching algorithm that allows the priori controller (expert) to correct the robot behavior in case of poor DRL controller performance, which can avoid random exploration during training. In addition, a gap detection algorithm is used as an online-mapless planner to compute a passable sub-goal within a limited field of view, ensuring that the robot receives dense rewards in complex scenarios. In summary, our method incorporates prior knowledge into network training by combining online-mapless planner and expert demonstration, which greatly improves training efficiency. We apply our method to mapless navigation tasks in both simulation and real worlds to demonstrate that the proposed method improves exploration efficiency by 62%, total reward by 42%, and training speed by 60% compared to the normal DRL methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data Availability

The demo video of our proposed mapless navigation scheme is available in https://www.youtube.com/playlist?list=PLxpfeFCAyx62pI58wV9sDCWBTtK9xgUSq.

References

  1. DeSouza, G.N., Kak, A.C.: Vision for mobile robot navigation: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 24(2), 237–267 (2002)

    Article  Google Scholar 

  2. Zhang, Z., Qin, J., Wang, S., Kang, Y., Liu, C.: Ulodnet: a unified lane and obstacle detection network towards drivable area understanding in autonomous navigation. J. Intell. Robot. Syst. 105(1), 1–11 (2022)

    Article  Google Scholar 

  3. Chang, X., Pan, H., Sun, W., Gao, H.: Yoltrack: multitask learning based real-time multiobject tracking and segmentation for autonomous vehicles. IEEE Trans. Neural Netw. Learn. Syst. 32(12), 5323–5333 (2022)

    Article  Google Scholar 

  4. Zhu, Y., Mottaghi, R., Kolve, E., Lim, J.J., Gupta, A., Fei-Fei, L., Farhadi, A.: Target-driven visual navigation in indoor scenes using deep reinforcement learning. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 3357–3364 (2017)

  5. Zhang, J., Springenberg, J.T., Boedecker, J., Burgard, W.: Deep reinforcement learning with successor features for navigation across similar environments. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2371–2378 (2017)

  6. Lillicrap, T., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning. arXiv:1509.02971 (2015)

  7. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv:1707.06347 (2017)

  8. Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al.: Soft actor-critic algorithms and applications. arXiv:1812.05905 (2018)

  9. Niroui, F., Zhang, K., Kashino, Z., Nejat, G.: Deep reinforcement learning robot for search and rescue applications: exploration in unknown cluttered environments. IEEE Robot. Autom. Lett. 4(2), 610–617 (2019)

    Article  Google Scholar 

  10. Li, H., Zhang, Q., Zhao, D.: Deep reinforcement learning-based automatic exploration for navigation in unknown environment. IEEE Trans. Neural Netw. Learn. Syst. 31(6), 2064–2076 (2019)

    Article  Google Scholar 

  11. Botteghi, N., Sirmacek, B., Mustafa, K., Poel, M., Stramigioli, S.: On reward shaping for mobile robot navigation: a reinforcement learning and slam based approach. arXiv:2002.04109 (2020)

  12. Tai, L., Liu, M.: Mobile robots exploration through cnn-based reinforcement learning. Robot. Biomimetics 3(1), 1–8 (2016)

    MathSciNet  Google Scholar 

  13. Kato, Y., Kamiyama, K., Morioka, K.: Autonomous robot navigation system with learning based on deep q-network and topological maps. In: 2017 IEEE/SICE International Symposium on System Integration (SII), pp. 1040–1046 (2017)

  14. Faust, A., Oslund, K., Ramirez, O., Francis, A., Tapia, L., Fiser, M., Davidson, J.: Prm-rl: long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5113–5120 (2018)

  15. Chen, G., Pan, L., Xu, P., et al.: Robot navigation with map-based deep reinforcement learning. In: 2020 IEEE International Conference on Networking, Sensing and Control (ICNSC), pp. 1-6 (2020)

  16. Argall, B., Chernova, S., Veloso, M., Browning, B.: A survey of robot learning from demonstration. Robot. Auton. Syst. 57(5), 469–483 (2009)

    Article  Google Scholar 

  17. Wu, Q., Gong, X., Xu, K., Manocha, D., Dong, J., Wang, J.: Towards target-driven visual navigation in indoor scenes via generative imitation learning. IEEE Robot. Autom. Lett. 6(1), 175–182 (2021)

    Article  Google Scholar 

  18. Rana, K., Dasagi, V., Talbot, B., Milford, M., Sünderhauf, N.: Multiplicative controller fusion: leveraging algorithmic priors for sample-efficient reinforcement learning and safe sim-to-real transfer. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6069–6076 (2020)

  19. Bellegarda, G., Byl, K.: An online training method for augmenting mpc with deep reinforcement learning. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5453–5459 (2020)

  20. Khatib, O.: Real-time obstacle avoidance for manipulators and mobile robots. In: 1985 IEEE International Conference on Robotics and Automation (ICRA), pp. 500–505 (1985)

  21. Fox, D., Burgard, W., Thrun, S.: The dynamic window approach to collision avoidance. IEEE Robot. Autom. Mag. 4(1), 23-33 (1997)

    Article  Google Scholar 

  22. Ballesteros, J., Urdiales, C., Velasco, A., Ramos-Jiménez, G.: A biomimetical dynamic window approach to navigation for collaborative control. IEEE Trans. Hum.-Mach. Syst. vol. 47, no. 6, pp. 1123–1133, 2017

  23. Khatib, O.: Real-time obstacle avoidance for manipulators and mobile robots. In: Autonomous Robot Vehicles, pp. 396–404 (1986)

  24. Gao, Z., Qin, J., Wang, S., Wang, Y.: Boundary gap based reactive navigation in unknown environments. IEEE/CAA J. Autom. Sinica 8(2), 468–477 (2021)

    Article  Google Scholar 

  25. Qiu, J., Ma, M., Wang, T., Gao, H.: Gradient descent-based adaptive learning control for autonomous underwater vehicles with unknown uncertainties. IEEE Trans. Neural Netw. Learn. Syst. 32(12), 5266–5273 (2021)

    Article  MathSciNet  Google Scholar 

  26. Li, Y., Qin, F., Du, S., Xu, D., Zhang, J.: Vision-based imitation learning of needle reaching skill for robotic precision manipulation. J. Intell. Robot. Syst. 101(1), 1–13 (2021)

    Article  Google Scholar 

  27. Yan, C., Qin, J., Liu, Q., Ma, Q., Kang, Y.: Mapless navigation with safety-enhanced imitation learning. IEEE Trans. Ind. Electron. in press (2022)

  28. Yan, C., Xiang, X., Wang, C.: Towards real-time path planning through deep reinforcement learning for a uav in dynamic environments. J. Intell. Robot. Syst. 98(2), 297–309 (2020)

    Article  Google Scholar 

  29. Mujahad, M., Fischer, D., Mertsching, B., Jaddu, H.: Closest gap based (cg) reactive obstacle avoidance navigation for highly cluttered environments. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1805–1812 (2010)

  30. Thrun, S.: Probabilistic robotics. Commun. ACM 45(3), 52–57 (2002)

    Article  Google Scholar 

  31. Grisetti, G., Stachniss, C., Burgard, W.: Improving grid-based slam with rao-blackwellized particle filters by adaptive proposals and selective resampling. In: 2005 IEEE International Conference on Robotics and Automation (ICRA), pp. 2432–2437 (2005)

Download references

Acknowledgements

We acknowledge the support of the GPU cluster built by MCCLab of Information Science and Technology Institution, USTC.

Funding

This work is supported by the Science and Technology Major Project of Anhui Province (Grant No. 202203a06020011), and by the Mobility Programme 2021 of Sino-German Center for Research Promotion (No. M-0582), and by the Open Fund of the Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences (Grant No. CASIA-KFKT-14). Part of the work of Q. Liu was carried out when he was a research fellow at the Chair of Information-Oriented Control, Technical University of Germany, Germany. (Corresponding Author: Jiahu Qin)

Author information

Authors and Affiliations

  1. Department of Automation, University of Science and Technology of China, Hefei, 230027, China

    Heng Li, Jiahu Qin, Qingchen Liu & Chengzhen Yan

  2. Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230088, China

    Jiahu Qin

Authors
  1. Heng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiahu Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qingchen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chengzhen Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Heng Li: Coding and writing; Jiahu Qin: Writing and review; Qingchen Liu: Writing and review; Chengzhen Yan: Review.

Corresponding author

Correspondence to Jiahu Qin.

Ethics declarations

Ethical Approval

Not applicable.

Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Conflicts of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Qin, J., Liu, Q. et al. An Efficient Deep Reinforcement Learning Algorithm for Mapless Navigation with Gap-Guided Switching Strategy. J Intell Robot Syst 108, 43 (2023). https://doi.org/10.1007/s10846-023-01888-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-023-01888-1

Keywords

Search

Navigation