Skip to main content
SpringerLink
Log in

Deep Reinforcement Learning of Map-Based Obstacle Avoidance for Mobile Robot Navigation

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Autonomous and safe navigation in complex environments without collisions is particularly important for mobile robots. In this paper, we propose an end-to-end deep reinforcement learning method for mobile robot navigation with map-based obstacle avoidance. Using the experience collected in the simulation environment, a convolutional neural network is trained to predict the proper steering operation of the robot based on its egocentric local grid maps, which can accommodate various sensors and fusion algorithms. We use dueling double DQN with prioritized experienced replay technology to update parameters of the network and integrate curriculum learning techniques to enhance its performance. The trained deep neural network is then transferred and executed on a real-world mobile robot to guide it to avoid local obstacles for long-range navigation. The qualitative and quantitative evaluations of the new approach were performed in simulations and real robot experiments. The results show that the end-to-end map-based obstacle avoidance model is easy to deploy, without any fine-tuning, robust to sensor noise, compatible with different sensors, and better than other related DRL-based models in many evaluation indicators.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Ingrand F, Ghallab M. Deliberation for autonomous robots: a survey. Artif Intell. 2017;247:10–44.

    Article  MathSciNet  Google Scholar 

  2. Minguez J, Lamiraux F, Laumond J-P. Motion planning and obstacle avoidance. In: Springer handbook of robotics. Springer, 2016;1177–202.

  3. Mohanan M, Salgoankar A. A survey of robotic motion planning in dynamic environments. Robot Auton Syst. 2018;100:171–85.

    Article  Google Scholar 

  4. Zhang W, Wei S, Teng Y, Zhang J, Wang X, Yan Z. Dynamic obstacle avoidance for unmanned underwater vehicles based on an improved velocity obstacle method. Sensors. 2017;17(12):2742.

    Article  Google Scholar 

  5. Zhou D, Wang Z, Bandyopadhyay S, Schwager M. Fast, on-line collision avoidance for dynamic vehicles using buffered Voronoi cells. IEEE Robot Autom Lett. 2017;2(2):1047–54.

    Article  Google Scholar 

  6. Rösmann C, Hoffmann F, Bertram T. Integrated online trajectory planning and optimization in distinctive topologies. Robot Auton Syst. 2017;88:142–53.

    Article  Google Scholar 

  7. Kahn G, Villaflor A, Ding B, Abbeel P, Levine S. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), 2018;1–8.

  8. Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A, et al. Mastering the game of go without human knowledge. Nature. 2017;550(7676):354–9.

    Article  Google Scholar 

  9. Vinyals O, Babuschkin I, Czarnecki WM, Mathieu M, Dudzik A, Chung J, Choi DH, Powell R, Ewalds T, Georgiev P, et al. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature. 2019;575(7782):350–4.

    Article  Google Scholar 

  10. Levine S, Pastor P, Krizhevsky A, Ibarz J, Quillen D. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Robot Res. 2018;37(4–5):421–36.

    Article  Google Scholar 

  11. Xie L, Wang S, Rosa S, Markham A, Trigoni N. Learning with training wheels: speeding up training with a simple controller for deep reinforcement learning. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), 2018;6276–83.

  12. Chen G, Cui G, Jin Z, Wu F, Chen X. Accurate intrinsic and extrinsic calibration of RGB-D cameras with GP-based depth correction. IEEE Sens J. 2018;19(7):2685–94.

    Article  Google Scholar 

  13. Chen G, Pan L, Chen Y, Xu P, Wang Z, Wu P, Ji J, Chen X. Robot navigation with map-based deep reinforcement learning. In: Proceedings of the 2020 IEEE international conference on networking, sensing and control (ICNSC), 2020;1–6.

  14. Giusti A, Guzzi J, Cireşan DC, He F-L, Rodríguez JP, Fontana F, Faessler M, Forster C, Schmidhuber J, Di Caro G, et al. A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robot Autom Lett. 2015;1(2):661–7.

    Article  Google Scholar 

  15. Gandhi D, Pinto L, Gupta A. Learning to fly by crashing. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS), 2017;3948–55.

  16. Tai L, Li S, Liu M. A deep-network solution towards model-less obstacle avoidance. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS), 2016;2759–64.

  17. Pfeiffer M, Schaeuble M, Nieto J, Siegwart R, Cadena C. From perception to decision: a data-driven approach to end-to-end motion planning for autonomous ground robots. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), 2017;1527–33.

  18. Fox D, Burgard W, Thrun S. The dynamic window approach to collision avoidance. IEEE Robot Autom Mag. 1997;4(1):23–33.

    Article  Google Scholar 

  19. Liu Y, Xu A, Chen Z. Map-based deep imitation learning for obstacle avoidance. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS), 2018;8644–9.

  20. Chen YF, Liu M, Everett M, How JP. Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), 2017;285–92.

  21. Chen YF, Everett M, Liu M, How JP. “Socially aware motion planning with deep reinforcement learning,” In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017;1343–1350.

  22. Tai L, Paolo G, Liu M. Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS), 2017;31–6.

  23. Long P, Fanl T, Liao X, Liu W, Zhang H, Pan J. Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), 2018;6252–9.

  24. Lobos-Tsunekawa K, Leiva F, Ruiz-del Solar J. Visual navigation for biped humanoid robots using deep reinforcement learning. IEEE Robot Autom Lett. 2018;3(4):3247–54.

    Article  Google Scholar 

  25. Wu K, Abolfazli Esfahani M, Yuan S, Wang H. Learn to steer through deep reinforcement learning. Sensors. 2018;18(11):3650.

    Article  Google Scholar 

  26. Zhang J, Springenberg JT, Boedecker J, Burgard W. Deep reinforcement learning with successor features for navigation across similar environments. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS), 2017;2371–8.

  27. Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double Q-learning. In: Proceedings of the thirtieth AAAI conference on artificial intelligence (AAAI), 2016.

  28. Wang Z, Schaul T, Hessel M, Hasselt H, Lanctot M, Freitas N. Dueling network architectures for deep reinforcement learning. In: Proceedings of the 33rd international conference on machine learning (ICML), 2016;1995–2003.

  29. Schaul T, Quan J, Antonoglou I, Silver D. Prioritized experience replay. In 2016.

  30. Lu DV, Hershberger D, Smart WD. Layered costmaps for context-sensitive navigation. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS), 2014;709–15.

  31. Ng AY, Harada D, Russell S. Policy invariance under reward transformations: theory and application to reward shaping. ICML. 1999;99:278–87.

    Google Scholar 

  32. Bengio Y, Louradour J, Collobert R, Weston J. Curriculum learning. In: Proceedings of the 26th annual international conference on machine learning (ICML). ACM, 2009;41–8.

  33. Elman JL. Learning and development in neural networks: the importance of starting small. Cognition. 1993;48(1):71–99.

    Article  Google Scholar 

  34. Koenig N, Howard A. Design and use paradigms for gazebo, an open-source multi-robot simulator. Proc IEEE/RSJ Int Confe Intell Robot Syst. 2004;3:2149–54.

    Google Scholar 

  35. Kingma DP, Ba J. Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.

  36. Ulrich I, Borenstein J. Vfh/sup*: local obstacle avoidance with look-ahead verification. Proc IEEE Int Conf Robot Autom. 2000;3:2505–11.

    Google Scholar 

  37. Koren Y, Borenstein J, et al. Potential field methods and their inherent limitations for mobile robot navigation. Proc IEEE Int Conf Robot Autom. 1991;2:1398–404.

    Google Scholar 

  38. Brito B, Everett M, How JP, Alonso-Mora J. Where to go next: learning a subgoal recommendation policy for navigation in dynamic environments. IEEE Robot Autom Lett. 2021;6(3):4616–23.

    Article  Google Scholar 

Download references

Funding

This research was supported by 2030 National Key AI Program of China (Grant 2018AAA0100500), National Natural Science Foundation of China (CN) (Grant 61573386) and Science and Technology Planning Project of Guangdong Province (Grant 2017B010110011).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, University of Science and Technology of China, Hefei, 230026, Anhui, People’s Republic of China

    Guangda Chen, Lifan Pan, Yu’an Chen, Zhiqiang Wang, Peichen Wu, Jianmin Ji & Xiaoping Chen

  2. School of Data Science, University of Science and Technology of China, Hefei, 230026, Anhui, People’s Republic of China

    Pei Xu

Authors
  1. Guangda Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lifan Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu’an Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhiqiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Peichen Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jianmin Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xiaoping Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianmin Ji.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is partially supported by the 2030 National Key AI Program of China 2018AAA0100500, the National Natural Science Foundation of China (No. 61573386), and Guangdong Province Science and Technology Plan Projects (No. 2017B010110011).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, G., Pan, L., Chen, Y. et al. Deep Reinforcement Learning of Map-Based Obstacle Avoidance for Mobile Robot Navigation. SN COMPUT. SCI. 2, 417 (2021). https://doi.org/10.1007/s42979-021-00817-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-021-00817-z

Keywords

Search

Navigation