Abstract
Unmanned surface vehicles (USVs) with autonomous capabilities is the future trend. The capability of path planning is particularly critical to ensure the safety of navigation at sea. The algorithms with known environmental information are no longer suitable for the complex and changeable marine environment. Deep reinforcement learning (DRL) can be better applied to uncertain environments as it obtains optimal policies through the interaction of agents. However, the sparse reward problem of reinforcement learning is more prominent in the path planning task. Agents can not get positive reward in a great number of interactions. To study the path planning problem of USV in uncertain environments, this paper proposes a deep Q-learning (DQN) model based on adaptive fuzzy reward. To address the sparse reward problem in path planning using reinforcement learning, we use fuzzy logic that conforms to human cognition to dynamically adjust the reward for different states so as to improve the performance of DQN algorithm. Through simulation experiments, the validity of our method under different environments is verified. The results show that our model can carry out path planning safely and effectively.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ho, J., Ermon, S.: Generative adversarial imitation learning. In: Advances in Neural Information Processing Systems, vol. 29 (2016)
Hu, Y., Wang, W., Jia, H., Wang, Y., et al.: Learning to utilize shaping rewards: a new approach of reward shaping. Adv. Neural. Inf. Process. Syst. 33, 15931–15941 (2020)
Jin, C., Krishnamurthy, A., Simchowitz, M., Yu, T.: Reward-free exploration for reinforcement learning. In: International Conference on Machine Learning, pp. 4870–4879. PMLR (2020)
Lei, X., Zhang, Z., Dong, P.: Dynamic path planning of unknown environment based on deep reinforcement learning. J. Robot. 2018 (2018)
Li, L., Wu, D., Huang, Y., Yuan, Z.M.: A path planning strategy unified with a colregs collision avoidance function based on deep reinforcement learning and artificial potential field. Appl. Ocean Res. 113, 102759 (2021)
Lin, X., Guo, R.: Path planning of unmanned surface vehicle based on improved q-learning algorithm. In: 2019 3rd International Conference on Electronic Information Technology and Computer Engineering (EITCE), pp. 302–306. IEEE (2019)
Liu, Q., Shu, H., Yuan, M., Wang, G.: Fuzzy hierarchical network embedding fusing structural and neighbor information. Inf. Sci. 603, 130–148 (2022)
Mnih, V., Badia, A.P., Mirza, M., Graves, A., et al.: Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning, pp. 1928–1937. PMLR (2016)
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., et al.: Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)
Padakandla, S.: A survey of reinforcement learning algorithms for dynamically varying environments. ACM Comput. Surv. (CSUR) 54(6), 1–25 (2021)
Papoudakis, G., Chatzidimitriou, K.C., Mitkas, P.A.: Deep reinforcement learning for doom using unsupervised auxiliary tasks. arXiv preprint arXiv:1807.01960 (2018)
Pathak, D., Agrawal, P., Efros, A.A., Darrell, T.: Curiosity-driven exploration by self-supervised prediction. In: International Conference on Machine Learning, pp. 2778–2787. PMLR (2017)
Peng, Y., Yang, Y., Cui, J., Li, X., et al.: Development of the USV ‘jinghai-i’and sea trials in the southern yellow sea. Ocean Eng. 131, 186–196 (2017)
Singh, Y., Sharma, S., Sutton, R., Hatton, D., Khan, A.: A constrained a* approach towards optimal path planning for an unmanned surface vehicle in a maritime environment containing dynamic obstacles and ocean currents. Ocean Eng. 169, 187–201 (2018)
Song, C.H.: Global path planning method for USV system based on improved ant colony algorithm. In: Applied Mechanics and Materials, vol. 568, pp. 785–788. Trans Tech Publ (2014)
Van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 2094–2100 (2016)
Yan, R.J., Pang, S., Sun, H.B., Pang, Y.J.: Development and missions of unmanned surface vehicle. J. Mar. Sci. Appl. 9(4), 451–457 (2010)
Zadeh, L.A.: Fuzzy sets. In: Fuzzy sets, fuzzy logic, and fuzzy systems: selected papers by Lotfi A Zadeh, pp. 394–432. World Scientific (1996)
Zhang, W., Wang, W., Zhai, H., Li, Q.: A deep reinforcement learning method for mobile robot path planning in unknown environments. In: 2021 China Automation Congress (CAC), pp. 5898–5902. IEEE (2021)
Zhang, W., Xu, Y., Xie, J.: Path planning of USV based on improved hybrid genetic algorithm. In: 2019 European Navigation Conference (ENC), pp. 1–7. IEEE (2019)
Zhang, Z.: Improved Adam optimizer for deep neural networks. In: 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), pp. 1–2. IEEE (2018)
Acknowledgments
This work is supported by the State Key Program of National Nature Science Foundation of China (61936001), the key cooperation project of Chongqing municipal education commission (HZ2021008), the Natural Science Foundation of Chongqing (cstc2019jcyj-cxttX0002, cstc2021ycjh-bgzxm0013).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Duan, Z., Wang, G., Liu, Q., Shi, Y. (2022). USV Path Planning Based on Adaptive Fuzzy Reward. In: Yao, J., Fujita, H., Yue, X., Miao, D., Grzymala-Busse, J., Li, F. (eds) Rough Sets. IJCRS 2022. Lecture Notes in Computer Science(), vol 13633. Springer, Cham. https://doi.org/10.1007/978-3-031-21244-4_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-21244-4_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-21243-7
Online ISBN: 978-3-031-21244-4
eBook Packages: Computer ScienceComputer Science (R0)