Abstract
Recently, artificial intelligence algorithms represented by reinforcement learning and deep learning have promoted the development of autonomous driving technology. For the shipping industry, research and development of maritime autonomous surface ships (MASS) has academic value and practical significance. In an unknown environment, MASS interacts with the environment to conduct behavioral decisions-making, intelligent collision avoidance, and path planning. Reinforcement learning balances exploration and exploitation to improve its own behavior by interacting with the environment to obtain rewarded data. Thus, to achieve intelligent collision avoidance and path planning for MASS in unknown environments, a path planning algorithm of MASS based on reinforcement learning is established. Firstly, the research status of unmanned ships and reinforcement learning is reviewed. The four basic elements of reinforcement learning are analyzed: environment model, incentive function, value function and strategy. Secondly, the port environment model, sensor model, MASS behavioral space, reward function, and action selection strategy were designed separately. Besides, the reward function consists of avoiding obstacles and approaching the target point. Finally, based on the python and pygame platform, a simulation experiment was carried out with Rizhao Harbor District as a case study to verify that this method has better self-adaptability. The model successfully avoids obstacles through online trial and error self-learning and plans adaptive paths in unknown environments.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, C., Geng, P., Zhang, X.: Path planning research on unmanned surface vessel based on improved potential field. Ship Eng. 37(9), 72–75 (2015)
Guo, Y., Wang, X.: UAV path planning based on improved quantum-behaved particle swarm optimization algorithm. Ship Eng. 45(1), 99–112 (2016)
Yan, M., Huang, B., Zhu, D.: A novel path planning algorithm based on neurodynamics for observation of underwater structures. Ship Ocean Eng. 46(2), 103–107, 112 (2017)
Chen, H.: Preliminary research on local path planning for unmanned surface vehicle. Doctoral dissertation of Dalian Maritime University (2016)
Duan, Q., Zhang, M., Zhang, J.: Underwater robot local path planning method based on fuzzy neural network. Ship Eng. 1, 54–58 (2001)
Liang, Q.: Reinforcement learning based mobile robot path planning in unknown environment. Mech. Electr. Eng. Mag. 29(4), 477–481 (2012)
Zhao, Y., Zheng, Z., Zhang, X., et al.: Q learning algorithm-based UAV path learning and obstacle avoidance approach. In: 36th Chinese Control Conference (CCC), Dalian, pp. 3397–3402. IEEE CPP (2017)
Konar, A., Chakraborty, I.G., Singh, S.J., et al.: A deterministic improved Q-learning for path planning of a mobile robot. IEEE Trans. Syst. Man Cybern. Syst. 43(5), 1141–1153 (2013)
Hwang, K.S., Jiang, W.C., Chen, Y.J.: Pheromone-based planning strategies in Dyna-Q learning. IEEE Trans. Industr. Inf. 13(2), 424–435 (2017)
Cruz, D.L., Yu, W.: Path planning of multi-agent systems in unknown environment with neural kernel smoothing and reinforcement learning. Neurocomputing 233, 34–42 (2017)
Dong, W.: Research on mobile robot piath planning based on Q-learning. Doctoral dissertation of Shandong University of Science and Technology (2013)
Liu, Q., Zhai, J., Zhang, Z., et al.: A survey on deep reinforcement learning. Chin. J. Comput. 41(1), 1–27 (2018)
Zhao, D., Shao, K., Zhu, Y., et al.: Review of deep reinforcement learning and discussions on the development of computer Go. Control Theory Appl. 33(6), 701–717 (2016)
Huang, B., Cao, G., Wang, Z.: Reinforcement learning theory, algorithms and application. J. Hebei Univ. Technol. 35(6), 34–38 (2006)
Szepesvari, C.: Algorithms for reinforcement learning. In: International Conference on Computing, pp. 103–127. Morgan & Claypool, San Rafael (2010)
Cheng, Y., Zhang, W.: Concise deep reinforcement learning obstacle avoidance for underactuated unmanned marine vessels. Neurocomputing 272, 63–73 (2017)
Acknowledgements
This work was supported by the National Natural Science Foundation of China (Grant 51779028), and the Fundamental Research Funds for the Central Universities (Grant 3132018146, 3132016315 and 3132016321).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wang, C., Zhang, X., Li, R., Dong, P. (2019). Path Planning of Maritime Autonomous Surface Ships in Unknown Environment with Reinforcement Learning. In: Sun, F., Liu, H., Hu, D. (eds) Cognitive Systems and Signal Processing. ICCSIP 2018. Communications in Computer and Information Science, vol 1006. Springer, Singapore. https://doi.org/10.1007/978-981-13-7986-4_12
Download citation
DOI: https://doi.org/10.1007/978-981-13-7986-4_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-7985-7
Online ISBN: 978-981-13-7986-4
eBook Packages: Computer ScienceComputer Science (R0)