Abstract
Unmanned surface vessels (USVs) have great significance and wide applications in many fields, whereas the control law designed with the analytical approach is too complicated to implement, subject to the level of hardware development. Confronted with obstacles, USVs use the conventional method to avoid them, but in many practical cases, it is difficult to devise the path in advance. Moreover, prior knowledge, including expert experience, may be challenging to introduce into a control system effectively. In this paper, a fuzzy categorical deep reinforcement learning-based framework is established to handle a sophisticated obstruction situation. The framework consists of an interactive observation module and a control module with fuzzy reward shaping. Experimental results verify that the performance of the USV with the framework is better than that of the USV using the path-following method. In addition, it is not necessary to arrange the path of the USV beforehand; the path is autonomously steered to the destination instead. With the benefit of the simple control law, the architecture is available for various levels of hardware.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Manley, J.E.: Unmanned surface vehicles, 15 years of development. In: OCEANS. IEEE, pp. 1–4 (2008)
Bertram, V.: Unmanned surface vehicles-a survey. Skibsteknisk Selsk. Cph. Den. 1, 1–14 (2008)
Villa, J L., Paez, J., Quintero, C., et al.: Design and control of an unmanned surface vehicle for environmental monitoring applications. In: IEEE Colombian conference on robotics and automation (CCRA). IEEE, pp. 1–5 (2016)
Wang, N., Sun, J.C., Er, M.J., et al.: A novel extreme learning control framework of unmanned surface vehicles. IEEE Trans. Cybern. 46(5), 1106–1117 (2016)
Smierzchalski, R., Michalewicz, Z.: Path planning in dynamic environments. Innov. Robot Mobil. Control 8, 135–153 (2005)
Peng, Z., Wang, D., Li, T., et al.: Leaderless and leader-follower cooperative control of multiple marine surface vehicles with unknown dynamics. Nonlinear Dyn. 74(1–2), 95–106 (2013)
Zhang, G., Zhang, X.: A novel DVS guidance principle and robust adaptive path-following control for underactuated ships using low frequency gain-learning. ISA Trans. 56, 75–85 (2015)
Zhang, G., Zhang, X.: Concise robust adaptive path-following control of underactuated ships using DSC and MLP. IEEE J. Ocean. Eng. 8(4), 685–694 (2014)
Fossen, T.I.: Handbook of Marine Craft Hydrodynamics and Motion Control. Wiley, New York (2011)
Perez, T.: Ship Motion Control: Course Keeping and Roll Stabilisation Using Rudder and Fins. Springer, Berlin (2006)
Vamvoudakis, K.G., Modares, H., Kiumarsi, B., et al.: Game theory-based control system algorithms with real-time reinforcement learning: how to solve multiplayer games online. IEEE Control Syst. 37(1), 33–52 (2017)
Mendelson, E.: Introducing Game Theory and Its Applications. CRC Press, Boca Raton (2016)
Wei, Q., Song, R., Yan, P.: Data-driven zero-sum neuro-optimal control for a class of continuous-time unknown nonlinear systems with disturbance using ADP. IEEE Trans. Neural Netw. Learn. Syst. 27(2), 444–458 (2016)
Harsanyi, J.C.: Games with incomplete information played by Bayesian players: part I. The basic model. Manag. Sci 50_supplement(12), 1804–1817 (2004)
Maschler, Michael, Solan, Eilon, Zamir, Shmuel: Game Theory. Cambridge University Press, Cambridge (2013)
Mei, S., Wang, Y., Sun, Z.: Robust economic dispatch considering renewable generation. In: 2011 IEEE PES Innovative Smart Grid Technologies Asia (ISGT). IEEE, pp. 1–5 (2011)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT press, Cambridge (1998)
Statistical, Sugiyama M.: Reinforcement Learning: Modern Machine Learning Approaches. CRC Press, Boca Raton (2015)
Zhao, D., Zhu, Y.M.E.C.: A near-optimal online reinforcement learning algorithm for continuous deterministic systems. IEEE Trans. Neural Netw. Learn. Syst. 26(2), 346–356 (2015)
Deng, Y., Bao, F., Kong, Y., et al.: Deep direct reinforcement learning for financial signal representation and trading. IEEE Trans. Neural Netw. Learn. Syst. 28(3), 653–664 (2017)
Bellman, R.: A Markovian Decision Process. RAND Corp, Santa Monica (1957)
Li, Y.: Deep reinforcement learning: an overview. arXiv preprint arXiv:1701.07274 (2017)
Mnih, V., Badia, A P., Mirza, M., et al.: Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning, pp. 1928–1937 (2016)
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M.: Deterministic policy gradient algorithms. In: The International Conference on Machine Learning (ICML) (2014)
Watkins, C.J.C.H., Dayan, P.: Q-learning. Mach. Learn. 8(3–4), 279–292 (1992)
Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Krizhevsky, A., Sutskever, I., Hinton, G E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F. (ed.) Advances in Neural Information Processing Systems, pp. 1097–1105. Curran Associates, New York, NY (2012)
Bellemare, M. G., Dabney, W., Munos, R.: A distributional perspective on reinforcement learning. In: International Conference on Machine Learning, pp. 449–458 (2017)
Caspi, I., Leibovich, G., Novik, G., Endrawis, S.: Reinforcement Learning Coach (2017). https://doi.org/10.5281/zenodo.1134899
Acknowledgements
This paper is partly supported by the National Science Foundation of China (61473183, 61521063, U1509211).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Cheng, Y., Sun, Z., Huang, Y. et al. Fuzzy Categorical Deep Reinforcement Learning of a Defensive Game for an Unmanned Surface Vessel. Int. J. Fuzzy Syst. 21, 592–606 (2019). https://doi.org/10.1007/s40815-018-0586-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40815-018-0586-0