Abstract
This paper revisits reinforcement learning for \(H_\infty \) control of affine nonlinear systems with partially unknown dynamics. By incorporating an impulsive momentum-based control into the conventional critic neural network, an impulsive accelerated reinforcement learning algorithm with a restart mechanism is proposed to improve the convergence speed and transient performance compared to traditional gradient descent-based techniques or continuously accelerated gradient methods. Moreover, by utilizing the quasi-periodic Lyapunov function method, sufficient condition for input-to-state stability with respect to approximation errors of the closed-loop system is established. A numerical example with comparisons is provided to illustrate the theoretical results.
This work is supported by the National Natural Science Foundation of China under Grant 62003104, the Guangxi Natural Science Foundation under Grant 2022GXNSFBA035649, the Guangxi Science and Technology Planning Project under Grant AD23026217, and the Guangxi University Natural Science and Technological Innovation Development Multiplication Plan Project under Grant 2023BZRC018.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Yang, X., Xu, M., Wei, Q.: Adaptive dynamic programming for nonlinear-constrained \(H_\infty \) control. IEEE Trans. Syst. Man Cybern. Syst. 53(7), 4393–4403 (2023)
Luo, B., Wu, H.N., Huang, T.: Off-policy reinforcement learning for \(H_\infty \) control design. IEEE Trans. Cybern. 45(1), 65–76 (2014)
Wang, D., Mu, C.X., Liu, D.R., Ma, H.W.: On mixed data and event driven design for adaptive-critic-based nonlinear \( H_\infty \) control. IEEE Trans. Neural Networks Learn. Syst. 29(4), 993–1005 (2017)
Kiumarsi, B., Vamvoudakis, K.G., Modares, H., Lewis, F.L.: Optimal and autonomous control using reinforcement learning: a survey. IEEE Trans. Neural Networks Learn. Syst. 29(6), 2042–2062 (2017)
Başar, T., Bernhard, P.: \(H_\infty \) Optimal Control and Related Minimax Design Problems: A Dynamic Game Approach. Springer, Boston (2008). https://doi.org/10.1007/978-0-8176-4757-5
Liang, X., Liu, Y., Chen, T., Liu, M., Yang, Q.: Federated transfer reinforcement learning for autonomous driving. In: IFederated and Transfer Learning, pp. 357–371 (2022)
Stanly, J.J., Priyadarsini, M.J.P., Parameshachari, B.D., Karimi, H.R., Gurumoorthy, S.: Deep \(Q\)-network with reinforcement learning for fault detection in cyber-physical systems. J. Circuits, Syst. Comput. 31(9), 2250158 (2022)
Liu, Q., Liu, Z., Xiong, B., Xu, W., Liu, Y.: Deep reinforcement learning-based safe interaction for industrial human-robot collaboration using intrinsic reward function. Adv. Eng. Inform. 49, 101360 (2021)
Yang, Y., Wan, Y., Zhu, J., Lewis, F.L.: \(H_\infty \) tracking control for linear discrete-time systems: model-free \(Q\)-learning designs. IEEE Control Syst. Lett. 5(1), 175–180 (2020)
Vamvoudakis, K.G., Ferraz, H.: Event-triggered \(H_\infty \) control for unknown continuous-time linear systems using \(Q\)-learning. In: 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 1376–1381 (2016)
Wilson, A.C., Recht, B., Jordan, M.I.: A Lyapunov analysis of accelerated methods in optimization. J. Mach. Learn. Res. 22(1), 5040–5073 (2021)
Pertsch, K., Lee, Y., Lim, J.: Accelerating reinforcement learning with learned skill priors. In: Conference on Robot Learning, pp. 188–204 (2021)
Poveda, J.I., Li, N.: Robust hybrid zero-order optimization algorithms with acceleration via averaging in time. Automatica, pp. 109361 (2021)
Ochoa, D.E., Poveda, J.I.: Accelerated continuous-time approximate dynamic programming via data-assisted hybrid control. IFAC-PapersOnLine 55(12), 561–566 (2022)
Su, W., Boyd, S., Candes, E.: A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. J. Mach. Learn. Res. 17(153), 1–43 (2016)
Wang, D., He, H.B., Liu, D.R.: Adaptive critic nonlinear robust control: a survey. IEEE Trans. Cybern. 47(10), 3429–3451 (2017)
Brendan, O.D., Candes, E.: Adaptive restart for accelerated gradient schemes. Found. Comput. Math. 15, 715–732 (2015)
Kamalapurkar, R., Walters, P., Rosenfeld, J., Dixon, W.: Reinforcement Learning for Optimal Feedback Control. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-78384-0
Hornik, K., Stinchcombe, M., White, H.: Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural Networks 3(5), 551–560 (1990)
Chowdhary, G., Johnson, E.: Concurrent learning for convergence in adaptive control without persistency of excitation. In: 49th IEEE Conference on Decision and Control (CDC), pp. 3674–3679 (2010)
Kokolakis, N.M.T., Vamvoudakis, K.G.: Safety-aware pursuit-evasion games in unknown environments using gaussian processes and finite-time convergent reinforcement learning. IEEE Trans. Neural Networks Learn. Syst. (2022). https://doi.org/10.1109/TNNLS.2022.3203977
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wu, Y., Luo, S., Jiang, Y. (2024). Impulsive Accelerated Reinforcement Learning for \(H_\infty \) Control. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Lecture Notes in Computer Science, vol 14448. Springer, Singapore. https://doi.org/10.1007/978-981-99-8082-6_15
Download citation
DOI: https://doi.org/10.1007/978-981-99-8082-6_15
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8081-9
Online ISBN: 978-981-99-8082-6
eBook Packages: Computer ScienceComputer Science (R0)