Abstract
This paper presents a Reinforcement Learning application using a recursive least squares (RLS) with an exponential forgetting (EF) factor to solve the Discrete Linear Quadratic Regulator problem. Temporal Difference learning based RLS algorithm is implemented to find a kernel matrix of the action value function (or Q-function) approximated by neural network. Based on the EF RLS, a New Exponential Forgetting (New EF) factor algorithm is developed by adding a covariance term to the forgetting factor to prevent the estimator windup problem. Numerical simulations on a fixed-wing aircraft are performed to show the effectiveness of the new EF RLS.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Sutton, R.S., Barto, A.G.: Reinforcement Learning - An Introduction, 2nd edn. MIT Press, Cambridge (2018)
Lewis, F.L., Vrabie, D., Syrmos, V.: Optimal Control, 3rd edn. Wiley, Hoboken (2012)
Subbaram, N.D.: Optimal Control Systems. CRC Press, Boca Raton (2002)
Vrabie, D., Vamvoudakis, K.G., Lewis, F.L.: Optimal adaptive control and differential games by reinforcement learning IT the institution of engineering and technology control engineering series, vol. 81, The Institution of Engineering and Technology (2013)
Lewis, F.L., Vrabie, D., Vamvoudakis, K.G.: Reinforcement learning and feedback control: using natural decision methods to design optimal adaptive controllers. IEEE Control Syst. Mag. 32(6), 76–105 (2012)
Vrabie, D., Lewis, F.L.: Neural network approach to continuous time direct adaptive optimal control for partially-unknown nonlinear systems. Neural Netw. 22(3), 237–246 (2009)
Bradtke, S.J., Ydstie, B.E., Barto, A.G.: Adaptive linear quadratic control using policy iteration. In: Proceedings of 1994 Americal Control Conference -ACC 1994, Baltimore, Maryland (1994)
Zhu, L.M., Modares, H., Peen, G.O., Lewis, F.L., Yue, B.: Adaptive suboptimal output-feedback control for linear systems using integral reinforcement learning. IEEE Trans. Control Syst. Technol. 23(1), 264–273 (2014)
Modares, H., Lewis., F.L.: Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning. IEEE Trans. Automat. Control 59(11), 3051–3056 (2014)
Ljung, L.: System Identification: Theory for the user, Englewood Cliffs. Prentice-Hall, Hoboken (1999)
Shin, H.S., Lee, H.I.: A new exponential forgetting algorithm for recursive least-squares parameter estimation. arXiv:2004.03910v [eess.SY], April 2020
Bittani, S., Bolzern, P., Campi, M.: Convergence and exponential convergence of identification algorithms with directional forgetting factor. Automatica 26(5), 929–932 (1990)
Campi, M.: On the convergence of minimum-variance directional-forgetting adaptive control scheme. Automatica 28(1), 221–225 (1992)
Johnstone, R.M., Johnson, C.R., Bitmead, R.R., Anderson, B.D.O.: Exponential convergence of recursive least squares with exponential forgetting factor. Syst. Control Lett. 2(2), 77–82 (1982)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Park, O., Shin, HS., Lee, HI., Antonios, T. (2022). Optimal and Adaptive Control Design Using Recursive Least Square with a New Exponential Forgetting Factor. In: Kim, J., et al. Robot Intelligence Technology and Applications 6. RiTA 2021. Lecture Notes in Networks and Systems, vol 429. Springer, Cham. https://doi.org/10.1007/978-3-030-97672-9_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-97672-9_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-97671-2
Online ISBN: 978-3-030-97672-9
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)