Reinforcement Learning of Structured Stabilizing Control for Linear Systems With Unknown State Matrix

Mukherjee, Sayak; Vu, Thanh Long

doi:10.1109/tac.2022.3155384

Title: Reinforcement Learning of Structured Stabilizing Control for Linear Systems With Unknown State Matrix

Journal Article · Wed Mar 01 00:00:00 EST 2023 · IEEE Transactions on Automatic Control

DOI:https://doi.org/10.1109/tac.2022.3155384· OSTI ID:1971347

^[1];

^[1]

Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

This paper delves into designing feedback control gains for a continuous-time linear quadratic regulator (LQR) problem that is constrained to certain predefined structure with unknown state matrix. We bring forth the ideas from reinforcement learning (RL) in conjunction with sufficient stability and performance guarantees in order to design these structured gains using the trajectory measurements of states and controls. Here we first formulate a model-based framework using dynamic programming (DP) to embed the structural constraint to the LQR gain computation in the continuous-time setting, and then subsequently, formulate a policy iteration RL algorithm that can alleviate the requirement of known state matrix in conjunction with maintaining the feedback gain structure. The design enables a distributed learning control design which is necessary for many large-scale cyber-physical systems. Theoretical guarantees are provided for stability and convergence of the structured reinforcement learning (SRL) algorithm. We validate our theoretical results with numerical simulations on a multi-agent networked linear time-invariant (LTI) dynamic system.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC05-76RL01830

OSTI ID:: 1971347

Report Number(s):: PNNL-SA-156272

Journal Information:: IEEE Transactions on Automatic Control, Vol. 68, Issue 3; ISSN 0018-9286

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (18)

Optimal and Autonomous Control Using Reinforcement Learning: A Survey Kiumarsi, Bahare; Vamvoudakis, Kyriakos G.; Modares, Hamidreza IEEE Transactions on Neural Networks and Learning Systems, Vol. 29, Issue 6 https://doi.org/10.1109/TNNLS.2017.2773458	journal	June 2018
Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approach Vamvoudakis, Kyriakos G. Systems & Control Letters, Vol. 100 https://doi.org/10.1016/j.sysconle.2016.12.003	journal	February 2017
Adaptive optimal control for continuous-time linear systems based on policy iteration Vrabie, D.; Pastravanu, O.; Abu-Khalaf, M. Automatica, Vol. 45, Issue 2 https://doi.org/10.1016/j.automatica.2008.08.017	journal	February 2009
Decentralized control: An overview Bakule, Lubomír Annual Reviews in Control, Vol. 32, Issue 1 https://doi.org/10.1016/j.arcontrol.2008.03.004	journal	April 2008
Distributed Control for Identical Dynamically Coupled Systems: A Decomposition Approach Massioni, Paolo; Verhaegen, Michel IEEE Transactions on Automatic Control, Vol. 54, Issue 1 https://doi.org/10.1109/TAC.2008.2009574	journal	January 2009
On an iterative technique for Riccati equation computations Kleinman, D. IEEE Transactions on Automatic Control, Vol. 13, Issue 1 https://doi.org/10.1109/TAC.1968.1098829	journal	February 1968
Quadratic invariance is necessary and sufficient for convexity Lessard, Laurent; Lall, Sanjay Proceedings of the 2011 American Control Conference https://doi.org/10.1109/ACC.2011.5990928	conference	June 2011
Block-Decentralized Model-Free Reinforcement Learning Control of Two Time-Scale Networks Mukherjee, Sayak; Chakrabortty, Aranya; Bai, He 2019 American Control Conference (ACC) https://doi.org/10.23919/ACC.2019.8815077	conference	July 2019
On Distributed Model-Free Reinforcement Learning Control With Stability Guarantee Mukherjee, Sayak; Vu, Thanh Long IEEE Control Systems Letters, Vol. 5, Issue 5 https://doi.org/10.1109/LCSYS.2020.3041218	journal	November 2021
Robust Adaptive Dynamic Programming for Large-Scale Systems With an Application to Multimachine Power Systems Jiang, Yu; Jiang, Zhong-Ping IEEE Transactions on Circuits and Systems II: Express Briefs, Vol. 59, Issue 10 https://doi.org/10.1109/TCSII.2012.2213353	journal	October 2012
A Characterization of Convex Problems in Decentralized Control$^ast$ Rotkowitz, M.; Lall, S. IEEE Transactions on Automatic Control, Vol. 51, Issue 2 https://doi.org/10.1109/TAC.2005.860365	journal	February 2006
Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics Jiang, Yu; Jiang, Zhong-Ping Automatica, Vol. 48, Issue 10 https://doi.org/10.1016/j.automatica.2012.06.096	journal	October 2012
Adaptive Fault-Tolerant Tracking Control for Discrete-Time Multiagent Systems via Reinforcement Learning Algorithm Li, Hongyi; Wu, Ying; Chen, Mou IEEE Transactions on Cybernetics, Vol. 51, Issue 3 https://doi.org/10.1109/TCYB.2020.2982168	journal	March 2021
Distributed Control Design for Systems Interconnected Over an Arbitrary Graph Langbort, C.; Chandra, R. S.; D'Andrea, R. IEEE Transactions on Automatic Control, Vol. 49, Issue 9 https://doi.org/10.1109/TAC.2004.834123	journal	September 2004
Structural Constrained Controllers for Linear Discrete Dynamic Systems Geromel, J. C.; Melo, E. P. IFAC Proceedings Volumes, Vol. 17, Issue 2 https://doi.org/10.1016/S1474-6670(17)61009-7	journal	July 1984
Perturbation Theory for Algebraic Riccati Equations Sun, Ji-guang SIAM Journal on Matrix Analysis and Applications, Vol. 19, Issue 1 https://doi.org/10.1137/S0895479895291303	journal	January 1998
Reduced-dimensional reinforcement learning control using singular perturbation approximations Mukherjee, Sayak; Bai, He; Chakrabortty, Aranya Automatica, Vol. 126 https://doi.org/10.1016/j.automatica.2020.109451	journal	April 2021
Reinforcement Learning: An Introduction Sutton, R. S.; Barto, A. G. IEEE Transactions on Neural Networks, Vol. 9, Issue 5 https://doi.org/10.1109/TNN.1998.712192	journal	September 1998

Similar Records

Model-based and Model-free Designs for an Extended Continuous-time LQR with Exogenous Inputs

Journal Article · Sun Aug 01 00:00:00 EDT 2021 · Systems and Control Letters · OSTI ID:1971347

Mukherjee, Sayak; Bai, He; Chakrabortty, Aranya

A Secure Learning Control Strategy via Dynamic Camouflaging for Unknown Dynamical Systems under Attacks

Conference · Fri Dec 31 00:00:00 EST 2021 · OSTI ID:1971347

Mukherjee, Sayak; Adetola, Veronica A.

On Distributed Model-Free Reinforcement Learning Control with Stability Guarantee

Journal Article · Mon Nov 01 00:00:00 EDT 2021 · IEEE Control Systems Letters · OSTI ID:1971347

Mukherjee, Sayak; Vu, Thanh Long

Related Subjects

42 ENGINEERING
structured learning
distributed control
reinforcement learning
stability guarantee
linear quadratic regulator

Title: Reinforcement Learning of Structured Stabilizing Control for Linear Systems With Unknown State Matrix

Citation Formats

References (18)

Similar Records

Related Subjects