Minimax Q-learning design for H_∞ control of linear discrete-time systems

Abstract

The H_∞ control method is an effective approach for attenuating the effect of disturbances on practical systems, but it is difficult to obtain the H_∞ controller due to the nonlinear Hamilton—Jacobi—Isaacs equation, even for linear systems. This study deals with the design of an H_∞ controller for linear discrete-time systems. To solve the related game algebraic Riccati equation (GARE), a novel model-free minimax Q-learning method is developed, on the basis of an offline policy iteration algorithm, which is shown to be Newton’s method for solving the GARE. The proposed minimax Q-learning method, which employs off-policy reinforcement learning, learns the optimal control policies for the controller and the disturbance online, using only the state samples generated by the implemented behavior policies. Different from existing Q-learning methods, a novel gradient-based policy improvement scheme is proposed. We prove that the minimax Q-learning method converges to the saddle solution under initially admissible control policies and an appropriate positive learning rate, provided that certain persistence of excitation (PE) conditions are satisfied. In addition, the PE conditions can be easily met by choosing appropriate behavior policies containing certain excitation noises, without causing any excitation noise bias. In the simulation study, we apply the proposed minimax Q-learning method to design an H_∞ load-frequency controller for an electrical power system generator that suffers from load disturbance, and the simulation results indicate that the obtained H_∞ load-frequency controller has good disturbance rejection performance.

摘要

H_∞控制是一种消除系统扰动的有效方式, 但是由于需要求解非线性哈密顿—雅克比—伊萨克斯方程, H_∞控制器往往很难得到, 即便对于线性系统. 本文考虑了线性离散时间系统的H_∞控制器设计问题. 为求解涉及的博弈代数黎卡提方程, 在离线策略算法基础上提出一种新型无模型极小极大Q-学习算法, 并证明离线策略迭代算法是求解博弈代数黎卡提方程的牛顿法. 提出的极小极大Q-学习算法采用离轨策略强化学习技术, 利用行为策略产生的系统状态数据, 可实现对最优控制器和最佳干扰策略的在线学习. 不同于当前Q-学习算法, 本文提出一种基于梯度的策略提高方法. 证明在一定持续激励条件下, 对于初始可行的控制策略并结合合适学习率, 提出的极小极大Q-学习算法可收敛到鞍点策略. 此外, 算法收敛所需的持续激励条件可通过选择包含一定噪声激励的合适行为策略实现, 且不会引起任何激励噪声偏差. 将提出的极小极大Q-学习算法用于受负载扰动的电力系统H_∞负载频率控制器设计, 仿真结果表明, 最终得到的H_∞负载频率控制器具有良好抗干扰性能.