Elsevier

Neurocomputing

Volume 165, 1 October 2015, Pages 90-98
Neurocomputing

Neural-network-based decentralized control of continuous-time nonlinear interconnected systems with unknown dynamics

https://doi.org/10.1016/j.neucom.2014.07.082Get rights and content

Abstract

In this paper, we establish a neural-network-based decentralized control law to stabilize a class of continuous-time nonlinear interconnected large-scale systems using an online model-free integral policy iteration (PI) algorithm. The model-free PI approach can solve the decentralized control problem for the interconnected system which has unknown dynamics. The stabilizing decentralized control law is derived based on the optimal control policies of the isolated subsystems. The online model-free integral PI algorithm is developed to solve the optimal control problems for the isolated subsystems with unknown system dynamics. We use the actor-critic technique based on the neural network and the least squares implementation method to obtain the optimal control policies. Two simulation examples are given to verify the applicability of the decentralized control law.

Introduction

Decentralized control method using local information of each subsystem is an efficient and effective way in the control of interconnected systems. This overcomes the limitations of the traditional control method that requires sufficient information between subsystems. Unlike a centralized controller, a decentralized controller can be designed independently for local subsystems and make full use of the local available signals for feedback. Therefore, the decentralized controllers have simpler architecture, and are more practical than the traditional centralized controllers. Various decentralized controllers have been established for large-scale interconnected systems in the presence of uncertainties and information structure constraints [1], [2], [3], [4], [5], [6], [7]. Generally speaking, a decentralized control law is composed of some noninteracting local controllers corresponding to the isolated subsystems, not the overall system. In many situations, the design of the isolated subsystems is very important. In [8], the decentralized controller was derived for the large-scale system using the optimal control policies of the isolated subsystems. Therefore, the optimal control method can be applied to facilitate the design process of the decentralized control law.

The optimal control problem of nonlinear systems has been widely studied in the past few decades. The optimal control policy can be obtained by solving Hamilton–Jacobi–Bellman (HJB) equation which is a partial differential equation. Because of the curse of dimensionality [9], this is a difficult task even in the case of completely known dynamics. Among the methods of solving the HJB equation, adaptive dynamic programming (ADP) has received increasing attention owing to its learning and optimal capacities [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20]. Reinforcement learning (RL) is another computational method and it can interactively find an optimal policy [21], [22], [23], [24]. Al-Tamimi et al. [25] proposed a greedy iterative ADP to solve the optimal control problem for nonlinear discrete-time systems. Park et al. [26] used multilayer neural networks (NNs) to design a finite-horizon optimal tracking neuro-controller for discrete-time nonlinear systems with quadratic cost function. Abu-Khalaf and Lewis [27] established an offline optimal control law for nonlinear systems with saturating actuators. Vamvoudakis and Lewis [28] derived a synchronous policy iteration (PI) algorithm to learn online continuous-time optimal control with known dynamics. Vrabie and Lewis [29] derived an integral RL method to obtain direct adaptive optimal control for nonlinear input-affine continuous-time systems with partially unknown dynamics. Jiang and Jiang [30] presented a novel PI approach for continuous-time linear systems with complete unknown dynamics. Liu et al. [31] extended the PI algorithm to nonlinear optimal control problem with unknown dynamics and discounted cost function. Lee et al. [32], [33] presented an integral Q-learning algorithm for continuous-time systems without the exact knowledge of the system dynamics.

It is difficult to obtain the exact knowledge of the system dynamics for large-scale systems, such as transportation systems and power systems. The novelty of this paper is that we relax the assumptions of exact knowledge of the system dynamics required in the optimal controller design presented in [8]. In this paper, we use an online model-free integral PI to solve the decentralized control of a class of continuous-time nonlinear interconnected systems. We establish the stabilizing decentralized control law by adding feedback gains to the local optimal polices of the isolated subsystems. The optimal control problems for the isolated subsystems with unknown dynamics are related to develop the decentralized control law. To implement this algorithm, a critic NN and an action NN are used to approximate the value function and control policy of the isolated subsystem, respectively. The effectiveness of the decentralized control law established in this paper is demonstrated by two simulation examples.

The rest of this paper is organized as follows. In Section 2, we present the decentralized control problem of the continuous-time nonlinear large-scale interconnected system. Section 3 presents the decentralized stabilization control law for the continuous-time interconnected system by adding appropriate feedback gains to the local optimal polices of the isolated subsystems. In Section 4, we derive a model-free PI algorithm using NN implementation to obtain the decentralized control law. Two simulation examples are provided in Section 5 to illustrate the effectiveness of the derived decentralized control law. In Section 6, we conclude the paper with a few remarks.

Section snippets

Problem formulation

We consider a continuous-time nonlinear large-scale system Σ composed of N interconnected subsystems described byΣ:ẋi(t)=fi(xi(t))+gi(xi(t))(ui(xi(t))+Zi(x(t)))i=1,2,,Nwhere xi(t)Rni is the state, ui(xi(t))Rmi is the control input vector of the ith subsystem. The overall state of the large-scale system Σ is denoted by x=[x1Tx2TxNT]TRn, where n=i=1Nni. The local states are represented by x1, x2, …, xN, whereas u1(x1), u2(x2), …, uN(xN) are local controls. For the ith subsystem, fi is a

Decentralized control law

In this section, we present the decentralized controller design. The optimal control problem of the isolated subsystems is described under the framework of HJB equations. The decentralized control law is derived by adding some local feedback gains to the isolated optimal control policies.

NN-based implementation using online model-free PI algorithm

In this section, we discuss the implementation of the decentralized control law presented in Section 3. We introduce the online PI algorithm in the first subsection. A model-free integral PI algorithm is derived to solve the optimal control problem with completely unknown dynamics in the second subsection. A NN-based implementation of the established model-free integral PI algorithm is discussed at last.

Numerical simulations

Two simulation examples are provided in this section to demonstrate the effectiveness of the decentralized control law established in this paper.

Conclusion

In this paper, a stabilizing decentralized control law for a class of nonlinear large-scale systems with unknown dynamics is established using a NN-based online model-free integral PI algorithm. The decentralized control law is derived by the optimal controllers of the isolated subsystems. We use an online model-free integral PI algorithm with an exploration to solve the HJB equations related to the optimal control problem of the isolated subsystems. To implement the constructed algorithm, we

Derong Liu received the B.S. degree in Mechanical Engineering from the East China Institute of Technology (now Nanjing University of Science and Technology), Nanjing, China, in 1982, the M.S. degree in Automatic Control Theory and Applications from the Institute of Automation, Chinese Academy of Sciences, Beijing, China, in 1987, and the Ph.D. degree in Electrical Engineering from the University of Notre Dame, Indiana, USA, in 1994. Dr. Liu was a Product Design Engineer with China North

References (33)

  • J.Y. Lee et al.

    Integral Q-learning and explorized policy iteration for adaptive optimal control of continuous-time linear systems

    Automatica

    (2012)
  • P. Ioannou

    Decentralized adaptive control of interconnected systems

    IEEE Trans. Autom. Control

    (1986)
  • J.T. Spooner et al.

    Decentralized adaptive control of nonlinear systems using radial basis neural networks

    IEEE Trans Autom. Control

    (1999)
  • K. Kalsi et al.

    Decentralized dynamic output feedback control of nonlinear interconnected systems

    IEEE Trans. Autom. Control

    (2010)
  • J. Lavaei

    Decentralized implementation of centralized controllers for interconnected systems

    IEEE Trans. Autom. Control

    (2012)
  • D. Liu et al.

    Decentralized stabilization for a class of continuous-time nonlinear interconnected systems using online learning optimal control approach

    IEEE Trans. Neural Netw. Learn. Syst.

    (2014)
  • Cited by (0)

    Derong Liu received the B.S. degree in Mechanical Engineering from the East China Institute of Technology (now Nanjing University of Science and Technology), Nanjing, China, in 1982, the M.S. degree in Automatic Control Theory and Applications from the Institute of Automation, Chinese Academy of Sciences, Beijing, China, in 1987, and the Ph.D. degree in Electrical Engineering from the University of Notre Dame, Indiana, USA, in 1994. Dr. Liu was a Product Design Engineer with China North Industries Corporation, Jilin, China, from 1982 to 1984. He was an Instructor with the Graduate School of the Chinese Academy of Sciences, Beijing, from 1987 to 1990. He was a Staff Fellow with General Motors Research and Development Center, from 1993 to 1995. He was an Assistant Professor with the Department of Electrical and Computer Engineering, Stevens Institute of Technology, from 1995 to 1999. He joined the University of Illinois at Chicago in 1999, and became a Full Professor of Electrical and Computer Engineering and of Computer Science in 2006. He was selected for the “100 Talents Program” by the Chinese Academy of Sciences in 2008, and now he serves as the Associate Director of The State Key Laboratory of Management and Control for Complex Systems at the Institute of Automation. He has published 15 books (six research monographs and nine edited volumes). Dr. Liu was an Associate Editor of Automatica from 2006 to 2009. Currently, he is an elected AdCom member of the IEEE Computational Intelligence Society and he is the Editor-in-Chief of the IEEE Transactions on Neural Networks and Learning Systems. He also serves as an Associate Editor of IEEE Transactions on Control Systems Technology, IEEE Transactions on Systems, Man, and Cybernetics: Systems, IEEE Transactions on Intelligent Transportation Systems, Soft Computing, Neurocomputing, Neural Computing and Applications, and Science in China Series F: Information Sciences. He was an Associate Editor of the IEEE Transactions on Circuits and Systems-I: Fundamental Theory and Applications from 1997 to 1999, the IEEE Transactions on Signal Processing from 2001 to 2003, the IEEE Transactions on Neural Networks from 2004 to 2009, the IEEE Computational Intelligence Magazine from 2006 to 2009, and the IEEE Circuits and Systems Magazine from 2008 to 2009, and the Letters Editor of the IEEE Transactions on Neural Networks from 2006 to 2008. He received the Faculty Early Career Development Award from the National Science Foundation in1999, the University Scholar Award from University of Illinois from 2006 to 2009, and the Overseas Outstanding Young Scholar Award from the National Natural Science Foundation of China in 2008. He is a Fellow of the IEEE and a Fellow of the International Neural Network Society.

    Chao Li received the B.S. degree in Mechatronics from the Nanjing University of Science and Technology in 2012. He is currently working toward the Ph.D. degree in the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. He is also with the University of Chinese Academy of Sciences, Beijing. His research interests include neural networks, reinforcement learning and adaptive dynamic programming.

    Hongliang Li received the B.S. degree in Mechanical Engineering and Automation from Beijing University of Posts and Telecommunications in 2010. He is currently working toward the Ph.D. degree in the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. He is also with the University of Chinese Academy of Sciences, Beijing. His research interests include neural networks, reinforcement learning, adaptive dynamic programming, game theory and multi-agent systems.

    Ding Wang received the B.S. degree in mathematics from the Zhengzhou University of Light Industry, Zhengzhou, China, the M.S. degree in operational research and cybernetics from Northeastern University, Shenyang, China, and the Ph.D. degree in Control Theory and Control Engineering from the Institute of Automation, Chinese Academy of Sciences, Beijing, China, in 2007, 2009, and 2012, respectively. He is currently an assistant professor with the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences. His research interests include adaptive dynamic programming, neural networks, and intelligent control.

    Hongwen Ma received the B.S. degree in Electric Engineering and Automation from the Nanjing University of Science and Technology in 2012. He is currently working toward the Ph.D. degree in the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. He is also with the University of Chinese Academy of Sciences, Beijing. His research interests include neural networks, networked control systems, and multi-agent systems.

    This work was supported in part by the National Natural Science Foundation of China under Grants 61034002, 61233001, 61273140, 61304086, and 61374105.

    View full text