Completely model-free RL-based consensus of continuous-time multi-agent systems

Abstract

In this paper, we study the consensus of continuous-time general linear multi-agent systems in the absence of the model information by using the adaptive dynamic programming (ADP) based reinforcement learning (RL) approach. The introduction of the RL approach is to learn the feedback gain matrix to fulfill the construction of the control algorithm to guarantee the reach of consensus only on the basis of the available information. For the state feedback control, the RL algorithm relates only to the state and the input of an arbitrary agent, while for the output feedback control, the RL algorithm depends only on the input and output information of an arbitrary agent, irrelevant any model information. Finally, numerical simulations are given to verify the main results.

Introduction

Since the start of this century, the cooperative control of multi-agent systems has absorbed considerable attention. The investigations on this field were carried out from the perspectives of communication topology [1], [2], [3], [4], the dynamics [5], [6], [7], [8], [9], the effectiveness of the time-delay [10] or the disturbances [11], [12] or external uncertainty [13] and so on. Consensus, as the most typical cooperative behavior, means that each agent in the multi-agent system reaches an agreement and then naturally receives a lot of attention [1], [3], [6], [7], [14]. The existing investigations on consensus include the leader-less consensus [6], [13], [15], the leader-following consensus [12], [16], [17], [18], [19] and containment of multi-agent systems with multiple leaders [20], [21], [22], [23]. For the investigations on the consensus of multi-agent systems, the construction of the feedback gain matrix in the control algorithm plays a crucial role which can influence or even determine the stability of consensus of systems. In [5] and [6], algebraic Riccati equation (ARE) is introduced to the construction of the feedback gain matrix, then the cooperative behavior can be achieved by only tuning the coupling strength constants. In [13], [15], the authors used the parameterized ARE based low-gain technique to solve the consensus of multi-agent systems subject to input saturation. Notice that the construction of the feedback gain matrix in the literature mentioned above related to the model information or named as the model matrices [5], [6], [13], [14], [15]. However, in practical application, the model information in the multi-agent system may not be acquired. How can we solve the consensus problem of the model-free multi-agent systems? In this paper, we will solve this problem by using the RL approach.

Recently, the RL approach is widely used in the cooperative control of multi-agent systems, which can learn and make decisions through interacting with the given environment. Roughly speaking, the RL based investigations on the consensus of multi-agent systems can be divided into two categories: the investigations on discrete-time multi-agent systems [21], [24], [25], [26] and those on continuous-time systems [19], [27]. Compared with the cooperative control of multi-agent systems without RL approach mentioned above, the most remarkable point is the RL results can deal with the coordination control of model-free multi-agent systems. In other words, with the help of the RL approach, we can design control protocol to drive the systems to achieve consensus without model information. As declared in [28], the model-free RL based consensus for discrete-time individual systems [29] is much more difficult than that of the continuous-time individual systems [29], [30] and this fact is also fit for multi-agent systems. The optimal consensus tracking and containment of discrete-time multi-agent systems were investigated in [21], [24] respectively by using the RL method. Also for discrete-time multi-agent systems, [25] solved the global consensus of multi-agent systems subject to input saturation. All the discrete-time investigations in [21], [24], [25], [26] were model-free and all the information related in the construction of the control algorithm was in the discrete-time format. References [29], [30] switched to the RL based consensus of continuous-time multi-agent systems without model information. However, the RL algorithms constructed in these two references were not in continuous-time essentially but provided by discretizing the model.

In this paper, we refocus on the RL based consensus of the continuous-time multi-agent systems with completely unknown model information. The most notable point is that all the theoretical analysis relates to schemes completely in continuous-time. The investigation includes the state feedback control and the output feedback one. For the state feedback control, the model-free ADP based RL method is used to learn the feedback gain matrix instead of solving a parameterized algebraic Riccati equation (ARE), since the ARE closely depends on the model information. The learning of the feedback gain matrix relates only to state and control input of arbitrary one agent of the multi-agent system and circumvents to the information communication among the multi-agent systems, without the requirement of the model information. Taking the fact that it is hard or even impossible to get the state of arbitrary one agent, we also investigate the RL based consensus of model-free multi-agent systems by using the output feedback control. The ADP based RL algorithm is design to learn the feedback gain matrix on the basis of only the output information and the control input of arbitrary one agent, irrelevant to any model information. In analogy to the operation in [28], we firstly express the estimated information of each agent in term of the output information and the control input by doing Laplace transformation and Laplace inverse transformation to each agent. Then, ADP based RL algorithm is given to learn the feedback gain matrix. It is the introduction of the motivated operation in [28], the objective of the consensus on continuous-time multi-agent systems without model information can be achieved, rather than the discretization used in [29], [30].

The rest of the paper is arranged as follows. In Section 2, we give preliminaries about graph theory and state the main problem of this paper. In Section 3, the RL based consensus will be solved via state feedback control, while in Section 4, the RL based consensus will be solved via output feedback control. Section 5 provides numerical simulations to verify the main results obtained in Sections 3 and 4. At last, Section 6 concludes the whole paper.

Nomenclature

$\mathbb{R}$ and ${\mathbb{R}}^{n\times m}$ denote the set of real numbers and the set of n × m real matrices, respectively. ${\mathbb{Z}}_{+}$ is the set of positive integers ${\mathbb{P}}^{n\times n}$ denotes the normed space of all $n-$by$-n$ real symmetric matrices, equipped with the induced matrix norm. ${\mathbb{P}}_{+}^{n\times n}=\{P\in {\mathbb{P}}^{n\times n}:P⪰0\}$. For any $A\in {\mathbb{R}}^{n\times n},$ $\text{vec}\left(A\right)=(a_1^T{a}_2^T\dots a_n^T)\in {\mathbb{R}}^{n^2\times 1},$ with $a_i\in {\mathbb{R}}^{n\times 1}$ being the ith column vector of A. For any $A\in {\mathbb{P}}_{+}^{n\times n},$

$$\text{vecs}\left(A\right)={(a_{11}{a}_{12}\dots a_{1n}{a}_{22}{a}_{23}\dots a_{2n}\dots a_{(n-1)n}{a}_{nn})}^T\in {\mathbb{R}}^{\frac{n(n+1)}{2}\times 1}.$$

⊗ represents the Kronecker product of two matrices (or vectors) with compatible dimension.