Elsevier

Automatica

Volume 76, February 2017, Pages 301-308
Automatica

Brief paper
Infinite horizon linear-quadratic Stackelberg games for discrete-time stochastic systems

https://doi.org/10.1016/j.automatica.2016.10.016Get rights and content

Abstract

In this paper, we consider infinite horizon linear-quadratic Stackelberg games for a discrete-time stochastic system with multiple decision makers. Necessary conditions for the existence of the Stackelberg strategy set are derived in terms of the solvability of cross-coupled stochastic algebraic equations (CSAEs). As an important application, the hierarchical H-constraint control problem for the infinite horizon discrete-time stochastic system with multiple channel inputs is solved using the Stackelberg game approach. Computational methods for solving the CSAEs are also discussed. A numerical example is provided to demonstrate the usefulness of the proposed algorithms.

Introduction

Many researchers have extensively investigated the dynamic games of continuous- and discrete-time systems (see Basar & Olsder, 1999 and the references therein). Recently, due to the growth of interest in multi-agent and cooperative systems, optimal cooperation and team collaboration have been widely investigated. For example, neural networks were utilized to find suitable approximations of the optimal-value function and saddle-point solutions (Zhang, Qin, Jiang, & Luo, 2014). The conditions for the existence of Pareto optimal solutions for linear-quadratic infinite horizon cooperative differential games were derived (Reddya & Engwerda, 2013). The design method for the synchronization control of discrete-time multi-agent systems on directed communication graphs has been discussed (Movric, You, Lewis, & Xie, 2013). The open-loop Nash games for a class of polynomial system via a state-dependent Riccati equation have been studied (Lizarraga, Basin, Rodriguez, & Rodriguez, 2015). Among various dynamic games, the Stackelberg game is one that involves hierarchical decisions among different decision makers, yielding significant and useful work (Abou-Kandil, 1990, Basar et al., 2010, Bensoussan et al., 2013, Bensoussan et al., 2014, Bensoussan et al., 2015, Jungers and Oara, 2010, Jungers et al., 2008, Jungers et al., 2011, Li et al., 2002, Medanic, 1978, Xu et al., 2015). Although many researchers have studied Stackelberg games, they have only focused on those for finite horizon deterministic continuous- or discrete-time systems. To the best of our knowledge, the Stackelberg game for infinite horizon discrete-time systems is still unsolved. The infinite horizon Stackelberg game is difficult to solve, because it involves some higher-order algebraic nonlinear matrix equations rather than difference equations in the finite horizon case (Abou-Kandil, 1990) and the team-optimal state feedback Stackelberg strategy case (Li et al., 2002).

Recent advances in the theory of discrete-time stochastic systems have led to the reconsideration of various control problems for infinite horizon discrete-time stochastic systems (Huang et al., 2008, Zhang et al., 2008). It has been shown that the optimal or mixed H2/H feedback controllers can be constructed by solving certain stochastic algebraic Riccati equations (SAREs). However, corresponding results have not been found in dynamic game settings, especially in dynamic games with multiple decision makers, except for Mukaidani, Tanabata, and Matsumoto (2014) who only considered Nash games. Taking into consideration the fact that the dynamic game approaches have found many applications in the control field, the investigation of hierarchical dynamic games for discrete-time stochastic systems is extremely promising.

The Stackelberg games for a class of stochastic systems have been considered (Mukaidani, 2013, Mukaidani, 2014a, Mukaidani, 2014b, Mukaidani and Xu, 2015). In Mukaidani (2013) and Mukaidani and Xu (2015), the Stackelberg games for the continuous-time stochastic systems with multiple decision makers have been studied. In Mukaidani, 2014a, Mukaidani, 2014b, the Stackelberg games for a discrete-time stochastic system with one leader and one follower have been studied. In particular, in Mukaidani, 2013, Mukaidani, 2014a, the mixed H2/H control problem was investigated where the follower is interpreted as a deterministic disturbance.

In this paper, we investigated the infinite horizon linear-quadratic Stackelberg game for a class of discrete-time linear stochastic systems with state-dependent noise. The results in Mukaidani (2014a) were merely preliminary results to study a standard Stackelberg game with one follower. In this paper, we have extended the results in Mukaidani (2014a) to the Stackelberg game with multiple followers. Moreover, we have studied the hierarchical H-constraint control problem with multiple decision-makers using this Stackelberg game approach. Although the Lagrange-multiplier technique was used to derive the necessary conditions in the same way as in Mukaidani and Xu (2015), the derivation procedures, and consequently the results, were completely different.

Notation: The notations used in this paper are fairly standard. The superscript T denotes matrix transpose. In denotes the n×n identity matrix. E[] denotes the expectation operator. Tr denotes the trace of a matrix. vec denotes the column vector of a matrix. δij denotes the Kronecker delta. block diag denotes the block diagonal matrix. X denotes a set of {X0,X1,,XN}. ρ() denotes the spectral radius function.

Section snippets

Preliminary

Before investigating the Stackelberg game, some useful lemmas are introduced. Consider the following discrete-time stochastic system. x(k+1)=Ax(k)+Bu(k)+Apx(k)w(k),x(0)=x0,y(k)=Cx(k), where x(k)Rn represents the state vector, u(k)Rm represents the control input, y(k)Rp represents the measured output, and w(k)R is a one-dimensional sequence of real random process defined in a complete probability space, which is a wide sense stationary, second-order process with E[w(k)]=0 and E[w(s)w(k)]=δsk

Stackelberg game with multiple followers

The following stochastic systems with N+1-decision makers involving state-dependent noise is considered. x(k+1)=Ax(k)+B0u0(k)+i=1NBiui(k)+Apx(k)w(k),x(0)=x0, where ui(k)Rmi, i=0,1,,N represents the ith control input. Moreover, i=0 represents the leader’s control input, and the other values for i represent the follower’s inputs. The initial state x(0)=x0 is assumed to be a random variable with a covariance matrix E[x(0)xT(0)]=In.

Without loss of generality, the following basic assumption is

Application to hierarchical H-constraint control

As an important application of the Stackelberg game, the hierarchical H-constraint control is investigated in this section. The following stochastic system is considered. x(k+1)=Ax(k)+B0u0(k)+i=1NBiui(k)+Dv(k)+Apx(k)w(k),x(0)=x0,z(k)=[[Cx(k)]Tu0T(k)u1T(k)uNT(k)]T, where v(k)Rnv represents the external disturbance. z(k)Rnz represents the controlled output.

The following preliminary results play an important role in the considered control problem.

Definition 3

Zhang et al., 2008

Consider the following autonomous stochastic

Numerical algorithm for solving (32)

To obtain the solutions of Eq. (32), the numerical algorithm based on the steepest descent method can be used. It is well known that the steepest descent method can be viewed as Euler’s method for solving ordinary differential equations. Thus, the convergence of the algorithm based on the steepest descent method is guaranteed, if the stability of the following nonlinear descriptor system is satisfied. M̃i̇=Φi(M̃i)=M̃i+ÃKTM̃iÃK+ApTM̃iAp+K̃iTRiiK̃i+Q̃K̃i,i=1,,N,M̃0̇=Φ0(M̃0)=M̃0+ÃKTM̃0ÃK+ApT

Numerical example

The efficiency of the proposed approach is demonstrated by resolving the horizontal vibrations suppression control problem for a six-dimensional discrete-time system, which is based on an aircraft model in level flight, and is subjected to wind gust turbulence (Gajić & Shen, 1993). In particular, the following features are illustrated in this example. (1) A non-cooperative hierarchical strategy set for practical control problem can be obtained by solving the CSAE (32). (2) The considered

Conclusions

In this paper, the infinite horizon Stackelberg games for discrete-time stochastic linear-quadratic systems with multiple decision-makers have been investigated. Firstly, the necessary conditions for the existence of the Stackelberg strategy set have been given in terms of the solvability of the CSAEs. Secondly, as an important application, the hierarchical H-constraint control problem has also been formulated and compared with the existing H2/H control formulation. It should be noted that

Hiroaki Mukaidani received his B.S. in Integrated Arts and Sciences from Hiroshima University in 1992 and his M.Eng. and Dr.Eng. degrees in Information Engineering from Hiroshima University in 1994 and 1997, respectively. In April 1998, he joined the Faculty of Information Science at Hiroshima City University as a Research Associate. Since April 2002, he has been with the Graduate School of Education at Hiroshima University as an Assistant Professor and Associate Professor. He became a Full

References (25)

  • T. Basar et al.

    Differential games with mixed leadership: The open-loop solution

    Applied Mathematics and Computation

    (2010)
  • H. Mukaidani et al.

    Stackelberg strategies for stochastic systems with multiple followers

    Automatica

    (2015)
  • W. Zhang et al.

    Infinite horizon stochastic H2/H control for discrete-time systems with state and disturbance dependent noise

    Automatica

    (2008)
  • H. Abou-Kandil

    Closed-form solution for discrete-time linear-quadratic Stackelberg games

    Journal of Optimization Theory and Applications

    (1990)
  • Basar, T., & Olsder, G.J. (1999). Dynamic Noncooperative Game Theory, Philadelphia: SIAM Series in Classics in Applied...
  • A. Bensoussan et al.

    Linear quadratic differential games with mixed leadership: the open-loop solution

    Numerical Algebra, Control and Optimization

    (2013)
  • A. Bensoussan et al.

    Feedback Stackelberg solutions of infinite-horizon stochastic differential games

  • A. Bensoussan et al.

    The maximum principle for global solutions of stochastic Stackelberg differential games

    SIAM Control and Optimization

    (2015)
  • Z. Gajić et al.

    Parallel algorithms for optimal control of large scale linear systems

    (1993)
  • Y. Huang et al.

    Infinite horizon linear quadratic optimal control for discrete-time stochastic systems

    Asian Journal of Control

    (2008)
  • Jungers, M., & Oara, C. (2010). Anti-palindromic pencil formulation for open-loop stackelberg strategy in...
  • Jungers, M., Trelat, E., & Abou-Kandil, H. (2008). A Stackelberg game approach to mixed H2/H∞ control, In Proceedings...
  • Cited by (0)

    Hiroaki Mukaidani received his B.S. in Integrated Arts and Sciences from Hiroshima University in 1992 and his M.Eng. and Dr.Eng. degrees in Information Engineering from Hiroshima University in 1994 and 1997, respectively. In April 1998, he joined the Faculty of Information Science at Hiroshima City University as a Research Associate. Since April 2002, he has been with the Graduate School of Education at Hiroshima University as an Assistant Professor and Associate Professor. He became a Full Professor at the Institute of Engineering at Hiroshima University in April 2012. He spent 10 months (from November 2007 to September 2008) as a research fellow at the Department of Electrical and Computer Engineering (ECE), University of Waterloo. His research interests include robust control, dynamic games, and the application of stochastic systems. Dr. Mukaidani is a member of the Institute of Electrical and Electronic Engineers (IEEE).

    Hua Xu received his B.Eng. and M.Eng. degrees in Electrical Engineering from Northeastern University, China, in 1982 and 1985, respectively, and his Dr.Eng. degree in Information Engineering from Hiroshima University, Japan, in 1993. He worked with Hiroshima University as a Research Associate and Associate Professor from 1993 to 1998. Since 1998, he has been with the Graduate School of Business Sciences, the University of Tsukuba, Tokyo, Japan, as an Associate Professor, and Professor. His current research interests include dynamic optimization, dynamic games and their applications in business aspects. He is a member of International Society of Dynamic Games and several other academic societies of Japan.

    This work was supported by JSPS KAKENHI Grant Numbers 26330027, 16K00029. The material in this paper was partially presented at the 2014 American Control Conference, June 4–6, 2014, Portland, OR, USA. This paper was recommended for publication in revised form by Associate Editor Michael V. Basin under the direction of Editor Ian R. Petersen.

    1

    Fax: +81 82 424 6476.

    View full text