An Arbitrary Starting Tracing Procedure for Computing Subgame Perfect Equilibria

Abstract

The computation of subgame perfect equilibrium in stationary strategies is an important but challenging problem in applications of stochastic games. In 2004, Herings and Peeters developed a homotopy method called stochastic linear tracing procedure to solve this problem. However, the starting point of their method requires to be explicitly calculated. To remedy this issue, we formulate an arbitrary starting linear tracing procedure in this paper. By introducing a homotopy variable ranging from two to zero, an artificial penalty game is developed, whose solutions construct a differentiable path after a well-chosen transformation of variables. The starting point of the path can be arbitrarily chosen, so that there is no need to employ additional algorithms to obtain it. Following the path, one can readily attain the “starting point” of the stochastic tracing procedure coined by Herings and Peeters. Then, as the homotopy variable changes from one to zero, the path essentially resumes to the stochastic tracing procedure. We prove that our method globally converges to a subgame perfect equilibrium in stationary strategies for the stochastic game of interest. Numerical results further illustrate the effectiveness and efficiency of our method.

Appendices

Appendices

A: Proof of Theorem 3.2

In this appendix, we intend to prove that zero is a regular value of $q(y,\mu ,t;\alpha )$ when $0< t\le 2$. This result is applied in the proof of Theorem 3.2.

First, we consider the case that $0<t<2$. For simplicity, we rewrite q as

$$\begin{aligned} q=(q_{\omega j}^{1i},q_{\omega }^{2i})^\top _{\omega \in \varOmega ,i\in N,j\in M_{\omega }^i}, \end{aligned}$$

where

$$\begin{aligned}&q_{\omega j}^{1i}(y,\mu ,t;\alpha )\\&\quad :=(1-\zeta (t))((1-\nu (t))(u^i(\omega ,s^i_{\omega j},x_\omega ^{-i}(y))+\delta \sum _{\bar{\omega }\in \varOmega } \pi (\bar{\omega }\;|\;\omega , s^{i}_{\omega j}, x^{-i}_{\omega }(y))\mu _{\bar{\omega }}^i)\\&\qquad +\nu (t)(u^i(\omega ,s^i_{\omega j},p_\omega ^{0,-i})+\delta \sum _{\bar{\omega }\in \varOmega } \pi (\bar{\omega }\;|\;\omega , s^{i}_{\omega j}, p^{0,-i}_{\omega })\mu _{\bar{\omega }}^i))\\&\qquad -\zeta (t)(x_{\omega j}^i(y)-x_{\omega j}^{0,i})+\lambda ^i_{\omega j}(y)-\mu _{\omega }^i-t(2-t)\alpha _{\omega j}^i, \end{aligned}$$

and

$$\begin{aligned} q_{\omega }^{2i}(y,\mu ,t;\alpha ):=\sum _{j\in M^i_{\omega }} x^i_{\omega j}(y)-1. \end{aligned}$$

Let $v:=(y,\mu ,t;\alpha )\in \mathbb R^m\times \varLambda \times (0,2) \times {\mathbb {R}}^{m}$. Then, the Jacobian matrix of q is a $(m+nd)\times (2m+nd+1)$ matrix, which can be written as

$$\begin{aligned} J:=\frac{\partial q}{\partial v}=\begin{pmatrix} \partial q^1 / \partial v\\ {} &{}\quad \partial q^2/ \partial v\end{pmatrix}= \begin{pmatrix} \dfrac{\partial q^1}{\partial y}&{}\quad \dfrac{\partial q^1}{\partial \mu }&{} \quad \dfrac{\partial q^1}{\partial t}&{}\quad \dfrac{\partial q^1}{\partial \alpha }\\ \dfrac{\partial q^2}{\partial y}&{}\quad 0&{}\quad 0&{}\quad 0 \end{pmatrix}, \end{aligned}$$

where $\dfrac{\partial q^1}{\partial y}\in {\mathbb {R}}^{m}\times {\mathbb {R}}^m$, $\dfrac{\partial q^1}{\partial \mu }\in {\mathbb {R}}^m \times {\mathbb {R}}^{nd}$, and $\dfrac{\partial q^1}{\partial t} \in {\mathbb {R}}^m \times {\mathbb {R}}^1$. Besides, $\dfrac{\partial q^1}{\partial \alpha }=-t(2-t) \mathbf{I _{\varvec{m}}}$, where $\mathbf{I _{\varvec{m}}}$ is an $m\times m$ identity matrix. Thus, when $0<t<2$, $\dfrac{\partial q^1}{\partial \alpha }$ is a diagonal matrix with full rank. Let $r_{\omega j}^i:=\max \{0,y_{\omega j}^i\}$ and $r_{\omega }^i$ be a vector $(r_{\omega 1}^i, r_{\omega 2}^i,\ldots ,r_{\omega m_{\omega }^i}^i)$. Then, $\dfrac{\partial q^2}{\partial y}=2\times \mathrm {diag} (r_{\omega }^i)$ is a $(nd\times m)$ matrix. Obviously, $\dfrac{\partial q^2}{\partial y}$ is of full-row rank, because for any pair of i and $\omega$, there must be at least one positive $r_{\omega j}^i$$(j\in M_{\omega }^i)$, and any two row vectors of the matrices are linearly independent. It follows from the basic operations of matrices that the Jacobian matrix J is of full-row rank.

Second, we discuss the case that $t=2$, where the Jacobian matrix is defined as $J_1$. Specifically speaking,

$$\begin{aligned}J_1:=\begin{pmatrix} J_{11}&{} \quad J_{12}\\ \dfrac{\partial q^2}{\partial y}&{}\quad 0 \end{pmatrix}\in {\mathbb {R}}^{(m+nd)\times (m+nd)}, \end{aligned}$$

where

$$\begin{aligned} J_{11}=\left( \dfrac{\partial x(y)}{\partial y}+\dfrac{\partial \lambda (y)}{\partial y}\right) =-2\times \text {diag}(y_{\omega j}^i)\in {\mathbb {R}}^{m\times m} \end{aligned}$$

is a diagonal matrix with full rank, and $J_{12}=\text {diag}(e_{m_{\omega }^i})\in {\mathbb {R}}^{m\times nd}$ with $e_{m_{\omega }^i}=(1,\ldots ,1)^{\top }$ being a $m_{\omega }^i$-dimensional column vector and $\sum \limits _{i\in N}\sum \limits _{\omega \in \varOmega }m_{\omega }^i=m$. Hence, $J_{12}$ is of full-column rank. It has been proved that $\dfrac{\partial q^2}{\partial y}$ is a full-row rank matrix. Then, $J_1$ is of full-rank. This completes the proof. $\square$

B: The Compactness of $\varPhi$

In this section, we prove the compactness of $\varPhi$. This result is used in Corollary 3.1. Let $S_0$ be the set of all $(x,\lambda ,\mu ,t)$ satisfying the system (9). Now, we show that $S_0$ is bounded. Because $x_{\omega j}^i$ is a probability with $0\le x_{\omega j}^i\le 1$, $x_{\omega j}^i$ is bounded, where $j\in M_{\omega }^i$, $\omega \in \varOmega$ and $i\in N$. Let $u^{i+}:=(u^{i+}_{\omega })_{\omega \in \varOmega }$ and $\pi ^+:=(\pi ^+({\bar{\omega }}|\omega ))_{{\bar{\omega }},\omega \in \varOmega }$ with

$$\begin{aligned} u^{i+}_{\omega }:=\max \,u^i(\omega ,x_{\omega })\quad \text {and}\quad \pi ^+({\bar{\omega }}|\omega ):=\max \,\pi ({\bar{\omega }}|\omega ,x_{\omega }). \end{aligned}$$

Rewriting Eq. (7) in a vector form, we get that

$$\begin{aligned} \mu ^i\le (1-\zeta (t))(u^{i+}+\delta \pi ^+\mu ^i)-\zeta (t)\big (((x_{\omega }^i-x_{\omega }^{0,i})^{\top }x_{\omega }^i)_{\omega \in \varOmega }\big ). \end{aligned}$$

Because $(1-(1-\zeta (t))\delta \pi ^+)>0$, we get the boundedness of $\mu$ that

$$\begin{aligned} \mu ^i\le (1-(1-\zeta (t))\delta \pi ^+)^{-1}[(1-\zeta (t))u^{i+}-\zeta (t)\big (((x_{\omega }^i-x_{\omega }^{0,i})^{\top }x_{\omega }^i)_{\omega \in \varOmega }\big )]. \end{aligned}$$

Moreover, from the first group of equations of the system (9), the boundedness of $\lambda _{\omega j}^i$ is obtained subsequently. Let $X:=\{x\in {\mathbb {R}}^m_+\;:\; 0\le x\le 1\}$ and $\varDelta :=\{\lambda \in {\mathbb {R}}^m_+\;:\; \lambda _0\le \lambda \le \lambda _1 \}$, where $\lambda _0$ and $\lambda _1$ are the lower bound and upper bound of $\lambda$, respectively. Similarly, we define $\varLambda :=\{\mu \in \mathbb R^{nd}\;:\;\mu _0\le \mu \le \mu _1\}$, where $\mu _0$ and $\mu _1$ are the lower bound and upper bound of $\mu$, respectively. Hence, $S_0 \subset X\times \varDelta \times \varLambda \times [0,2]$ is a nonempty, convex and compact set. Then, the compactness of $\varPhi$ is established immediately. $\square$