On Canonical Forms for Zero-Sum Stochastic Mean Payoff Games

Abstract

We consider two-person zero-sum mean payoff undiscounted stochastic games and obtain sufficient conditions for the existence of a saddle point in uniformly optimal stationary strategies. Namely, these conditions enable us to bring the game, by applying potential transformations, to a canonical form in which locally optimal strategies are globally optimal, and hence the value for every initial position and the optimal strategies of both players can be obtained by playing the local game at each state. We show that these conditions hold for the class of additive transition (AT) games, that is, the special case when the transitions at each state can be decomposed into two parts, each controlled completely by one of the two players. An important special case of AT-games form the so-called BWR-games which are played by two players on a directed graph with positions of three types: Black, White and Random. We give an independent proof for the existence of a canonical form in such games, and use this result to derive the existence of a canonical form (and hence, of a saddle point in uniformly optimal stationary strategies) in a wide class of games, which includes stochastic games with perfect information (PI), switching controller (SC) games and additive rewards, additive transition (ARAT) games. Unlike the proof for AT-games, our proof for the BWR-case does not rely on the existence of a saddle point in stationary strategies. We also derive some algorithmic consequences from these our reductions to BWR-games, in terms of solving PI-, and ARAT-games in sub-exponential time.

Appendices

Appendix A: Related Results from Theory of Markov Chains

Given a n×n transition matrix P, the Cesáro partial sums $\frac{1}{k+1} \sum_{i=1}^{k} P^{i}$ converge, as k→∞, to the limit Markov matrix Q such that:

• (i) PQ=QP=QQ=Q;

• (ii) $\operatorname {rank}(I - P) + \operatorname {rank}Q = n$.

• (iii) For each n-vector c system Px=x, Qx=c has a unique solution.

• (iv) matrix I−(P−Q) is nonsingular and

  $$ H(\delta ) = \sum_{i = 0}^\infty \delta ^i \bigl(P^i - Q\bigr) \rightarrow H = \bigl(I - (P - Q)\bigr)^{-1} - Q \quad \mbox{as } \delta \rightarrow1^-. $$

  (62)
• (v)
  $$H(\delta ) Q = Q H(\delta ) = H Q = Q H = 0 \quad \mbox{and} \quad (I - P) H = H (I - P) = I -Q. $$

Claim (iv) (which is used in Sect. 7.4) was proved in 1962 by Blackwell [4], while for the remaining four claims, he cited the text-book in finite Markov chains by Kemeny and Snell [28] (that was published, in fact, one year later, in 1963).

Appendix B: Proof of Lemma 3

Let us first fix the strategy $\bar{\beta}$ of Black, and compute the uniformly best response by White by solving a controlled Markov chain problem (see, e.g., [35]). It is well-known that this can be done by solving a linear program whose dual LP provides us with a potential vector y∈ℝ^V such that

$$ g^v\geq \mathrm {Val}\bigl(A^v(y)\bigr) $$

(63)

holds for all states v∈V. Let us next fix $\bar{\alpha}$ and compute similarly the best response of Black, providing analogously a potential vector z∈ℝ^V satisfying

$$ g^v\leq \mathrm {Val}\bigl(A^v(z)\bigr) $$

(64)

for all states v∈V. Since adding a constant to a potential vector does not change the potential transformation and the value matrices, we can assume w.l.o.g. that

$$ y\leq z. $$

(65)

Let us define a matrix valued mapping B ^v(d) for all states v∈V and vectors d∈ℝ^V by

$$B^v(d) = A^v(d)-d^vJ_{|K^v|\times|L^v|}. $$

Then we have by (65) that B ^v(z)≤B ^v(y) (componentwise), and since the value function of matrix games is monotone we can conclude by (63) and (64), and by the fact that changing the payoff matrix by a constant changes the value of the game by the same constant that

$$ g^v-z^v \leq \mathrm {Val}\bigl(B^v(z) \bigr)\leq \mathrm {Val}\bigl(B^v(y)\bigr)\leq g^v-y^v, $$

(66)

for all states v∈V. Note that if g−z≤g−d≤g−y, then we have B ^v(z)≤B ^v(d)≤B ^v(y) for all v∈V, and hence by the above cited properties of the value function and by (66)

$$ \mathrm {Val}\bigl(B^v(z)\bigr)\leq \mathrm {Val}\bigl(B^v(d) \bigr) \leq \mathrm {Val}\bigl(B^v(y)\bigr) \quad \text{for all states } v\in V. $$

(67)

Since the mapping F:g−d↦Val(B(d)) (where Val(B(d))=(Val(B ^v(d)) : v∈V)) is Lipschitz-continuous and since by property (67) and (66) it maps the compact box [g−z,g−y] into a subset of itself, we can conclude by Brouwer’s theorem that F has a fixed point, that is there exists a potential vector x∈[y,z] (i.e., g−x∈[g−z,g−y]) for which g−x=F(g−x)=Val(B(x)). This implies that g=Val(A(x)), completing our proof. □

Appendix C: Proof of the Implication $\mathrm{(B2)}\Rightarrow\mathrm{(A1)}$

Let g,x,y∈ℝ^V be the vectors satisfying condition (B2). Then there exist strategies and such that, for all states v∈V, the following hold: (1) $\bar{\alpha}^{v} G^{v}(g)\beta\ge g^{v}$ and $\bar{\alpha}^{v} A^{v}(x)\beta \ge g^{v}$ for all β∈Δ(L ^v), and (2) $\alpha^{v} G^{v}(g)\bar{\beta}^{v}\le g^{v}$ and $\alpha^{v} A^{v}(y)\bar{\beta}^{v}\le g^{v}$ for all α∈Δ(K ^v).

Fix a starting position v ₀=w. It is enough to show that White can guarantee at least g ^w while Black can guarantee at most g ^w. We only show the former statement since the latter can be shown similarly. At time i, we will let White play his/her locally optimal (stationary) strategy $\bar{\alpha}^{v}$ whenever (s)he is at position v _i =v, while Black chooses an arbitrary, not necessarily stationary, strategy , where is the history of the play leading to v _i =v and is the set of all such histories. Let us note that and denote by β ^v,i∈Δ(L ^v) the Markovian strategy given by .

Consider a play w=v ₀,v ₁,v ₂,… (where each v _i is a random variable). By (7) and the fact that potential transformations do not change the Cesáro sum (Sect. 2.4), it is enough to show that $\mathbb {E}[b_{i}(x)]\ge g^{w}$ for all i. Note that

(68)

We prove by induction on i=0,1,2,… , that ∑ _v g ^v⋅Pr[v _i =v]≥g ^w, which will imply the lemma by (68). Indeed, the statement is trivially true for i=0. For any i, we have

and the latter is at least g ^w by the induction hypothesis. □

Appendix D: Some Examples

Example 2

Vrieze [46, Chap. 8] showed an example, see Fig. 2, for a stochastic game which has values and uniformly optimal stationary strategies, and which has no canonical form. We can see that condition (A2) is violated. In this game we have V={1,2,3}, states 2 and 3 are absorbing with |K ²|=|K ³|=|L ²|=|L ³|=1, while in state 1 we have |K ¹|=|L ¹|=3. The reward matrix of state one is shown in Fig. 2 together with the transition probabilities which are all zero or one.

Fig. 2

In this game Γ, we have |V|=3 states, |K ¹|=|L ¹|=3, |K ²|=|L ²|=1, and |K ³|=|L ³|=1. The numbers in the state matrices are the local rewards. All transition probabilities are zero or one, and arcs in the picture indicate the probability 1 transitions. The thick arc in the picture indicates that for pairs of strategies from the top left 2×2 area in state 1 the game remains in state 1 with probability 1. This game has values and uniformly optimal stationary strategies, but it does not have a canonical form

This game has values, g=(0,−1,1), and unique uniformly optimal stationary strategies, namely $\alpha^{1}=(\frac{1}{2},\frac{1}{2},0)$ and $\beta^{1}=(\frac{1}{2},\frac{1}{2},0)$, and the trivial strategies in states 2 and 3. We have

$$G^1 = \left ( \begin{array}{c@{\quad }c@{\quad }c}0&0&1\\0&0&-1\\1&-1&0 \end{array} \right ). $$

For a potential vector x∈ℝ^V we can assume w.l.o.g. that x ₁=0, and thus we have

$$A^1(x) = \left ( \begin{array}{c@{\quad }c@{\quad }c}1&-1&-1-x_3\\-1&1&-1-x_2\\1-x_3&1-x_2&0 \end{array} \right ). $$

Here, $\bar {K}^{1}=\{(\frac{1}{2},\frac{1}{2},0),(1,0,0)\}$, $\bar {L}^{1}=\{(\frac{1}{2},\frac{1}{2},0),(0,1,0)\}$, and only the first vectors are optimal in the matrix game with payoffs A ¹(x) (for any potential transformation), and thus α ¹ and β ¹ given above are the unique optimal strategies. For the canonical form for some potential vector x∈ℝ^V (x ₁=0), we would need the inequalities that α ¹ A ¹(x)≥0 and A ¹(x)β ¹≤0, implying that $-1-\frac{x_{2}+x_{3}}{2}\geq0$ and $1-\frac{x_{2}+x_{3}}{2}\leq0$, leading to a contradiction. Consequently, this example does not have a canonical form.

Example 3

Raghavan, Tijs, and Vrieze [40] showed an example, see Fig. 3, for an AT game, with a rational input data, in which the optimal values and strategies are irrational. This example is ergodic, with states V={1,2} and values $g^{1}=g^{2}=-(6-\sqrt{30})^{2}$. The vector $x=(0,22-4\sqrt{30})$ is a potential transformation providing the canonical form for this example. We have K ¹=K ²=L ¹=L ²={1,2}, and the strategies $\alpha ^{1}=\beta^{1}=(\frac{-4+\sqrt{30}}{2},\frac{6-\sqrt{30}}{2})$, and $\alpha^{2}=\beta^{2}=(\frac{-9+2\sqrt{30}}{3},\frac{12-2\sqrt {30}}{3})$ are the uniformly optimal stationary strategies.

Fig. 3

This example has two states with |K ¹|=|L ¹|=|K ²|=|L ²|=2. In each cell in the figure, we have the reward in the top left corner, while the transition probabilities to states 1 and 2 (in this order) are in bottom right area. This is an ergodic AT-game with irrational values and optimal strategies

Appendix E: Summary of Implications

Figure 4 summarizes the implications between game properties considered in the paper.

Fig. 4

Summary of the results and considered game classes. Most implications are by Theorem 1. Example 2 shows that property (B2) does not imply (B3). Theorem 3 claims that properties (A2) and (B6) together imply (B4), which is equivalent with the existence of the canonical form (B1). Dotted lines indicate subclass relations