Continuous-Time Stochastic Games of Fixed Duration

Abstract

We study nonzero-sum continuous-time stochastic games, also known as continuous-time Markov games, of fixed duration. We concentrate on Markovian strategies. We show by way of example that equilibria need not exist in Markovian strategies, but they always exist in Markovian public-signal correlated strategies. To do so, we develop criteria for a strategy profile to be an equilibrium via differential inclusions, both directly and also by modeling continuous-time stochastic as differential games and using the Hamilton–Jacobi–Bellman equations. We also give an interpretation of equilibria in mixed strategies in continuous time and show that approximate equilibria always exist.

Appendix: Proofs Omitted from Sect. 5

Proof of Lemma 2

Note first that ∥A⋅B∥_∞≤C _∞∥A∥_∞⋅∥B∥_∞ for an appropriate C _∞>0 and for any two |Z|×|Z| matrices A,B.¹⁹ Further note that ∥P _u ∥_∞≤1 for any Markovian strategy u. Using these observations and (2.1) gives:

Therefore, an application of Gronwall’s inequality (see, e.g., Hirsch and Smale [18]) gives (5.6) for appropriate M>0. To prove (5.7), note that (3.1) shows that for each player $p \in \mathcal {P}$ and z∈Z,

and so enlarging M appropriately completes the proof of (5.7). □

Proof of Lemma 3

For the simplicity of notation, we will prove only (a); (b) is proved by an almost identical argument, albeit with more cumbersome notation. Let $\mathcal {T}= \{0 = t_{0} < \cdots< t_{k} < T \}$, fix τ satisfying $\ell(\mathcal {T}) < \tau\leq\min[T,2\ell(\mathcal {T})]$, and let 0=m ₀<m ₁<⋯<m _n =k+1 (recall that t _k+1=T) be such that $\tau< t_{m_{j}} - t_{m_{j-1}} < 2\tau$ for all 1≤j<n; such a selection is possible. Denote

$$D_j := \sup_{t \in\{ t_0, \dots, t_{m_j} \}} \bigl\Vert P_u(t) - P_w(t)\bigr\Vert _\infty. $$

Clearly D ₀=0. We wish to bound D _n . Fix 1≤j<n and let $\mathcal {T}|_{j} = \{ t_{0},\ldots,t_{m_{j} - 1} \}$ and $\mathcal {T}|_{j+} = \{t_{m_{j}},\ldots,t_{m_{j+1} - 1}\}$; these can be viewed as restrictions of the partition to $[0,t_{m_{j}}]$ and to $[t_{m_{j}},t_{m_{j+1}}]$, respectively. We will denote by v| _j , v| _j+ generic elements of the sets $S(\mathcal {T}|_{j}) \cong\prod_{p \in \mathcal {P}}(I^{p})^{Z \times \mathcal {T}|_{j}}$, $S(\mathcal {T}|_{j+}) \cong\prod_{p \in \mathcal {P}}(I^{p})^{Z \times \mathcal {T}|_{j+}}$, respectively. Note that each $w \in \varSigma(\mathcal {T})$ induces elements of $\varSigma(\mathcal {T}|_{j}) \cong\prod_{p \in \mathcal {P}}(\Delta(I^{p}))^{Z \times \mathcal {T}|_{j}}$, $\varSigma(\mathcal {T}|_{j+})\cong \prod_{p \in \mathcal {P}}(\Delta(I^{p}))^{Z \times \mathcal {T}|_{j+}}$ in the natural way, and this induced distribution is a product distribution on $\varSigma(\mathcal {T}|_{j+1}) = \varSigma(\mathcal {T}|_{j}) \otimes\varSigma(\mathcal {T}|_{j+})$. Note that for $t \leq t_{m_{j}}$, $P_{\hat{v}|_{j}}(t)$ is well defined,²⁰ and for $t_{m_{j}} \leq t < t_{m_{j+1}}$ and any z∈Z, $\hat{v}|_{j+}(z,t)$ is well defined, see (5.1). Also observe that by the definition of w, for any $t_{m_{j}} \leq p<q < t_{m_{j+1}}$, we have, for all z∈Z,

$$ \int _{t_p}^{t_q} \mu\bigl(u(z,s)\bigr)\,ds = \sum _{v|_{j+} \in \mathcal {T}|_{j+}} \int_{t_p}^{t_q} w[v|_{j+}]\cdot\mu\bigl(\hat{v}|_{j+}(z,s)\bigr)\,ds, $$

(10.1)

where we recall that a pure strategy $v \in S(\mathcal {T})$ induces a pure Markovian strategy by (5.1). Finally, we note that for any Markovian strategy ϕ, by (2.1),

$$ \bigl\Vert P_\phi(t) - P_\phi (s)\bigr\Vert _\infty\leq \int_s^t \bigl\Vert P_\phi(s)\mu\bigl(\phi(s)\bigr)\bigr\Vert _\infty \,ds \leq C_\infty \Vert \mu\Vert _\infty|t-s|, $$

(10.2)

where C _∞ is as in the proof of Lemma 2. Hence, letting m _j ≤p<m _j+1 and denoting $\tau_{0} = t_{m_{j}}$, τ=t _p −τ ₀, we have

Therefore,

(10.3)

(10.4)

By (10.2), the first two terms together are bounded by C _∞(∥μ∥_∞)² τ ². As for the second term, note that for 0≤s≤τ,

Using (10.1), we have

(10.5)

Hence, (10.3) and (10.5), together with $\tau< t_{m_{j+1}} - t_{m_{j}} \leq2\ell(\mathcal {T})$ for all 0≤j<n, imply

$$D_{j+1} \leq D_j + 4 C_\infty\bigl(\Vert \mu\Vert _\infty\bigr)^2 \bigl(\ell(\mathcal {T})\bigr)^2 + 2 D_j \cdot \ell(\mathcal {T}) \cdot C_\infty\Vert \mu\Vert _\infty $$

with initial condition D ₀=0. Denoting $A = 1 + 2 \ell(\mathcal {T}) \cdot C_{\infty}\Vert \mu\Vert _{\infty}\geq1$ and $B = 4 C_{\infty}(\Vert \mu\Vert _{\infty})^{2} (\ell(\mathcal {T}))^{2} > 0$, we see that D _j+1≤A⋅D _j +B with D ₀=0, and therefore,

$$D_n \leq B \cdot\bigl(1 + A + \cdots+ A^{n-1}\bigr) = B \frac{A^{n} - 1}{A - 1}. $$

We have $n \cdot\ell(\mathcal {T}) < T$, where recall that $t_{m_{n}} = T$, and therefore $n < \frac{T}{\ell(\mathcal {T})}$. Hence,

$$A^n = \bigl(1 + \ell(\mathcal {T}) \cdot C_\infty\Vert \mu\Vert _\infty\bigr)^{n} \leq\bigl(1 + 2\cdot\ell(\mathcal {T}) C_\infty\Vert \mu\Vert _\infty\bigr)^{\frac{T}{\ell(\mathcal {T})}} \leq e^{2 T C_\infty\Vert \mu\Vert _\infty}. $$

Since $A - 1 = \ell(\mathcal {T}) \cdot C_{\infty}\Vert \mu\Vert _{\infty}$, we have for some C′>0 independent of $\ell(\mathcal {T})$,

$$\ell(\mathcal {T}) \frac{A^{n} - 1}{A - 1} \leq C', $$

and therefore,

$$D_n \leq\frac{B}{\ell(\mathcal {T})}C' = 4 C_\infty\bigl(\Vert \mu\Vert _\infty\bigl)^2 C' \ell (\mathcal {T}) \leq8 C_\infty\bigl(\Vert \mu\Vert _\infty\bigr)^2 C' \ell( \mathcal {T}); $$

this gives, denoting C=8C _∞(∥μ∥_∞)² C′,

$$ D_n \leq C\cdot\ell( \mathcal {T}). $$

(10.6)

Finally, from (10.2) we have

which completes the proof of (5.8), with K=C+4C _∞∥μ∥_∞. Now we prove (5.9). Fix 0≤j<n and denote $\tau= t_{m_{j+1}} - t_{m_{j}}$. Using (5.8) and techniques similar to those above, we have

Summing over j=0,…,n−1 and recalling that $n < \frac{T}{\ell(\mathcal {T})}$, $\tau\leq2\ell(\mathcal {T})$ gives

Enlarging K accordingly gives (5.9). □

Proof of Lemma 4

First, we prove

$$\sup_{u^p \in \mathfrak {A}^p} \gamma^p_{u^p,w^{-p}}(z) \leq \sup_{w^p \in \varSigma^p(\mathcal {T})} \gamma^p_{w^p,w^{-p}}(z) + C \cdot\ell(\mathcal {T}) $$

for appropriate C>0. Given $w^{p} \in\varSigma^{p}(\mathcal {T})$, set $u = \hat{w}$; by parts (a) and (b) of Lemma 3,

Next we prove, that for all z∈Z,

$$\sup_{w^p \in\varSigma^p(\mathcal {T})} \gamma^p_{w^p,w^{-p}}(z) \leq \sup_{u^p \in \mathfrak {A}^p} \gamma^p_{u^p,w^{-p}}(z) + C \cdot\ell(\mathcal {T}) $$

for appropriate C>0. We will actually prove a bit more. We show that it suffices to take only pure Markovian strategies in the supremum on the right-hand side. Fix $w^{p} \in\varSigma^{p}(\mathcal {T}),w^{-p} \in \varSigma^{-p}(\mathcal {T})$. It suffices to show that there is a pure Markovian strategy u ^p satisfying

$$ \bigl\Vert \gamma^p_{w^p,w^{-p}} - \gamma^p_{u^p,w^{-p}}\bigr\Vert _\infty \leq C \cdot \ell(\mathcal {T}) $$

(10.7)

for some C>0 that is independent of w ^p or of $\mathcal {T}$. For each interval J=[t _j ,t _j+1] induced by $\mathcal {T}$, divide J into subintervals (some of which may be degenerate) $I_{1},\ldots,I_{|I^{p}|}$, with ℓ(I _a )=ℓ(J)⋅w ^p(t _j )[a], where ℓ(⋅) will be used to denote the length of an interval, and a denotes the ath element of I ^p in some ordering. Then define u ^p(t)[a]=1 if t∈I _a . Note then that by the definition of u ^p and denoting w=(w ^p,w ^−p), we have for any interval J=[τ ₀,τ ₁] in the partition induced by $\mathcal {T}$,

(10.8)

Note also that w ^p is an $\mathcal {T}$-adapted version of u ^p. Combining (5.8) and (5.10) of Lemma 3 gives

(10.9)

for K as in Lemma 3. As a result, for each interval J=[τ ₀,τ ₁] induced by the partition $\mathcal {T}$, we have, using (10.2) and techniques similar to those used in the proof of Lemma 3,

where the last inequality is because of (10.9) and (10.8). Denoting L=|Z|⋅(2C _∞∥μ∥_∞+1)⋅∥r∥_∞ and summing over all J induced by the partition induced by $\mathcal {T}$ gives

where we have used the following elementary claim: For any T,D>0,

$$\max_{\{\sum_{i=1}^n a_i = T, \forall i, 0 \leq a_i \leq D\}} \sum_{i=1}^n a_i^2 \leq\biggl\lceil\frac{T}{D} \biggr\rceil \cdot D^2 \leq T\cdot D + D^2, $$

where ⌈⋅⌉ denotes the rounding-up function. Indeed, in our case, $\ell(J) \leq\ell(\mathcal {T})$ for each J induced by $\mathcal {T}$. □