On the Computation of Value Correspondences for Dynamic Games

Abstract

Recursive game theory provides theoretic procedures for computing the equilibrium payoff or value sets of repeated games and the equilibrium payoff or value correspondences of dynamic games. In this paper, we propose and implement outer and inner approximation methods for equilibrium value correspondences that naturally occur in the analysis of dynamic games. The procedure utilizes set-valued step functions. We provide an application to a bilateral insurance game with storage.

Appendix

The following algorithms provide the details of the computational procedure for the outer and inner monotone approximations of $B(V)(s,k)$ using hyperplanes. The correspondence $V(s,k)$ is a candidate for the equilibrium value correspondence.

______________________________________

Outer Approximation Algorithm

• 1. Fixed Inputs:

  • (a) Exogenous parameters.

  • (b) Partition $\mathcal {K} = \{K_q\}$.

  • (c) Search directions: ${\mathcal {R}}=\{r_1,\ldots , r_{\overline{\ell }_r} \}$, with each $\{r_{\ell }\in {\mathbb {R}}^I \ : \ ||r_{\ell }|| =1\}$.

  • (d) Convergence criterion $\varepsilon >0$.

• 2. First Iteration Inputs:

  • (a) Set of boundary points $\{v_{\ell }(s,q)|\ell = 1,\ldots , \overline{\ell }\}$ for each $(s,q)$.

  • (b) Set of corresponding normals: ${\mathcal {Z}}=\{z_{1},\ldots ,{z}_{\overline{\ell }}\}$. Together (a) and (b) define the initial outer approximation $V^{\mathbb {O}}(s,k)=\cup _{q} \omega ^{\mathbb {O}}_{s,q}(k)$ where

    $$\begin{aligned} \omega ^{\mathbb {O}}_{s,q}(k) = {\left\{ \begin{array}{ll} \cap _{\ell =1}^{\overline{\ell }}\{v\;|\;z_{\ell }\cdot v\le z_{\ell }\cdot v_\ell (s,q)=c_{\ell }(s,q)\}, &{} \text {if}\,k \in K_{q}, \\ \emptyset &{} \text {otherwise}. \end{array}\right. } \end{aligned}$$

    for all $v\in$ co $\cup _{k^{\prime }\in K_{q}}V(s,k^{\prime })$.

• 3. Optimization: Compute the new subgradient and boundary point sets that together represent an outer approximation to $B^L({V})$. For each $s \in S$, each partition element $K_{q}$ and each ${r_{\ell }}\in {{\mathcal {R}}}$

  $$\begin{aligned} c_{\ell }^{+}(s,q)&= {\max _{a,v^{\prime },k \in K_{q}}}\ r_{\ell }\cdot [(1-\beta )u(s,k,a)+ \beta \sum _{S} v^{\prime }(s^{\prime })\Pi (s^{\prime }|s)] \;\; \text {s.t.} \\&(i)\;v^{\prime }(s^{\prime })\in V^{\mathbb {O}}(s^{\prime },f(s,k,a)), \ a \in A(s,k), \nonumber \\&(ii)\;\text {for each } i \in \{1,\ldots ,I\}, \nonumber \\&\;\;\;\;\ (1-\beta )u^{i}(s,k,a)+\beta \sum _{S} v^{i\prime }(s^{\prime })\Pi (s^{\prime }|s)\ge \nonumber \\&\;\;\;\;\; \;\;\;\;\;\;\;\;\; \max _{a_{-i}\in A^{i}(s,k)} \min _{\tilde{v}^i(s^{\prime })\in V^{\mathbb {O}}(s^{\prime },f(s,k,a_{-i},a^{\prime }))} \ \ (1-\beta )u^{i}(s,k,a_{-i},a^{\prime })\nonumber \\ {}&+\,\, \beta \sum _{S}\tilde{v}^{i}(s^{\prime })\Pi (s^{\prime }|s). \nonumber \end{aligned}$$

  (6)

• 4. Output: $V^{+}(k,s)={\hat{B}}(V)(s,k)=\cup _{q}\omega _{s,q}^{{\mathbb {O}}^{+}}(k)$, where

  $$\begin{aligned} \omega _{s,q}^{{\mathbb {O}}^{+}}(k) = {\left\{ \begin{array}{ll} \cap _{\ell =1}^{\overline{\ell }_{r}}\{v\;|\;r_{\ell }\cdot v\le c_{\ell }^{+}(s,q)\} &{} \text {if}\,k \in K_{q}, \\ \emptyset &{} \text {otherwise.} \end{array}\right. } \end{aligned}$$

• 5. Check for convergence: Stop if $d_{V} ({V}^{+}, {V}) < \varepsilon$. Else set ${V}^{\mathbb {O}}={V}^{+}$, ${\mathcal {Z}}=R$, and $\overline{\ell } = \overline{\ell }_r$ and return to Step 3.

______________________________________

The main computational work is done in the optimization step. In this step, for each search direction $r_{\ell }$ and element of the partition $\mathcal {K}$, we find an action profile $a$, continuation values $v^{\prime }(s^{\prime }) \in V(s^{\prime },f(s,k,a))$, and current state $k$ in $K_{q}$ which makes $a$ incentive compatible and maximizes a weighted sum of player payoffs where the weights are given by $r_{\ell }$.⁹

In the outer approximation case, the set of search directions, ${{\mathcal {R}}}$, and the set of hyperplane normals for the approximation of $V(s,k)$, ${\mathcal {Z}}$, coincide after the initial iteration. Correspondences are represented by sets of bounding hyperplanes, one set for each block in the partition and each state $s$. In this way, an approximated correspondence can be reduced to a collection $\{c_{\ell }(s,q)\}$ in ${\mathbb {R}}^{SQI\overline{\ell }}$, with each $c_{\ell }(s,q)$ determining the position of the $\ell$-th hyperplane on the $q$-th block in state $s$.

For inner approximations, the set of hyperplane normals vary from iteration to iteration, depending on the number and position of the points on the convex hull of $V(s,k)$. The bounding hyperplanes are constructed from the line segments that connect the extreme points belonging to the convex hull of $V(s,k)$. In this case, an approximated correspondence cannot be reduced to a collection of $\{c_{\ell }(s,q)\}$ in ${\mathbb {R}}^{SQI\overline{\ell }}$ but can be reduced to a collection of $\{v_{\ell }(s,q)\}$ in $\cap _{k^{\prime }\in K_{q}}V(s,k^{\prime })$ for each $(s,q)$. This requires a slight modification to the output and updating used in the outer approximation algorithm. Below is the step by step description of the algorithm for computing an inner approximation to $B(V)$.

______________________________________

Inner Approximation Algorithm

• 1. Fixed Inputs:

  • (a) Exogenous parameters

  • (b) Partition ${\mathcal {K}} = \{K_q\}$.

  • (c) Search directions: ${{\mathcal {R}}}=\{r_1,\ldots , r_{\overline{\ell }_r} \}$, with each $\{r_{\ell } \in {\mathbb {R}}^I \ : \ ||r_{\ell }|| =1\}$.

  • (d) Convergence criterion: $\varepsilon >0$.

• 2. First Iteration Inputs:

  • (a) Set of points $\{v_{\ell }(s,q)|\ell = 1,\ldots , \overline{\ell }\}$ in $\cap _{k^{\prime }\in K_{q}}V(s,k^{\prime })$, for each $(s,q)$. These points define the initial inner approximation $V^{\mathbb {I}}(s,k)=\cap _{q} \omega ^{\mathbb {I}}_{s,q}(k)$ where

    $$\begin{aligned} \omega ^{\mathbb {I}}_{s,q}(k) = {\left\{ \begin{array}{ll} \text {co} \ \{v_{\ell }(s,q)|\ell = 1,\ldots , \overline{\ell }\} &{} \text {if}\,k \in K_{q}, \\ {\mathbb {R}}^{I} &{} \text {otherwise}. \end{array}\right. } \end{aligned}$$

• 3. Optimization: Compute the new sets of point from which the convex hulls that together represent an inner approximation to $B^L({V})$ are constructed. For each $s \in S$, each partition element $K_{q}$ and each ${r_{\ell }}\in {{\mathcal {R}}}$

  $$\begin{aligned} \{a,v^{\prime },k\}_{\ell }^{+}(s,q)&= \mathop {{{\mathrm{argmax}}}}\limits _{a,v^{\prime },k \in K_{q}}\;\;\; r_{\ell }\cdot [(1-\beta )u(s,k,a)+ \beta \sum _{S} v^{\prime }(s^{\prime })\Pi (s^{\prime }|s)] \ \text {s.t} \\&(i)\;v^{\prime }(s^{\prime })\in V^{\mathbb {I}}(s^{\prime },f(s,k,a)), \ a \in A(s,k), \nonumber \\&(ii)\;\text {for each } i \in \{1,\ldots ,I\}, \nonumber \\&\;\;\;\;\ (1-\beta )u^{i}(s,k,a)+\beta \sum _{S} v^{i\prime }(s^{\prime })\Pi (s^{\prime }|s)\ge \nonumber \\&\;\;\;\;\; \ \max _{a_{-i}\in A^{i}(s,k)} \min _{\tilde{v}^i(s^{\prime })\in V^{\mathbb {I}}(s^{\prime },f(s,k,a_{-i},a^{\prime }))} \ (1-\beta )u^{i}(s,k,a_{-i},a^{\prime })\nonumber \\&\quad +\, \beta \sum _{S}\tilde{v}^{i}(s^{\prime })\Pi (s^{\prime }|s). \nonumber \end{aligned}$$

  (7)

• 4. Output: $V^{+}(k,s)={\hat{B}}(V)(s,k)=\cap _{q}\omega _{s,q}^{{\mathbb {I}}^{+}}(k)$, with

  $$\begin{aligned} \omega _{s,q}^{{\mathbb {I}}^{+}}(k) = {\left\{ \begin{array}{ll} \text {co\ } \{v^{+}_{\ell }(s,q)|\ell = 1,\ldots , \overline{\ell }\} &{} \text {if}\,k \in K_{q}, \\ {\mathbb {R}}^{I} &{} \text {otherwise.} \end{array}\right. } \end{aligned}$$

  where $v^{+}_{\ell }(s,q)$ is equal to $[(1-\beta )u(s,k,a)+ \beta \sum _{S} v^{\prime }(s^{\prime })\Pi (s^{\prime }|s)]$ evaluated at $\{a,v^{\prime },k\}_{\ell }^{+}(s,q)$.

• 5. Check for convergence: Stop if $d_V ({V}^{+}, {V}) < \varepsilon$. Else set ${V}^{\mathbb {I}}(s,k)={V}^{+}(s,k)$ and go back to Step 3.

See Judd et al. [12] and Yeltekin et al. [16] for the details of constructing convex hulls similar to those required in Step 4.