Optimizing over pure stationary equilibria in consensus stopping games

Abstract

Consensus decision-making, a widely utilized group decision-making process, requires the consent of all participants. We consider consensus stopping games, a class of stochastic games arising in the context of consensus decision-making that require the consent of all players to terminate the game. We show that a consensus stopping game may have many pure stationary equilibria, which in turn raises the question of equilibrium selection. Given an objective criterion, we study the NP-hard problem of finding a best pure stationary equilibrium. We characterize the pure stationary equilibria, show that they form an independence system, and develop several families of valid inequalities. We then solve the equilibrium selection problem as a mixed-integer linear program by a branch-and-cut approach. Our computational results demonstrate the effectiveness of our approach over a commercial solver.

Appendix

Proof of Proposition 1

(iii) $(\Rightarrow )$ Suppose that there exists an equilibrium in $A_{\mathbf {x}}$, denoted by $\tilde{\mathbf {a}}$. We show that strategy profile $\bar{\mathbf {a}}$ is an equilibrium as well. Since $\bar{\mathbf {a}},\tilde{\mathbf {a}}\in A_{\mathbf {x}}$, we must have $\mathbf {w}^{\tilde{\mathbf {a}}}=\mathbf {w}^{\bar{\mathbf {a}}}$ by part (i). There are two cases:

If $x(s)=1$, then for all $i \in \mathcal {N}$, $\tilde{a}_i(s)=\bar{a}_i(s)=1$, and

$$\begin{aligned} w_i^{\bar{\mathbf {a}}}(s)=w_i^{\tilde{\mathbf {a}}}(s)&=\max \left\{ \left( \prod \limits _{\begin{array}{c} j\in \mathcal {N},\\ j\ne i \end{array}}\tilde{a}_j(s)\right) u_i(s,1)\right. \nonumber \\&\qquad \left. +\,\left( 1-\prod \limits _{\begin{array}{c} j\in \mathcal {N},\\ j\ne i \end{array}}\tilde{a}_j(s)\right) F_i(s,\mathbf {w}_i^{\tilde{\mathbf {a}}}),F_i(s,\mathbf {w}_i^{\tilde{\mathbf {a}}})\right\} \\&=\max \left\{ \left( \prod \limits _{\begin{array}{c} j\in \mathcal {N},\\ j\ne i \end{array}}\bar{a}_j(s)\right) u_i(s,1)\right. \nonumber \\&\qquad \left. +\,\left( 1-\prod \limits _{\begin{array}{c} j\in \mathcal {N},\\ j\ne i \end{array}}\bar{a}_j(s)\right) F_i(s,\mathbf {w}_i^{\bar{\mathbf {a}}}),F_i(s,\mathbf {w}_i^{\bar{\mathbf {a}}})\right\} , \end{aligned}$$

where the second equality follows from the fact that $\tilde{\mathbf {a}}$ is an equilibrium and part (ii).

If $x(s)=0$, then for all $i \in \mathcal {N}$, $\bar{a}_i(s)=0$, and

$$\begin{aligned} w_i^{\bar{\mathbf {a}}}(s)=F_i(s,\mathbf {w}_i^{\bar{\mathbf {a}}})&= \max \left\{ \left( \prod \limits _{\begin{array}{c} j\in \mathcal {N},\\ j\ne i \end{array}}\bar{a}_j(s)\right) u_i(s,1)\right. \nonumber \\&\quad \left. +\,\left( 1-\prod \limits _{\begin{array}{c} j\in \mathcal {N},\\ j\ne i \end{array}}\bar{a}_j(s)\right) F_i(s,\mathbf {w}_i^{\bar{\mathbf {a}}}),F_i(s,\mathbf {w}_i^{\bar{\mathbf {a}}})\right\} , \end{aligned}$$

where the first equality follows from part (i).

In summary, the Eq. (3) is satisfied for all $s\in \mathscr {S}, i\in \mathcal {N}$. Therefore, $\bar{\mathbf {a}}$ is an equilibrium by part (ii).

$(\Leftarrow )$ Follows directly from the definitions of $\bar{\mathbf {a}}$ and $A_{\mathbf {x}}$. $\square$

Proof of Proposition 3

Recall that under a fixed strategy profile $\mathbf {x}$, the payoffs of each player represent a stationary Markov reward process, and hence can be calculated by value iteration [10]. Let $[w^{\mathbf {x}}_{i}(s)]^n$ denote the value associated with state $s\in \mathscr {S}$ at the nth iteration of the value iteration algorithm under strategy profile $\mathbf {x}$ for player i. Furthermore, we initialize our value iteration with payoffs of player i under $\bar{\mathbf {x}}$, i.e., $[w^{\mathbf {x}}_{i}(s)]^0=w_{i}^{\bar{\mathbf {x}}}(s)$ for all $s \in \mathscr {S}$.

(i) From Proposition 2 (i) and the hypothesis about the relation between $\mathbf {x}$ and $\bar{\mathbf {x}}$, there are four mutually exclusive cases for s:

• If $s \in \mathscr {S}^1_{i,nv}(\bar{\mathbf {x}})$, then $x(s)=1$ and $[w^{\mathbf {x}}_{i}(s)]^1=u_{i}(s,1)=w_{i}^{\bar{\mathbf {x}}}(s)=[w^{\mathbf {x}}_{i}(s)]^0$.

• If $s \in \mathscr {S}^0(\bar{\mathbf {x}})$, then $x(s)=0$ and $[w^{\mathbf {x}}_{i}(s)]^1=F_i(s,[\mathbf {w}^{\mathbf {x}}_{i}]^0)=F_i(s,\mathbf {w}^{\bar{\mathbf {x}}}_{i})=w_{i}^{\bar{\mathbf {x}}}(s)=[w^{\mathbf {x}}_{i}(s)]^0$.

• If $s \in \mathscr {S}^1_{i,v}(\bar{\mathbf {x}})$ and $x(s)=1$, then $[w^{\mathbf {x}}_{i}(s)]^1=u_{i}(s,1)=w^{\bar{\mathbf {x}}}_{i}(s)=[w^{\mathbf {x}}_{i}(s)]^0$.

• If $s \in \mathscr {S}^1_{i,v}(\bar{\mathbf {x}})$ and $x(s)=0$, then $[w^{\mathbf {x}}_{i}(s)]^1=F_i(s,[\mathbf {w}^{\mathbf {x}}_{i}]^0)=F_i(s,\mathbf {w}^{\bar{\mathbf {x}}}_{i})\ge u_{i}(s,1)=w^{\bar{\mathbf {x}}}_{i}(s)=[w^{\mathbf {x}}_{i}(s)]^0$ where the inequality follows from $s \in \mathscr {S}^1_{i,v}(\bar{\mathbf {x}})$.

From all four cases, it follows that $[w^{\mathbf {x}}_{i}(s)]^1 \ge [w^{\mathbf {x}}_{i}(s)]^0$ for all $s \in \mathscr {S}$. From the monotonicity of the dynamic programming operator induced by strategy profile $\mathbf {x}$ for player i [6], it follows that for any n, $[w^{\mathbf {x}}_{i}(s)]^{n+1} \ge [w^{\mathbf {x}}_{i}(s)]^n$ for all $s \in \mathscr {S}$. As a result, $w^{\mathbf {x}}_{i}(s)= \lim \limits _{n \rightarrow \infty }[w^{\mathbf {x}}_{i}(s)]^n \ge [w^{\mathbf {x}}_{i}(s)]^0=w^{\bar{\mathbf {x}}}_{i}(s)$.

(ii) The proof is similar to that of part (i). $\square$

Remark 1

A nice property of Proposition 2 (ii) is that when we assess equilibrium conditions for a strategy profile $\mathbf {x}$, we only need to check if the Bellman–Shapley equation (5) is satisfied for each $s\in \mathscr {S}, i\in \mathcal {N}$ in which $x(s)=1$ since (5) is trivially satisfied for each $s\in \mathscr {S}, i\in \mathcal {N}$ in which $x(s)=0$. In particular, this implies that the strategy profile $\mathbf {0}$ is an equilibrium.

Proof of Lemma 1

$(\Rightarrow )$ If $s \in \mathscr {S}^1(\mathbf {x})$, then $w_i^{\mathbf {x}}(s)=u_i(s,1)\ge F_i(s,\mathbf {w}^{\mathbf {x}}_i)$ by Proposition 2. Therefore, $u_i(s,1) \ge F_i(s,\mathbf {w}^{\mathbf {x}}_i)$ for all $s \in \mathscr {S}^1(\mathbf {x}), i \in \mathcal {N}$. It follows that $\mathscr {S}^1_{i,v}(\mathbf {x})= \emptyset$ for all $i \in \mathcal {N}$.

$(\Leftarrow )$$\mathscr {S}^1(\mathbf {x})=\mathscr {S}^1_{i,nv}(\mathbf {x}) \cup \mathscr {S}^1_{i,v}(\mathbf {x})$ and $\mathscr {S}^1_{i,v}(\mathbf {x})= \emptyset$ for all $i \in \mathcal {N}$. It follows that $\mathscr {S}^1(\mathbf {x})=\mathscr {S}^1_{i,nv}(\mathbf {x})$ for all $i \in \mathcal {N}$. Therefore, if $s \in \mathscr {S}^1(\mathbf {x})$, then for all $i \in \mathcal {N}$, $u_i(s,1) \ge F_i(s,\mathbf {w}^{\mathbf {x}}_i)$ and Proposition 2 (i) implies that $w_i^{\mathbf {x}}(s)=u_i(s,1)$. Hence, the Bellman–Shapley equation (5) is satisfied for all $s \in \mathscr {S}^1(\mathbf {x}),i \in \mathcal {N}$. As noted in Remark 1, the Bellman–Shapley equation (5) is trivially satisfied for all $s \in \mathscr {S}^0(\mathbf {x}),i \in \mathcal {N}$. Consequently, $\mathbf {x}$ is an equilibrium by Proposition 2 (ii). $\square$

Proof of Proposition 4

(i) As $\bar{\mathbf {x}}$ is an equilibrium, $\mathscr {S}^1_{i,v}(\bar{\mathbf {x}})= \emptyset$ for all $i \in \mathcal {N}$ by Lemma 1, and the hypothesis states that $\mathscr {S}^1(\mathbf {x})\subseteq \mathscr {S}^1(\bar{\mathbf {x}})$. Therefore, $\mathscr {S}^1_{i,v}(\bar{\mathbf {x}}) \subseteq \mathscr {S}^1(\mathbf {x})\subseteq \mathscr {S}^1(\bar{\mathbf {x}})$ for all $i \in \mathcal {N}$. The result follows from Proposition 3 (ii).

(ii) If $s \in \mathscr {S}^1(\mathbf {x})$, then $s \in \mathscr {S}^1(\bar{\mathbf {x}})=\mathscr {S}^1_{i,nv}(\bar{\mathbf {x}}) \cup \mathscr {S}^1_{i,v}(\bar{\mathbf {x}})=\mathscr {S}^1_{i,nv}(\bar{\mathbf {x}})$ for all $i \in \mathcal {N}$ since $\mathscr {S}^1_{i,v}(\bar{\mathbf {x}})=\emptyset$ for all $i \in \mathcal {N}$ by Lemma 1. This implies that $u_i(s,1) \ge F_i(s,\mathbf {w}^{\bar{\mathbf {x}}}_i)$ for all $s \in \mathscr {S}^1(\mathbf {x}), i \in \mathcal {N}$. Moreover, $w_{i}^{\bar{\mathbf {x}}}(s) \ge w_{i}^{\mathbf {x}}(s)$ for all $s \in \mathscr {S}, i \in \mathcal {N}$ by part (i), and hence by monotonicity of $F_i(s,\mathbf {v})$ with respect to $\mathbf {v}$, it follows that $F_i(s,\mathbf {w}^{\bar{\mathbf {x}}}_i) \ge F_i(s,\mathbf {w}^{\mathbf {x}}_i)$ for all $s \in \mathscr {S}, i \in \mathcal {N}$. Therefore, $u_i(s,1) \ge F_i(s,\mathbf {w}^{\bar{\mathbf {x}}}_i) \ge F_i(s,\mathbf {w}^{\mathbf {x}}_i)$ for all $s \in \mathscr {S}^1(\mathbf {x}), i \in \mathcal {N}$. So, $\mathscr {S}^1_{i,v}(\mathbf {x})=\emptyset$ for all $i \in \mathcal {N}$. It follows from Lemma 1 that $\mathbf {x}$ is an equilibrium. $\square$

Proof of Corollary 2

Immediate from Remark 1 and Proposition 4 (ii). $\square$

Proof of Proposition 5

(i) Immediate from Remark 1, Proposition 4 (i), and the assumption that $c_i(s) \ge 0$ for all $s \in \mathscr {S}, i \in \mathcal {N}$.

(ii) If $\sum _{s \in \mathscr {S}^0(\bar{\mathbf {x}})}x(s)>0$, (8) is redundant. Otherwise, $\sum _{s \in \mathscr {S}^0(\bar{\mathbf {x}})}x(s)=0$, implying $\mathscr {S}^1(\mathbf {x})\subseteq \mathscr {S}^1(\bar{\mathbf {x}})$, so (8) cuts off an equilibrium $\mathbf {x}$ if and only if $\mathscr {S}^1(\mathbf {x})\subset \mathscr {S}^1(\bar{\mathbf {x}})$. Moreover, if $\mathscr {S}^1(\mathbf {x})\subset \mathscr {S}^1(\bar{\mathbf {x}})$, then $\mathbf {x}$ is a non-maximal equilibrium since $\bar{\mathbf {x}}$ is an equilibrium. $\square$

Proof of Proposition 6

Let $\Xi _{\bar{\mathbf {x}}}$ be the set of strategy profiles $\tilde{\mathbf {x}}$ for which there exists an state $\hat{s} \in \mathscr {S}^0_{i,v}(\bar{\mathbf {x}}) \cup \mathscr {S}^1_{i,v}(\bar{\mathbf {x}})$ such that:

• $\tilde{x}(s)=1$ for all $s \in \mathscr {S}^1_{i,nv}(\bar{\mathbf {x}})$,

• $\tilde{x}(s)=0$ for all $s \in \mathscr {S}^0_{i,nv}(\bar{\mathbf {x}})$,

• $\tilde{x}(\hat{s})=1$ for some $\hat{s} \in \mathscr {S}^0_{i,v}(\bar{\mathbf {x}}) \cup \mathscr {S}^1_{i,v}(\bar{\mathbf {x}})$,

• $\tilde{x}(s)=0$ for all $s \in \mathscr {S}^0_{i,v}(\bar{\mathbf {x}}) \cup \mathscr {S}^1_{i,v}(\bar{\mathbf {x}}) \slash \{\hat{s}\}$.

We show that each $\tilde{\mathbf {x}} \in \Xi _{\bar{\mathbf {x}}}$ is not an equilibrium. Since $\hat{s} \in \mathscr {S}^0_{i,v}(\bar{\mathbf {x}}) \cup \mathscr {S}^1_{i,v}(\bar{\mathbf {x}})$, there are two mutually exclusive cases for $\hat{s}$:

Case 1 If $\hat{s} \in \mathscr {S}^1_{i,v}(\bar{\mathbf {x}})$, then $w^{\tilde{\mathbf {x}}}_i(s) \ge w^{\bar{\mathbf {x}}}_i(s)$ for all $s \in \mathscr {S}$ by Proposition 3 (i). As a result, $F_i(s,\mathbf {w}^{\tilde{\mathbf {x}}}_i) \ge F_i(s,\mathbf {w}^{\bar{\mathbf {x}}}_i)$ for all $s \in \mathscr {S}$. In particular, $F_i(\hat{s},\mathbf {w}^{\tilde{\mathbf {x}}}_i) \ge F_i(\hat{s},\mathbf {w}^{\bar{\mathbf {x}}}_i)> u_i(\hat{s},1)$. Since $F_i(\hat{\mathbf {s}},\mathbf {w}^{\tilde{\mathbf {x}}}_i)> u_i(\hat{s},1)$ and $\tilde{x}(\hat{s})=1$, the Bellman–Shapley equation (5) is violated at state $\hat{s}$ under $\tilde{\mathbf {x}}$, so it is not an equilibrium.

Case 2 If $\hat{s} \in \mathscr {S}^0_{i,v}(\bar{\mathbf {x}})$, then consider strategy profile $\breve{\mathbf {x}}$ defined as follows:

• $\breve{x}(s)=1$ for all $s \in \mathscr {S}^1_{i,nv}(\bar{\mathbf {x}})$,

• $\breve{x}(s)=0$ for all $s \in \mathscr {S}\slash \mathscr {S}^1_{i,nv}(\bar{\mathbf {x}})$.

Therefore, $\tilde{\mathbf {x}}$ and $\breve{\mathbf {x}}$ take the same value for all $s \in \mathscr {S}\slash \{\hat{s}\}$. By Corollary 1, if the strategy profile $\breve{\mathbf {x}}$ is not an equilibrium, $\tilde{\mathbf {x}}$ cannot be an equilibrium since $\mathscr {S}^1(\breve{\mathbf {x}}) \subseteq \mathscr {S}^1(\tilde{\mathbf {x}})$.

Suppose $\breve{\mathbf {x}}$ is an equilibrium. By Proposition 3 (i), $w^{\breve{\mathbf {x}}}_i(s) \ge w^{\bar{\mathbf {x}}}_i(s)$ for all $s \in \mathscr {S}$. Therefore, $F_i(s,\mathbf {w}^{\breve{\mathbf {x}}}_i)\ge F_i(s,\mathbf {w}^{\bar{\mathbf {x}}}_i)>u_i(s,1)$ for all $s \in \mathscr {S}^0_{i,v}(\bar{x})$. In particular, $F_i(\hat{s},\mathbf {w}^{\breve{\mathbf {x}}}_i)> u_i(\hat{s},1)$. Moreover, suppose that $\tilde{\mathbf {x}}$ is an equilibrium; thus $w^{\tilde{\mathbf {x}}}_i(s) \ge w^{\breve{\mathbf {x}}}_i(s)$ for all $s \in \mathscr {S}$ by Proposition 4 (i). As a result, $F_i(s,\mathbf {w}^{\tilde{\mathbf {x}}}_i)\ge F_i(s,\mathbf {w}^{\breve{\mathbf {x}}}_i)$ for all $s \in \mathscr {S}$. In particular, $F_i(\hat{s},\mathbf {w}^{\tilde{\mathbf {x}}}_i)\ge F_i(\hat{s},\mathbf {w}^{\breve{\mathbf {x}}}_i)> u_i(\hat{s},1)$, and this means that the Bellman–Shapley equation (5) is violated at state $\hat{s}$ under $\tilde{\mathbf {x}}$. Therefore, $\tilde{\mathbf {x}}$ is not an equilibrium, which is a contradiction.

So far, we have shown that each $\tilde{\mathbf {x}} \in \Xi _{\bar{\mathbf {x}}}$ is not an equilibrium. For any strategy profile $\mathbf {x}$, satisfying the conditions of Proposition 6, there exists a strategy profile $\tilde{\mathbf {x}} \in \Xi _{\bar{\mathbf {x}}}$ such that $\mathscr {S}^1(\tilde{\mathbf {x}}) \subseteq \mathscr {S}^1(\mathbf {x})$, so $\mathbf {x}$ cannot be an equilibrium by Corollary 1. $\square$

Proof of Proposition 7

If $\sum \nolimits _{s \in \mathscr {S}^1_{i,nv}(\bar{\mathbf {x}})}[1-x(s)]>0$, then (9) is redundant. Now, consider the other case in which $\sum \nolimits _{s \in \mathscr {S}^1_{i,nv}(\bar{\mathbf {x}})}[1-x(s)]=0$. This implies $\mathscr {S}^1_{i,nv}(\bar{\mathbf {x}}) \subseteq \mathscr {S}^1(\mathbf {x})$. If $\mathbf {x}$ is an equilibrium and $\mathscr {S}^1_{i,nv}(\bar{\mathbf {x}}) \subseteq \mathscr {S}^1(\mathbf {x})$, then $\mathscr {S}^1(\mathbf {x}) \cap (\mathscr {S}^0_{i,v}(\bar{\mathbf {x}}) \cup \mathscr {S}^1_{i,v}(\bar{\mathbf {x}}))= \emptyset$ by Proposition 6. This is equivalent to saying that if $\mathbf {x}$ is an equilibrium and $\sum \nolimits _{s \in \mathscr {S}^1_{i,nv}(\bar{\mathbf {x}})}[1-x(s)]=0$, then $x(s)=0$ for all $s \in \mathscr {S}^0_{i,v}(\bar{\mathbf {x}}) \cup \mathscr {S}^1_{i,v}(\bar{\mathbf {x}})$. $\square$

Recall that $\mathbf {d}$ is the payoff profile of the strategy profile $\mathbf {0}$.

Lemma 2

• (i) $d_i(s)= F_i(s,\mathbf {d}_i)$ for all $s \in \mathscr {S}, i \in \mathcal {N}$.

• (ii) If a strategy profile $\mathbf {x}$ is an equilibrium, then $w_i^{\mathbf {x}}(s) \ge d_i(s)$ for all $s \in \mathscr {S}, i \in \mathcal {N}$.

• (iii) If $u_i(\bar{s},1) < d_i(\bar{s})$ for some $\bar{s}\in \mathscr {S}, i \in \mathcal {N}$, then $x(\bar{s})=0$ for each equilibrium $\mathbf {x}$.

Proof

• (i) Immediate from Proposition 2 (i).

• (ii) Note that $\mathscr {S}^1(\mathbf {0})= \emptyset \subseteq \mathscr {S}^1(\mathbf {x})$ for each equilibrium $\mathbf {x}$. The result follows from Proposition 4 (i).

• (iii) Substituting $\mathbf {0}$ for $\bar{\mathbf {x}}$ in Proposition 7 implies that the following set of inequalities are valid for $\varPsi$.

  $$\begin{aligned}&\sum _{s \in \mathscr {S}^0_{i,v}(\mathbf {0}) \cup \mathscr {S}^1_{i,v}(\mathbf {0})}x(s)\le \left( \left| \mathscr {S}^0_{i,v}(\mathbf {0})\right| +\left| \mathscr {S}^1_{i,v}(\mathbf {0})\right| \right) \sum _{s \in \mathscr {S}^1_{i,nv}(\mathbf {0})}[1-x(s)]=0&\forall i \in \mathcal {N}, \end{aligned}$$

  where we have made use of the fact that $\mathscr {S}^1_{i,nv}(\mathbf {0}) \subseteq \mathscr {S}^1(\mathbf {0})=\emptyset$, to write the equality. Note that $\bar{s}\in \mathscr {S}^0_{i,v}(\mathbf {0})$ since $u_i(\bar{s},1) < d_i(\bar{s})$. By the above set of inequalities, $x(\bar{s})=0$ is valid for $\varPsi$, and hence $x(\bar{s})=0$ for each equilibrium $\mathbf {x}$. $\square$

Proof of Proposition 8

(i) $(\Rightarrow )$$\mathbf {x}\in \mathbb {B}^{|\mathscr {S}|}$ by (10f). For each $s \in \mathscr {S}$, there are two cases:

If $x(s)=0$, $w_i(s)=F_i(s,\mathbf {w}_i)$ for all $i \in \mathcal {N}$ by (10a)–(10b).

If $x(s)=1$, $w_i(s)=u_i(s,1)$ for all $i \in \mathcal {N}$ by (10c)–(10d).

As a result, $\mathbf {w}$ is associated payoff profile of $\mathbf {x}$ by Proposition 2 (i). It also follows from (10a) that the Bellman–Shapley equation (5) is satisfied for all $s \in \mathscr {S}^1(\mathbf {x})$. Therefore, $\mathbf {x}$ is an equilibrium by Proposition 2 (ii).

$(\Leftarrow )$ Suppose $\mathbf {x}$ is an equilibrium with associated payoff profile $\mathbf {w}$. Constraint (10f) is satisfied since $\mathbf {x}$ is a (pure) strategy profile. Constraint (10a) is satisfied by Proposition 2 (ii). For each $s \in \mathscr {S}$, there are two cases:

If $x(s)=0$, then $w_i(s)=F_i(s,\mathbf {w}_i)$ by Proposition 2 (i). Constraint (10b) is obviously satisfied, and (10c) is satisfied by Lemma 2 (ii). Constraint (10d) is satisfied since:

$$\begin{aligned} w_i(s)=F_i(s,\mathbf {w}_i) \le F_i(s,\mathbf {V}_i), \end{aligned}$$

where the inequality follows from the definition of $\mathbf {V}_i$.

If $x(s)=1$, then $w_i(s)=u_i(s,1)$ by Proposition 2 (i). Constraint (10b) is satisfied since:

$$\begin{aligned} w_i(s)&=u_i(s,1) \le u_i(s,1) +F_i(s,\mathbf {w}_i)-F_i(s,\mathbf {d}_i)\nonumber \\&= u_i(s,1) +F_i(s,\mathbf {w}_i)-d_i(s), \end{aligned}$$

where the first inequality follows from Lemma 2 (ii), and the second equality follows from Lemma 2 (i). Constraints (10c)–(10d) are obviously satisfied.

(ii) If $u_i(s,0), u_i(s,1) \ge 0$ for all $s \in \mathscr {S}, i \in \mathcal {N}$, it can easily be shown by value iteration that $d_i(s)\ge 0$ for all $s \in \mathscr {S}, i \in \mathcal {N}$. The rest of the proof is straightforward.

$\square$

Proof of Proposition 9

Recall that $\gamma _i(s)$ for all $s \in \mathscr {S}, i \in \mathcal {N}$, are optimal dual multipliers of $\mathcal {R}_i(\bar{\mathbf {x}})$. Let $\mathcal {P}_i(\bar{\mathbf {x}})$ denote the subproblem associated with each player i in (11a)–(11b) when $\mathbf {x}$ is set to $\bar{\mathbf {x}}$:

$$\begin{aligned} \mathcal {P}_i(\bar{\mathbf {x}}): \max&\sum _{ s \in \mathscr {S}} c_i(s) w_i(s) \\ s.t.\\&w_i(s) \ge F_i(s,\mathbf {w}_i)&\forall s \in \mathscr {S},\\&w_i(s) \le F_i(s,\mathbf {w}_i)+[u_i(s,1)-d_i(s)]\bar{x}(s)&\forall s \in \mathscr {S}, \\&w_i(s)\ge [u_i(s,1)-d_i(s)] \bar{x}(s)+d_i(s)&\forall s \in \mathscr {S}, \\&w_i(s)\le u_i(s,1)\bar{x}(s)+F_i(s,\mathbf {V}_i)[1-\bar{x}(s)]&\forall s \in \mathscr {S},\\&w_i(s) \text { unrestricted}&\forall s \in \mathscr {S}. \end{aligned}$$

In order to generate a Benders’ optimality cut in Step (2c), optimal dual multipliers of $\mathcal {P}_i(\bar{\mathbf {x}})$ are needed while we only know optimal dual multipliers of $\mathcal {R}_i(\bar{\mathbf {x}})$. In fact, optimal dual multipliers of $\mathcal {R}_i(\bar{\mathbf {x}})$ and $\mathcal {P}_i(\bar{\mathbf {x}})$ are closely related. Let $\pi _{i,1}(s),\pi _{i,2}(s),\pi _{i,3}(s),\pi _{i,4}(s)$ be optimal dual multipliers of $\mathcal {P}_i(\bar{\mathbf {x}})$ associated with equations (10a)–(10d), for all $s \in \mathscr {S}, i \in \mathcal {N}$, respectively. It can easily be seen that for all $i \in \mathcal {N}$:

$$\begin{aligned}&\pi _{i,1}(s)=\pi _{i,2}(s)=0&\forall s \in \mathscr {S}^1(\bar{\mathbf {x}}), \end{aligned}$$

(19a)

$$\begin{aligned}&\pi _{i,1}(s)+\pi _{i,2}(s)=\gamma _i(s)&\forall s \in \mathscr {S}^0(\bar{\mathbf {x}}), \end{aligned}$$

(19b)

$$\begin{aligned}&\pi _{i,3}(s)=\pi _{i,4}(s)=0&\forall s \in \mathscr {S}^0(\bar{\mathbf {x}}), \end{aligned}$$

(19c)

$$\begin{aligned}&\pi _{i,3}(s)+\pi _{i,4}(s)=\gamma _i(s)&\forall s \in \mathscr {S}^1(\bar{\mathbf {x}}), \end{aligned}$$

(19d)

$$\begin{aligned}&\pi _{i,1}(s) \le 0, \pi _{i,2}(s)\ge 0, \pi _{i,3}(s) \le 0, \pi _{i,4}(s) \ge 0&\forall s \in \mathscr {S}. \end{aligned}$$

(19e)

At each iteration of the Decomposition Algorithm, $\mathcal {R}_i(\bar{\mathbf {x}})$ is solved, and $\gamma _i(s)$ is obtained for all $s \in \mathscr {S}, i \in \mathcal {N}$. There are multiple dual optimal solutions for $\mathcal {P}_i(\bar{\mathbf {x}})$, and we may use any $\pi _{i,1}(s),\pi _{i,2}(s),\pi _{i,3}(s),\pi _{i,4}(s)$, satisfying (19a)–(19e), to generate a Benders’ optimality cut. A Benders’ optimality cut is as follows:

$$\begin{aligned} \theta _i\le&\sum _{s\in \mathscr {S}}\pi _{i,1}(s) \,u_i(s,0) + \pi _{i,2}(s)\, \Big (u_i(s,0) + [u_i(s,1)-d_i(s)]x(s)\Big ) \nonumber \\&\quad +\, \pi _{i,3}(s)\, \Big ([u_i(s,1)-d_i(s)] x(s)+d_i(s) \Big )\nonumber \\&\quad +\, \pi _{i,4}(s)\, \Big ( u_i(s,1) x(s) +F_i(s,\mathbf {V}_i) [1-x(s)]\Big )&\forall i \in \mathcal {N}. \end{aligned}$$

(20)

Since $\mathcal {R}_i(\bar{\mathbf {x}})$ is not degenerate, $\langle \gamma _i(s)\rangle _{s\in \mathscr {S}}$ is the unique dual solution of $\mathcal {R}_i(\bar{\mathbf {x}})$. In order to generate a Pareto-optimal Benders’ optimality cut, we need to find $\pi ^*_{i,1}(s), \pi ^*_{i,2}(s),\pi ^*_{i,3}(s),\pi ^*_{i,4}(s)$ for all $s\in \mathscr {S}$ such that they minimize the term on the right-hand side of (20) subject to (19a)–(19e) for each equilibrium $\mathbf {x}$. The right-hand side of (20) subject to (19a)–(19e) may be rewritten as follows:

$$\begin{aligned}&\sum _{s\in \mathscr {S}}\pi _{i,1}(s) \,u_i(s,0)+\pi _{i,2}(s)\,\Big (u_i(s,0)+[u_i(s,1)-d_i(s)]x(s)\Big ) \nonumber \\&\quad +\, i_{i,3}(s)\, \Big ([u_i(s,1)-d_i(s)] x(s)+d_i(s) \Big )\nonumber \\&\quad +\,\pi _{i,4}(s)\,\Big (u_i(s,1) x(s)+F_i(s,\mathbf {V}_i)[1-x(s)]\Big ) \nonumber \\&=\sum _{s \in \mathscr {S}^0(\bar{\mathbf {x}})} [\pi _{i,1}(s)+\pi _{i,2}(s)] \,u_i(s,0)\nonumber \\&\quad +\,\pi _{i,2}(s)\,[u_i(s,1)-d_i(s)]x(s) +\,\sum _{s \in \mathscr {S}^1(\bar{\mathbf {x}})} [\pi _{i,3}(s)+\pi _{i,4}(s)]\, u_i(s,1) x(s)\nonumber \\&\quad +\, [\pi _{i,3}(s)\,d_i(s)+\pi _{i,4}(s)\,F_i(s,\mathbf {V}_i)][1-x(s)]\nonumber \\&=\sum _{s \mathscr {S}^0(\bar{\mathbf {x}})} \gamma _i(s) \,u_i(s,0)+\pi _{i,2}(s)\,[u_i(s,1)-d_i(s)]x(s)\nonumber \\&\quad +\,\sum _{s \in \mathscr {S}^1(\bar{\mathbf {x}})} \gamma _i(s) \, u_i(s,1) x(s)+[\pi _{i,3}(s)\,d_i(s) +\pi _{i,4}(s)\,F_i(s,\mathbf {V}_i)][1-x(s)], \end{aligned}$$

(21)

where the first and second equality follow from (19a), (19c) and (19b), (19d), respectively. In order to minimize (21) subject to (19a)–(19e), we may seek to minimize (21) for each $s \in \mathscr {S}$ separately. There are three mutually exclusive cases for s:

• 1. If $s \in \mathscr {S}^0(\bar{\mathbf {x}})$ and $u_i(s,1)\ge d_i(s)$, then $\pi ^*_{i,1}(s):=\min \{0,\gamma _i(s)\}=-\gamma ^-_i(s), \pi ^*_{i,2}(s):=\max \{0,\gamma _i(s)\}=\gamma ^+_i(s),\pi ^*_{i,3}(s):=0,\pi ^*_{i,4}(s):=0$ minimizes (21) subject to (19a)–(19e).

• 2. If $s \in \mathscr {S}^0(\bar{\mathbf {x}})$ and $u_i(s,1) < d_i(s)$, then x(s) is equal to 0 for any equilibrium $\mathbf {x}$ by Lemma 2. As a result, $\pi ^*_{i,1}(s):=-\gamma ^-_i(s), \pi ^*_{i,2}(s):=\gamma ^+_i(s),\pi ^*_{i,3}(s):=0,\pi ^*_{i,4}(s):=0$ minimizes (21) subject to (19a)–(19e).

• 3. If $s \in \mathscr {S}^1(\bar{\mathbf {x}})$, then $\pi ^*_{i,1}(s):=0, \pi ^*_{i,2}(s):=0,\pi ^*_{i,3}(s):=-\gamma ^-_i(s),\pi ^*_{i,4}(s):=\gamma ^+_i(s)$ minimizes (21) subject to (19a)–(19e) since $F_i(s,\mathbf {V}_i) \ge F_i(s,\mathbf {d}_i)=d_i(s)$.

$\square$

Proof of Proposition 10

If $s \in \mathscr {S}^0(\bar{\mathbf {x}}_2)$, the Bellman–Shapley equation (5) is obviously satisfied for all $i \in \mathcal {N}$. Conversely, if $s \in \mathscr {S}^1(\bar{\mathbf {x}}_2)$, then

$$\begin{aligned}&w^{\bar{\mathbf {x}}_2}_i(s)=u_i(s,1)= V_i(s) \ge F_i(s,\mathbf {V}_i) \ge F_i(s,\mathbf {w}^{\bar{\mathbf {x}}_2}_i)&\forall i\in \mathcal {N}, \end{aligned}$$

where the first equality follows from Proposition 2 (i), the second equality follows from the definition of $\bar{\mathbf {x}}^2$, and the first and second inequalities follow from the definition of $\mathbf {V}_i$. Hence, the Bellman–Shapley equation (5) is satisfied for all $s \in \mathscr {S}^1(\bar{\mathbf {x}}_2), i \in \mathcal {N}$.

$\square$

Random instance generation We generated two categories of two-player consensus stopping game instances as follows. For the first (second) category, the size of $S_i$ is equal to 40 (60) for both players. Recall from Sect. 8.1 that (1) the game state is $s=(s_1,s_2)$, i.e., $\mathscr {S}=S_1 \times S_2$, and (2) the game transition probability matrix $\mathscr {P}$ is the Kronecker product of the individual transition probability matrices, i.e., $\mathscr {P}(s^{\prime }_1,s^{\prime }_2|s_1,s_2)=P_i(s^{\prime }_1|s_1) P_i(s^{\prime }_2|s_2)$. In many practical applications of consensus stopping games such as war termination and organ exchange, which were discussed in Sect. 1, transition in state of the game is slow, i.e., the game most likely remains in the same state at the next period as that of the current period. For this reason, we randomly generated a set of individual transition probability matrices that are highly diagonal, i.e., the diagonal entries are close to 1. Specifically, in generating our transition matrices we used the notion of increasing failure rate (IFR) property. The IFR property has its origins in maintenance optimization and reliability literature [4], but it has been recently shown that data in varying real contexts, primarily in healthcare and service operations, empirically exhibit IFR property [2]. The transition matrices we generated are designed to be moderately sparse but do not have any diagonal entry that is less than 0.99. Such transition matrices can be encountered in real-life dynamic settings where decision epochs are spaced very close to each other so that leaving the state of the system in one period is not very likely. It is also common to see sparse transition matrices with large diagonal entries when solutions of a large-scale dynamic decision-making problem are approximated through state aggregation. To ensure our transition matrices have the IFR property and the specified threshold probabilities in their diagonals, we simulated their entries iteratively starting from the top row. In each particular row, we simulated the entries from left to right in column order after randomly fixing the diagonal entry in the specified range. All entries of the same row are generated from a uniform distribution whose boundaries are imposed by a corresponding partial row sum from the previous row due to the IFR property. While such order restrictions can disallow some entries to be positive, we also allowed each nondiagonal entry to be 0 with probability 0.01. Across all transition matrices we generated, on average, the transition matrices for 40-state-per-player instances were 53% sparse whereas the transition matrices for 60-state-per-player instances were 63% sparse.

For each $i \in \{1,2 \}$ and $s\in \mathscr {S}$, the rewards $u_i(s,0)$ and $u_i(s,1)$ only depend on $s_i$. For each $i \in \{1,2 \}$ and $s \in \{ (s_1,s_2) \in \mathscr {S}: s_i\ne |S_i|\}$, $u_i(s,0)$ is generated according to a uniform distribution on the interval $[-150,50]$, and $u_i(s,1)$ is equal to $\min \nolimits _{s \in \mathscr {S}} d_i(s) +rand(s_i) \frac{|S_i|-s_i}{|S_i|} \max \nolimits _{s \in \mathscr {S}} d_i(s)$, where $rand(s_i)$ is a random number from a uniform distribution on [0, 1]. In addition, the last state (i.e., $s_i=|S_i|$) is absorbing, and its continuation and stopping rewards are 0 and $-\,\infty$, respectively. Finally, the initial state of the game $\hat{s}$ is generated by a discrete uniform distribution between $\frac{|S_i|}{4}$ and $\frac{|S_i|}{2}$.