CEGAR for compositional analysis of qualitative properties in Markov decision processes

Abstract

We consider Markov decision processes (MDPs) which are a standard model for probabilistic systems. We focus on qualitative properties for MDPs that can express that desired behaviors of the system arise almost-surely (with probability 1) or with positive probability. We introduce a new simulation relation to capture the refinement relation of MDPs with respect to qualitative properties, and present discrete graph algorithms with quadratic complexity to compute the simulation relation. We present an automated technique for assume-guarantee style reasoning for compositional analysis of two-player games by giving a counterexample guided abstraction-refinement approach to compute our new simulation relation. We show a tight link between two-player games and MDPs, and as a consequence the results for games are lifted to MDPs with qualitative properties. We have implemented our algorithms and show that the compositional analysis leads to significant improvements.

Appendix

We start with an example that shows that also for alternating games combined simulation is finer that the intersection of simulation and alternating-simulation relation.

Fig. 7

Games $G, G'$ such that $G\leqslant _{\mathcal {S}}G'$ and $G\leqslant _{\mathcal {A}}G'$, but $G\not \leqslant _{\mathcal {C}}G'$

Example 7

Figure 7 shows two alternating games $G, G'$, where the circular states belong to Player 1 and the rectangular states belong to Player 2, white nodes are labeled by proposition p and gray nodes by proposition q. The largest simulation and alternating-simulation relations between $G$ and $G'$ are: ${\mathcal {S}}_{\max }=\{(s_0, t_0),(s_1, t_1),(s_2, t_2), (s_3, t_1)\}, {\mathcal {A}}_{\max }=\{(s_0, t_0),(s_0, t_4),(s_2, t_2), (s_3, t_3), (s_1, t_3), (s_1, t_1)\}$. Formula $\langle \!\langle 1 \rangle \!\rangle (\Box (p \wedge \langle \!\langle 1,2 \rangle \!\rangle ({\mathsf {true}}\, \, {\mathcal {U}}\, q)))$ is satisfied in state $s_0$, but not in state $t_0$, hence $(s_0, t_0)\not \in {\mathcal {C}}_{\max }$.$\square$

We now present detailed proofs of Lemma 1 and Theorem 2 in the context of alternating games.

Lemma 8

Given two alternating games $G$ and $G'$, let ${\mathcal {C}}_{\max }$ be the combined simulation. For all $(s,s') \in {\mathcal {C}}_{\max }$ the following assertions hold:

• 1. For all Player 1 strategies $\sigma$ in $G$, there exists a Player 1 strategy $\sigma '$ in $G'$ such that for every play $\omega ' \in {\mathsf {Plays}}(s',\sigma ')$ there exists a play $\omega \in {\mathsf {Plays}}(s,\sigma )$ such that $\omega \leqslant _{\mathcal {C}}\omega '$.

• 2. For all pairs of strategies $\sigma$ and $\theta$ in $G$, there exists a pair of strategies $\sigma '$ and $\theta '$ in $G'$ such that ${\mathsf {Play}}(s,\sigma ,\theta ) \leqslant _{\mathcal {C}}{\mathsf {Play}}(s',\sigma ',\theta ')$,

Proof

Assertion 1 As the states of Player 1 and Player 2 are distinguished by the ${\mathsf {turn}}$ atomic proposition, it follows from the fact that $(s,s') \in {\mathcal {C}}_{\max }$, that either (i) $s \in S_1$ and $s' \in S'_1$ or (ii) $s \in S_2$ and $s' \in S'_2$.

For the first case (i) we consider a winning strategy $\sigma ^{{\mathcal {C}}}$ in $G^{\mathcal {C}}$ such that for all $(s,s') \in {\mathcal {C}}_{\max }$ and against all strategies $\theta ^{\mathcal {C}}$ we have ${\mathsf {Play}}((s,s'),\sigma ^{\mathcal {C}},\theta ^{\mathcal {C}}) \in \llbracket \Box (\lnot p) \rrbracket _{G^{\mathcal {C}}}$. Given the Player 1 strategy $\sigma$ in $G$ we construct $\sigma '$ in $G'$ using the strategy $\sigma ^{\mathcal {C}}$. Let h be an arbitrary history in $G^{\mathcal {C}}$ that visits only states of type $(S\times S')$ that are in ${\mathcal {C}}_{\max }$ and ends in $(s,s')$. Consider a history $w \cdot s$ in $G$ and $w'\cdot s'$ in $G'$. Let $\sigma (w \cdot s) = a$, we define $\sigma '(w' \cdot s')$ as action $a' = \sigma ^{{\mathcal {C}}}(h \cdot ((s,s'),{\mathsf {Alt}},2) \cdot ((s,s'),{\mathsf {Alt}},a,2))$, i.e., action $a'$ corresponds to the choice of the proponents winning strategy $\sigma ^{\mathcal {C}}$ in response to the adversarial choice of checking step-wise alternating-simulation followed by action a in $G$. As both s and $s'$ are Player-1 states we have that $\vert \delta (s,a) \vert =1$ and $\vert \delta '(s',a') \vert =1$. Let $(t,t')$ be the unique state reached in 2 steps from $((s,s'),{\mathsf {Alt}},a,a',2)$ in $G^{\mathcal {C}}$. Assume towards contradiction that ${\mathcal {L}}^{\mathcal {C}}((t,t')) = \{ p \}$, then there exists a strategy for adversary that reaches a loosing state while the proponent plays a winning strategy $\sigma ^{\mathcal {C}}$ and the contradiction follows. For the second case (ii) we have that states s and $s'$ belong to Player 2, and there is a single action available for $\sigma '$.

Assertion 2 The proof is similar to the first assertion, and instead of using the step-wise alternating-simulation gadget for strategy construction (of the first item) we use the step-wise simulation gadget from $G^{\mathcal {C}}$ to construct the strategy pairs. $\square$

Theorem 7

For all alternating games $G$ and $G'$ we have ${\mathcal {C}}_{\max } = \preccurlyeq _{C}^*= \preccurlyeq _{C}$.

Proof

First implication: We first prove the implication ${\mathcal {C}}_{\max }\subseteq \preccurlyeq _{C}^*$. We will show the following assertions:

• For all states s and $s'$ such that $(s,s') \in {\mathcal {C}}_{\max }$, we have that every ${\text {C-ATL}}^*$ state formula satisfied in s is also satisfied in $s'$.

• For all plays $\omega$ and $\omega '$ such that $\omega \leqslant _{\mathcal {C}}\omega '$, we have that every ${\text {C-ATL}}^*$ path formula satisfied in $\omega$ is also satisfied in $\omega '$.

We will prove the theorem by induction on the structure of the formulas. The interesting cases for the induction step are formulas $\langle \!\langle 1 \rangle \!\rangle (\varphi )$ and $\langle \!\langle 1,2 \rangle \!\rangle (\varphi )$, where $\varphi$ are path formulas.

• Assume $s \models \langle \!\langle 1 \rangle \!\rangle (\varphi )$ and $(s,s') \in {\mathcal {C}}_{\max }$. It follows that there exists a strategy $\sigma \in \varSigma$ that ensures the path formula $\varphi$ from state s against any strategy $\theta \in \varTheta$. We want to show that $s' \models \langle \!\langle 1 \rangle \!\rangle (\varphi )$. By Lemma 8 (item 1) we have that there exists a strategy $\sigma '$ for Player 1 from $s'$ such that for every play $\omega ' \in {\mathsf {Plays}}(s',\sigma ')$ there exists a play $\omega \in {\mathsf {Plays}}(s,\sigma )$ such that $\omega \leqslant _{\mathcal {C}}\omega '$. By inductive hypothesis we have that $s' \models \langle \!\langle 1 \rangle \!\rangle (\varphi )$.

• Assume $s \models \langle \!\langle 1,2 \rangle \!\rangle (\varphi )$ and $(s,s') \in {\mathcal {C}}_{\max }$. It follows that there exist strategies $\sigma \in \varSigma , \theta \in \varTheta$ that ensure the path formula $\varphi$ from state s. By Lemma 8 (item 2) we have that there exist strategies $\sigma '$ and $\theta '$ such that the two plays $\omega ' = {\mathsf {Play}}(s',\sigma ',\theta ')$ and $\omega ={\mathsf {Play}}(s,\sigma ,\theta )$ satisfy $\omega \leqslant _{\mathcal {C}}\omega '$. By inductive hypothesis we have that $s' \models \langle \!\langle 1,2 \rangle \!\rangle (\varphi )$.

• Consider a path formula $\varphi$. If $\omega \leqslant _{\mathcal {C}}\omega '$, then by inductive hypothesis for every sub-formula $\varphi '$ of $\varphi$ we have that if $\omega \models \varphi '$ then $\omega '\models \varphi '$. It follows that if $\omega \models \varphi$ then $\omega '\models \varphi$.

Second implication: It remains to prove the second implication $\preccurlyeq _{C}^* \subseteq \preccurlyeq _{C}\subseteq {\mathcal {C}}_{\max }$. We prove that from the assumption that $(s,s') \not \in {\mathcal {C}}_{\max }$ we can construct a ${\text {C-ATL}}$ formula $\varphi$ such that $s \models \varphi$ and $s' \not \models \varphi$. We refer to the formula $\varphi$ as a distinguishing formula. Assume that given states s and $s'$ we have that $(s,s') \not \in {\mathcal {C}}_{\max }$, then there exists a winning strategy in the corresponding combined-simulation game for the adversary from state $(s,s')$, i.e., there exists a strategy $\theta ^{\mathcal {C}}$ such that against all strategies $\sigma ^{\mathcal {C}}$ we have ${\mathsf {Play}}((s,s'),\sigma ^{\mathcal {C}},\theta ^{\mathcal {C}})$ reaches a state labeled by p. As memoryless strategies are sufficient for both players in $G^{\mathcal {C}}$ [55], there also exists a bound $i \in {\mathbb {N}}$, such that the proponent fails to match the choice of the adversary in at most i turns. We construct the ${\text {C-ATL}}$ formula $\varphi$ inductively:

Base case::

Assume $(s,s') \not \in {\mathcal {C}}_{\max }$ and let 0 be the number of turns the adversary needs to play in order to win. It follows that $(s,s')$ is a winning state for the adversary, i.e., ${\mathcal {L}}^{{\mathcal {C}}}((s,s')) = \{p\}$. It follows that ${\mathcal {L}}(s) \ne {\mathcal {L}}'(s')$. There are two options: (i) there exists an atomic proposition $q \in {\mathsf {AP}}$ that is true in s and not true in $s'$ and distinguishes the two states, or (ii) there exists an atomic proposition $q \in {\mathsf {AP}}$ that is not true in s and true in $s'$, in that case the formula $\lnot q$ distinguishes the two states.

Induction step::

Assume $(s,s') \not \in {\mathcal {C}}_{\max }$ and let $n+1$ be the number of turns the adversary needs to play in order to win. As the states of Player 1 and Player 2 are distinguished by the ${\mathsf {turn}}$ atomic proposition, it follows that either (i) $s \in S_1$ and $s' \in S'_1$ or (ii) $s \in S_2$ and $s' \in S'_2$. Otherwise the adversary could win in 0 turns from $(s,s')$.

We first consider case (i), i.e., $(s,s') \in S_1 \times S'_1$. The adversary can choose whether to verify (1) step-wise alternating-simulation (${\mathsf {Alt}}$) or (2) step-wise simulation (${\mathsf {Sim}}$). After that he chooses an action a to be played according the adversarial strategy $\theta ^{\mathcal {C}}$ in state $(s,s')$, such that no matter what the proponent plays, the adversary will win in n turns. We consider two cases: (1) the adversary checks for step-wise alternating-simulation relation (${\mathsf {Alt}}$), or (2) the adversary checks for step-wise simulation relation (${\mathsf {Sim}}$). For case (1) we have that there exists an action a for the adversary such that for all actions $a'$ of the proponent the adversary can win in n turns from the unique successor $(t,t')$ of $(s,s')$ given ${\mathsf {Alt}}$ and a was played by the adversary and $a'$ by the proponent. From the induction hypothesis there exists a ${\text {C-ATL}}$ formula $\varphi _n$ such that $t \models \varphi _n$ and $t' \not \models \varphi _n$. We define the formula $\varphi _{n+1}$ that distinguishes states s and $s'$ as $\langle \!\langle 1 \rangle \!\rangle (\bigcirc \varphi _n)$. For case (2), where the adversary plays ${\mathsf {Sim}}$ the proof is exactly the same, as step-wise simulation turn from Player 1 states coincides with step-wise alternating-simulation turn.

Next we first consider case (ii), i.e., $(s,s') \in S_2 \times S'_2$. The adversary can choose whether to verify (1) step-wise alternating-simulation (${\mathsf {Alt}}$) or(2) step-wise simulation (${\mathsf {Sim}}$). We start with first case (1): there is a unique action a available to the adversary from state $((s,s'),{\mathsf {Alt}},2)$ and similarly a unique action $a'$ for the proponent from $((s,s'),a,{\mathsf {Alt}},1)$. The adversary chooses an action $t'$ from the $((s,s'),a,a',{\mathsf {Alt}},2)$ according to the winning strategy and the proponent chooses some action $t_i$ from a set of available successor $(t_1,t_2, \ldots , t_m)$. As the adversary follows a winning strategy $\theta ^{\mathcal {C}}$ we have that it wins from all states $(t_i,t')$ for $1 \le i \le m$ in at most n turns. From the induction hypothesis there exist ${\text {C-ATL}}$ formulas $\varphi ^i_n$ such that $t_i \models \varphi ^i_n$ and $t' \not \models \varphi ^i_n$. We define the formula $\varphi _{n+1}$ that distinguishes states s and $s'$ as $\langle \!\langle 1 \rangle \!\rangle (\bigcirc (\bigvee _{1 \le i \le m} \varphi ^i_n)$. For case (2) where the adversary verifies the step-wise simulation step, the proof is analogous. The formula that distinguishes states s and $s'$ is $\langle \!\langle 1,2 \rangle \!\rangle ((\bigcirc \bigvee _{1 \le i \le m} \varphi ^i_n))$.

The desired result follows. $\square$