Verified Probabilistic Policies for Deep Reinforcement Learning

Abstract

Deep reinforcement learning is an increasingly popular technique for synthesising policies to control an agent’s interaction with its environment. There is also growing interest in formally verifying that such policies are correct and execute safely. Progress has been made in this area by building on existing work for verification of deep neural networks and of continuous-state dynamical systems. In this paper, we tackle the problem of verifying probabilistic policies for deep reinforcement learning, which are used to, for example, tackle adversarial environments, break symmetries and manage trade-offs. We propose an abstraction approach, based on interval Markov decision processes, that yields probabilistic guarantees on a policy’s execution, and present techniques to build and solve these models using abstract interpretation, mixed-integer linear programming, entropy-based refinement and probabilistic model checking. We implement our approach and illustrate its effectiveness on a selection of reinforcement learning benchmarks.

Appendix: Proof of Theorem 1

We provide here a proof of Theorem 1, from Sect. 3, which states that:

Given a state $s\in S$ of an RL execution model DTMP, and abstract state $\hat{s}\in \hat{S}$ of the corresponding controller abstraction IMDP for which $s\in \hat{s}$, we have:

$$ { Pr _{s}^{}}(\Diamond ^{\leqslant k} fail ) \ \leqslant \ { Pr _{\hat{s}}^{\max \max }}(\Diamond ^{\leqslant k} fail ) $$

By the definition of ${ Pr _{\hat{s}}^{\max \max }}(\cdot )$, it suffices to show that there is some policy $\sigma$ and some environment policy $\tau$ in the IMDP such that:

$$\begin{aligned} { Pr _{s}^{}}(\Diamond ^{\leqslant k} fail ) \ \leqslant \ { Pr _{\hat{s}}^{\sigma ,\tau }}(\Diamond ^{\leqslant k} fail ) \end{aligned}$$

(2)

Recall that, in the construction of the IMDP (see Definition 7), an abstract state $\hat{s}$ is associated with a partition of subsets $\hat{s}_j$ of $\hat{s}$, each of which is used to define the j-labelled choice in state $\hat{s}$. Let $\sigma$ be the policy that picks in each state s (regardless of history) the unique index $j_s$ such that $s\in \hat{s}_{j_s}$. Then, let $\tau$ be the environment policy that selects the upper bound of the interval for every transition probability. We use function $\hat{{\mathbf {P}}}_{\tau }$ to denote the chosen probabilities, i.e., we have $\hat{{\mathbf {P}}}_{\tau }(\hat{s},j_s,\hat{s}') = \hat{{\mathbf {P}}}_{U}(\hat{s},j_s,\hat{s}')$ for any $\hat{s},j_s,\hat{s}'$.

The probabilities ${ Pr _{\hat{s}}^{\sigma ,\tau }}(\Diamond ^{\leqslant k} fail )$ for these policies, starting in $\hat{s}$, are defined similarly to those for discrete-time Markov processes (see Sect. 2):

$$\begin{aligned} { Pr _{\hat{s}}^{\sigma ,\tau }}(\Diamond ^{\leqslant k} fail )= \left\{ \begin{array}{cl} 1 &{} \text {if } \hat{s}\models fail \\ 0 &{} \text {if } \hat{s}\not \models fail \wedge k{\,=\,}0 \\ \sum \limits _{\hat{s}'\in \text {supp}(\hat{{\mathbf {P}}}(\hat{s},j_s,\cdot ))}\hat{{\mathbf {P}}}(\hat{s},j_s,\hat{s}'){\cdot }{ Pr _{\hat{s}'}^{\sigma ,\tau }}(\Diamond ^{\leqslant k-1} fail ) &{} \text {otherwise.} \end{array}\right. \end{aligned}$$

Since this is defined recursively, we prove (2) by induction over k. For the case $k=0$, the definitions of ${ Pr _{s}^{}}(\Diamond ^{\leqslant 0} fail )$ and ${ Pr _{\hat{s}}^{}}(\Diamond ^{\leqslant 0} fail )$ are equivalent: they equal 1 if $s\models fail$ (or $\hat{s}\models { fail }$) and 0 otherwise. From Definition 7, $s\models { fail }$ implies $\hat{s}\models { fail }$. Therefore, ${ Pr _{s}^{}}(\Diamond ^{\leqslant 0} fail ) \ \leqslant \ { Pr _{\hat{s}}^{\sigma ,\tau }}(\Diamond ^{\leqslant 0} fail )$.

Next, for the inductive step, we will assume, as the inductive hypothesis, that ${ Pr _{s'}^{}}(\Diamond ^{\leqslant k-1} fail ) \ \leqslant \ { Pr _{\hat{s}'}^{\sigma ,\tau }}(\Diamond ^{\leqslant k-1} fail )$ for $s'\in S$ and $\hat{s}'\in \hat{S}$ with $s'\in \hat{s}'$. If $\hat{s}\models { fail }$ then ${ Pr _{\hat{s}}^{\sigma ,\tau }}(\Diamond ^{\leqslant k} fail )=1 \ \geqslant \ { Pr _{s}^{}}(\Diamond ^{\leqslant k} fail )$. Otherwise we have:

$$ {{\begin{array}{rcll} &{}&{} { Pr _{\hat{s}}^{\sigma ,\tau }}(\Diamond ^{\leqslant k} fail ) \\ &{} = &{} \sum \nolimits _{\hat{s}'\in \text {supp}(\hat{{\mathbf {P}}}_{\tau }(\hat{s},j_s,\cdot ))}\hat{{\mathbf {P}}}_{\tau }(\hat{s},j_s,\hat{s}')\cdot { Pr _{\hat{s}'}^{}}(\Diamond ^{\leqslant k-1} fail ) \ &{} \text {by defn. of } \sigma \text { and }{ Pr _{\hat{s}}^{\sigma ,\tau }}(\Diamond ^{\leqslant k} fail ) \\ &{} = &{} \sum \nolimits _{\hat{s}'\in \text {supp}(\hat{{\mathbf {P}}}_{U}(\hat{s},j_s,\cdot ))}\hat{{\mathbf {P}}}_{U}(\hat{s},j_s,\hat{s}')\cdot { Pr _{\hat{s}'}^{}}(\Diamond ^{\leqslant k-1} fail ) \ &{} \text {by defn. of } \tau \\ &{} = &{} \sum \nolimits _{a\in {A}}\pi _U(\hat{s},a)\cdot { Pr _{\hat{E}(\hat{s}_j,a)}^{}}(\Diamond ^{\leqslant k-1} fail ) \ &{} \text {by defn. of }\hat{{\mathbf {P}}}_{U}(\hat{s},j,\hat{s}') \\ &{} \geqslant &{} \sum \nolimits _{a\in {A}}\pi (s,a)\cdot { Pr _{\hat{E}(\hat{s}_j,a)}^{}}(\Diamond ^{\leqslant k-1} fail ) \ &{} \text {since } s\in \hat{s} \text { and by Defn.6 } \\ &{} \geqslant &{} \sum \nolimits _{a\in {A}} \pi (s,a)\cdot { Pr _{E(s,a)}^{}}(\Diamond ^{\leqslant k-1} fail ) &{} \text {by induction and since, by} \\ &{}&{}&{} \text {Defn. 5}, E(s,w)\in \hat{E}(\hat{s}_j,w) \\ &{} = &{} \sum \nolimits _{s'\in \text {supp}({\mathbf {P}}(s,\cdot ))}{\mathbf {P}}(s,s')\cdot { Pr _{s'}^{}}(\Diamond ^{\leqslant k-1} fail ) &{} \text {by defn. of } {\mathbf {P}}(s,s') \\ &{} = &{} { Pr _{s}^{}}(\Diamond ^{\leqslant k} fail ) &{} \text {by defn. of }{ Pr _{s}^{}}(\Diamond ^{\leqslant k} fail ) \\ \end{array}}} $$

which completes the proof.