Assumption Generation for Learning-Enabled Autonomous Systems

Abstract

Providing safety guarantees for autonomous systems is difficult as these systems operate in complex environments that require the use of learning-enabled components, such as deep neural networks (DNNs) for visual perception. DNNs are hard to analyze due to their size (they can have thousands or millions of parameters), lack of formal specifications (DNNs are typically learnt from labeled data, in the absence of any formal requirements), and sensitivity to small changes in the environment. We present an assume-guarantee style compositional approach for the formal verification of system-level safety properties of such autonomous systems. Our insight is that we can analyze the system in the absence of the DNN perception components by automatically synthesizing assumptions on the DNN behaviour that guarantee the satisfaction of the required safety properties. The synthesized assumptions are the weakest in the sense that they characterize the output sequences of all the possible DNNs that, plugged into the autonomous system, guarantee the required safety properties. The assumptions can be leveraged as run-time monitors over a deployed DNN to guarantee the safety of the overall system; they can also be mined to extract local specifications for use during training and testing of DNNs. We illustrate our approach on a case study taken from the autonomous airplanes domain that uses a complex DNN for perception.

Appendix: \(\textsc {PRISM}\) Encoding for TaxiNet with Safety Monitor

We show the $\textsc {PRISM}$ code for $M_2$ and the safety monitor in Fig. 6. We use the output of step 4 in procedure BuildAssumption(Algorithm 3.1) as a safety monitor, i.e., the assumption LTS has both err and sink states, with a transition to err state interpreted as the system aborting. The encoding closely follows the transitions of the assumption computed for $M_1$ over alphabet $\varSigma =Est$.

Fig. 6.

TaxiNet $M_2$ and safety monitor in $\textsc {PRISM}$.

Variable $\mathtt{{pc}}$ encodes a program counter. $M_2$ is encoded as mapping the actual system state (represented with variables $\texttt{cte}$ and $\texttt{he}$) to different estimated states (represented with variables $\mathtt{{cte\_est}}$ and $\mathtt{{he\_est}}$). The transition probabilities are empirically estimated based on profiling the DNN; for simplicity we update $\mathtt{{cte\_est}}$ and $\mathtt{{he\_est}}$ in sequence. The monitor maintains its state using variable $\mathtt{{Q}}$ (initially $\mathtt{{0}}$); it transitions to its next state after $\mathtt{{cte\_est}}$ and $\mathtt{{he\_est}}$ have been updated; the abort state ($\mathtt{{Q=-1}}$) traps behaviours that are not allowed by the assumption; there are no outgoing transitions from such an abort state.