Formal Verification of Neural Network Controllers for Collision-Free Flight

Abstract

We investigate a method for formally verifying the absence of adversarial examples in a neural network controller. Our approach applies to networks with piecewise affine activation units, which may be encoded symbolically as a piecewise affine mapping from inputs to outputs. The approach rests on characterizing and bounding a critical subset of the state space where controller action is required, partitioning this critical subset, and using satisfiability modulo theories (SMT) to prove nonexistence of safety counterexamples on each of the resulting partition elements. We demonstrate this approach on a simple collision avoidance neural network controller, trained with reinforcement learning to avoid collisions in a simplified simulated environment. After encoding the network weights in SMT, we formally verify safety of the neural network controller on a subset of the critical partition elements, and determine that the rest of the critical set partition elements are potentially unsafe. We further experimentally confirm the existence of actual adversarial collision scenarios in 90% of the identified potentially unsafe critical partition elements, indicating that our approach is reasonably tight.

A. Schmidt—This work was supported through JHU/APL internal R&D funds.

The views expressed herein are solely those of the authors, and no official support or endorsement by the Defense Nuclear Facilities Safety Board or the U.S. Government is intended or should be inferred.

Appendix

The minimum (maximum) relative acceleration $a_\text {min}$ ($a_\text {max}$) allowed so that the intruder passes safely above (below) the ownship is given by the following functions

$$\begin{aligned} a_\text {min}(z,v,t_1,t_2)&\equiv \inf \{ a \mid (a,z,v,t_1,t_2) \in A \}\end{aligned}$$

(18)

$$\begin{aligned} a_\text {max}(z,v,t_1,t_2)&\equiv \sup \{ a \mid (a,z,v,t_1,t_2) \in B \}, \end{aligned}$$

(19)

where

$$\begin{aligned} A&\equiv \left\{ (a,z,v,t_1,t_2) \mid \forall t\in [t_1, t_2],~z + vt + \frac{at^2}{2} > H \right\} \nonumber \\&= \left\{ (a,z,v,t_1,t_2) \mid \bigvee _{i=1}^6 \psi _A^{(1)}(a,z,v,t_1,t_2) \right\} \subseteq \mathbf {R}^5 \end{aligned}$$

(20)

$$\begin{aligned} B&\equiv \left\{ (a,z,v,t_1,t_2) \mid \forall t\in [t_1, t_2],~z + vt + \frac{at^2}{2} < -H \right\} \nonumber \\&= \left\{ (a,z,v,t_1,t_2) \mid \bigvee _{i=1}^6 \psi _B^{(i)}(a,z,v,t_1,t_2) \right\} \subseteq \mathbf {R}^5, \end{aligned}$$

(21)

$$\begin{aligned} \psi _A^{(1)}&\equiv a=0\wedge t_1 v+z>H\wedge H<t_2 v+z \\ \psi _A^{(2)}&\equiv v=0\wedge a t_1^2+2 z>2\,H\wedge a t_2^2+2 z>2\,H \\ \psi _A^{(3)}&\equiv a \left( a t_1+v\right)>0\wedge a t_1^2+2 t_1 v+2 z>2\,H \wedge a t_2^2+2 t_2 v+2 z>2\,H \\ \psi _A^{(4)}&\equiv a t_1^2+2 t_1 v+2 z>2\,H\wedge a t_2^2+2 t_2 v+2 z>2\,H \wedge a v\ge 0\wedge a\ne 0 \\ \psi _A^{(5)}&\equiv a t_1^2+2 t_1 v+2 z>2\,H\wedge a t_2^2+2 t_2 v+2 z>2\,H \wedge a \left( 2 a (H-z)+v^2\right)<0 \\ \psi _A^{(6)}&\equiv a t_1^2+2 t_1 v+2 z>2\,H\wedge a t_2^2+2 t_2 v+2 z>2\,H \wedge a \left( a t_2+v\right) <0, \end{aligned}$$

$$\begin{aligned} \psi _B^{(1)}&\equiv a=0\wedge H+t_1 v+z<0\wedge H+t_2 v+z<0 \\ \psi _B^{(2)}&\equiv v=0\wedge a t_1^2+2 (H+z)<0\wedge a t_2^2+2 (H+z)<0 \\ \psi _B^{(3)}&\equiv a \left( a t_1+v\right) >0\wedge a t_1^2+2 (H+z)+2 t_1 v<0 \wedge a t_2^2+2 (H+z)+2 t_2 v<0 \\ \psi _B^{(4)}&\equiv a t_1^2+2 (H+z)+2 t_1 v<0 \wedge a t_2^2+2 (H+z)+2 t_2 v<0 \wedge a v\ge 0 \wedge a\ne 0 \\ \psi _B^{(5)}&\equiv a \left( 2 a (H+z)-v^2\right)<0 \wedge a t_1^2+2 (H+z)+2 t_1 v<0 \\&\qquad \wedge a t_2^2+2 (H+z)+2 t_2 v<0 \\ \psi _B^{(6)}&\equiv a t_1^2+2 (H+z)+2 t_1 v<0\wedge a \left( a t_2+v\right)<0 \wedge a t_2^2+2 (H+z)+2 t_2 v<0 \end{aligned}$$

Note that the sets A and B are semialgebraic. For the decomposed dynamics, we have $t_2=t_1+T$, where T is the fixed horizontal conflict duration. The infimum in $a_\text {min}$ be determined by computing the infimum over each semialgebraic component $\psi _A^{(i)}$ and keeping track of the valid sets. Similarly, the supremum in $a_\text {max}$ can be found by maximizing a over each $\psi _B^{(i)}$.