Reluplex: a calculus for reasoning about deep neural networks

Abstract

Deep neural networks have emerged as a widely used and effective means for tackling complex, real-world problems. However, a major obstacle in applying them to safety-critical systems is the great difficulty in providing formal guarantees about their behavior. We present a novel, scalable, and efficient technique for verifying properties of deep neural networks (or providing counter-examples). The technique is based on the simplex method, extended to handle the non-convex Rectified Linear Unit (ReLU) activation function, which is a crucial ingredient in many modern neural networks. The verification procedure tackles neural networks as a whole, without making any simplifying assumptions. We evaluated our technique on a prototype deep neural network implementation of the next-generation airborne collision avoidance system for unmanned aircraft (ACAS Xu). Results show that our technique can successfully prove properties of networks that are an order of magnitude larger than the largest networks that could be verified previously.

Appendices

Appendix

1.1 Verifying properties in DNNs with ReLUs is NP-complete

Let N be a DNN with ReLUs and let $\varphi$ denote a property that is a conjunction of linear constraints on the inputs $\mathbf {x}$ and outputs $\mathbf {y}$ of N, i.e. $\varphi = \varphi _1(\mathbf {x})\wedge \varphi _2(\mathbf {y})$. We say that $\varphi$ is satisfiable on N if there exists an assignment $\alpha {}$ for the variables $\mathbf {x}$ and $\mathbf {y}$ such that $\alpha {}(\mathbf {y})$ is the result of propagating $\alpha {}(\mathbf {x})$ through N and $\alpha {}$ satisfies $\varphi$.

Claim

The problem of determining whether $\varphi$ is satisfiable on N for a given DNN N and a property $\varphi$ is NP-complete.

Proof

We first show that the problem is in NP. A satisfiability witness is simply an assignment $\alpha {}(\mathbf {x})$ for the input variables $\mathbf {x}$. This witness can be checked by feeding the values for the input variables forward through the network, obtaining the assignment $\alpha {}(\mathbf {y})$ for the output values, and checking whether $\varphi _1(\mathbf {x})\wedge \varphi _2(\mathbf {y})$ holds under the assignment $\alpha {}$.

Next, we show that the problem is NP-hard, using a reduction from the 3-SAT problem. We will show how any 3-SAT formula $\psi$ can be transformed into a DNN with ReLUs N and a property $\varphi$, such that $\varphi$ is satisfiable on N if and only if $\psi$ is satisfiable.

Let $\psi = C_1\wedge C_2\wedge \ldots \wedge C_n$ denote a 3-SAT formula over variable set $X = \{x_1,\ldots , x_k\}$, i.e. each $C_i$ is a disjunction of three literals $q_i^1 \vee q_i^2 \vee q_i^3$ where the q’s are variables from X or their negations. The question is to determine whether there exists an assignment $a:X\rightarrow \{0,1\}$ that satisfies $\psi$, i.e. that satisfies all the clauses simultaneously.

For simplicity, we first show the construction assuming that the input nodes take the discrete values 0 or 1. Later we will explain how this limitation can be relaxed, so that the only limitation on the input nodes is that they be in the range [0, 1].

We begin by introducing the disjunction gadget which, given nodes $q_1,q_2,q_3\in \{0,1\}$, outputs a node $y_i$ that is 1 if $q_1+ q_2+ q_3\ge 1$ and 0 otherwise. The gadget is shown below for the case that the $q_i$ literals are all variables (i.e. not negations of variables):

The disjunction gadget can be regarded as calculating the expression

$$\begin{aligned} y_i = 1 - \max {}(0, 1 - \sum _{j=1}^3q_i^j) \end{aligned}$$

If there is at least one input variable set to 1, $y_i$ will be equal to 1. If all inputs are 0, $y_i$ will be equal to 0. The crux of this gadget is that the ReLU operator allows us to guarantee that even if there are multiple inputs set to 1, the output $y_i$ will still be precisely 1.

In order to handle any negative literals $q_i^j\equiv \lnot x_j$, before feeding the literal into the disjunction gadget we first use a negation gadget:

This gadget simply calculates $1-x_j$, and then we continue as before.

The last part of the construction involves a conjunction gadget:

Assuming all nodes $y_1,\ldots ,y_n$ are in the domain $\{0,1\}$, we require that node y be in the range [n, n]. Clearly this holds only if $y_i=1$ for all i.

Finally, in order to check whether all clauses $C_1,\ldots ,C_n$ are simultaneously satisfied, we construct a disjunction gadget for each of the clauses (using negation gadgets for their inputs as needed), and combine them using a conjunction gadget:

where the input variables are mapped to each $t_i$ node according to the definition of clause $C_i$. As we discussed before, node $y_i$ will be equal to 1 if clause $C_i$ is satisfied, and will be 0 otherwise. Therefore, node y will be in the range [n, n] if and only if all clauses are simultaneously satisfied. Consequently, an input assignment $a:X\rightarrow \{0,1\}$ satisfies the input and output constraints on the network if and only if it also satisfies the original $\psi$, as needed.

The construction above is based on the assumption that we can require that the input nodes take values in the discrete set $\{0,1\}$, which does not fit our assumption that $\varphi _1(\mathbf {x})$ is a conjunction of linear constraints. We show now how this requirement can be relaxed.

Let $\epsilon >0$ be a very small number. We set the input range for each variable $x_i$ to be [0, 1], but we will ensure that any feasible solution has $x_i\in [0,\frac{\epsilon }{2}]$ or $x_i\in [1-\frac{\epsilon }{2}, 1]$. We do this by adding to the network for each $x_i$ an auxiliary gadget that uses ReLU nodes to compute the expression

$$\begin{aligned} \max {}(0, \epsilon - x) + \max {}(0, x - 1 + \epsilon ), \end{aligned}$$

and requiring that the output node of this gadget be in the range $[\frac{\epsilon }{2}, \epsilon ]$. It is straightforward to show that for $x\in [0,1]$, this holds if and only if $x\in [0,\frac{\epsilon }{2}]$ or $x\in [1-\frac{\epsilon }{2},1]$.

The disjunction gadgets in our construction then change accordingly. The $y_i$ nodes at the end of each gadget will no longer take just the discrete values $\{0,1\}$, but instead be in the range $[0,3\cdot \frac{\epsilon }{2}]$ if all inputs were in the range $[0,\frac{\epsilon }{2}]$, or in the range $[1-\frac{\epsilon }{2},1]$ if at least one input was in the range $[1-\frac{\epsilon }{2},1]$.

If every input clause has at least one node in the range $[1-\frac{\epsilon }{2},1]$ then all $y_i$ nodes will be in the range $[1-\frac{\epsilon }{2},1]$, and consequently y will be in the range $[n(1-\frac{\epsilon }{2}), n]$. However, if at least one clause does not have a node in the range $[1-\frac{\epsilon }{2},1]$ then y will be smaller than $n(1-\frac{\epsilon }{2})$ (for $\epsilon < \frac{2}{n+3}$). Thus, by requiring that $y\in [n(1-\frac{\epsilon }{2}), n]$, the input and output constraints will be satisfiable on the network if and only if $\psi$ is satisfiable; and the satisfying assignment can be constructed by treating every $x_i\in [0,\frac{\epsilon }{2}]$ as 0 and every $x_i\in [1-\frac{\epsilon }{2}, 1]$ as 1. $\square$

1.2 The Reluplex calculus is sound and complete

We define a derivation tree as a tree where each node is a configuration whose children (if any) are obtained by applying to it one of the derivation rules. A derivation tree $D$ derives a derivation tree $D'$ if $D'$ is obtained from $D$ by applying exactly one derivation rule to one of $D$’s leaves. A derivation is a sequence $D_i$ of derivation trees such that $D_0$ has only a single node and each $D_i$ derives $D_{i+1}$. A refutation is a derivation ending in a tree, all of whose leaves are $\texttt {UNSAT}{}$. A witness is a derivation ending in a tree, at least one of whose leaves is $\texttt {SAT}{}$. If $\phi$ is a conjunction of atoms, we say that $\mathcal {D}$ is a derivation from $\phi$ if the initial tree in $\mathcal {D}$ contains the configuration initialized from $\phi$. A calculus is sound if, whenever a derivation $\mathcal {D}$ from $\phi$ is either a refutation or a witness, $\phi$ is correspondingly unsatisfiable or satisfiable, respectively. A calculus is complete if there always exists either a refutation or a witness starting from any $\phi$.

In order to prove that the Reluplex calculus is sound, we first prove the following lemmas:

Lemma 1

Let $\mathcal {D}$ denote a derivation starting from a derivation tree $D_0$ with a single node $s_0 = \langle \mathcal {B}_0, T_0, l_0, u_0, \alpha {}_0, R_0 \rangle$. Then, for every derivation tree $D_i$ appearing in $\mathcal {D}$, and for each node $s = \langle \mathcal {B}, T, l, u, \alpha {}, R\rangle$ appearing in $D_i$ (except for the distinguished nodes SAT and UNSAT), the following properties hold:

• (i) if an assignment satisfies T, then it also satisfies $T_0$; and

• (ii) the assignment $\alpha {}$ satisfies T (i.e., $\alpha {}$ satisfies all equations in T).

Proof

The proof is by induction on i. For $i = 0$, the claim holds trivially (recall that $\alpha {}_0$ assigns every variable to 0). Now, suppose the claim holds for some i and consider $D_{i+1}$. $D_{i+1}$ is equivalent to $D_i$ except for the addition of one or more nodes added by the application of a single derivation rule d to some node s with tableau T and assignment $\alpha$. Because s appears in $D_i$, we know by the induction hypothesis that an assignment that satisfies T also satisfies $T_0$, and that $\alpha {}$ satisfies T. Let $s'$ be a new node (not a distinguished node SAT or UNSAT) with tableau $T'$ and assignment $\alpha {}'$, introduced by the rule d. Note that d cannot be $\mathsf {ReluSuccess}$ , ${\mathsf {Failure}}_\mathsf{1}{}$, or ${\mathsf {Failure}}_\mathsf{2}$, as these introduce only distinguished nodes; note also that if d is $\mathsf {deriveLowerBound}$ or $\mathsf {deriveUpperBound}$ then both the tableau and the assignment are unchanged, so both properties are trivially preserved.

Suppose d is $\mathsf {Pivot}_{1}$, $\mathsf {Pivot}_{2}$ or $\mathsf {PivotForRelu}$ . For any of these rules, $\alpha {}' = \alpha {}$ and $T'=\textit{pivot}{}(T,i,j)$ for some i and j. Observe that by definition of the pivot operation, the equations of T logically entail those of $T'$ and vice versa, and so they are satisfied by exactly the same assignments. Alternatively, suppose d is $\mathsf {ReluSplit}$ . For the child node corresponding to the inactive case ($u(x_i):=\min (u(x_i),0)$, $l(x_j):=\max (l(x_j),0)$ and $u(x_j):=\min (u(x_j),0)$), the tableau and assignment are unchanged. For the active case ($l(x_i):=\max (l(x_i),0)$), the tableau and assignment are changed by the addEq operation. This operation adds a single equation with a fresh variable as its left hand side, and then extends the assignment to assign this fresh variable a value that satisfies the new equation; the assignments of all other variables are unchanged. From these observations, both properties follow easily.

The remaining cases are when d is $\mathsf {Update}$ , $\mathsf {Update}_{b}$ or $\mathsf {Update}_{f}$ . For these rules, $T' = T$, from which property (i) follows trivially. For property (ii), we first note that $\alpha {}'=\textit{update}{}(\alpha {},x_i,\delta )$ for some i and $\delta$. By definition of the update operation, because $\alpha {}$ satisfied the equations of T, $\alpha {}'$ continues to satisfy these equations and so (because $T' = T$) $\alpha {}'$ also satisfies $T'$. $\square$

Lemma 2

Let $\mathcal {D}$ denote a derivation starting from a derivation tree $D_0$ with a single node $s_0 = \langle \mathcal {B}_0, T_0, l_0, u_0, \alpha {}_0, R_0 \rangle$. If there exists an assignment $\alpha {}^*$ (not necessarily $\alpha {}_0$) such that $\alpha {}^*$ satisfies $T_0$, for every pair $\langle x_i, x_j\rangle \in R$ it holds that $\alpha ^*(x_j)=\max (0,\alpha ^*(x_i))$, and $l_0(x_i)\le \alpha {}^*(x_i)\le u_0(x_i)$ for all i, then for each derivation tree $D_i$ appearing in $\mathcal {D}$ at least one of these two properties holds:

• (i) $D_i$ has a SAT leaf.

• (ii) $D_i$ has a leaf $s = \langle \mathcal {B}, T, l, u, \alpha {}, R\rangle$ (that is not a distinguished node SAT or UNSAT) such that $l(x_i)\le \alpha {}^*(x_i)\le u(x_i)$ for all i, and $\alpha ^*$ satisfies T.

Proof

The proof is again by induction on i. For $i=0$, property (ii) holds trivially. Now, suppose the claim holds for some i and consider $D_{i+1}$. $D_{i+1}$ is equivalent to $D_i$ except for the addition of one or more nodes added by the application of a single derivation rule d to a leaf s of $D_i$.

Due to the induction hypothesis, we know that $D_i$ has a leaf $\bar{s}$ that is either a SAT leaf or that satisfies property (ii). If $\bar{s}\ne s$, then $\bar{s}$ also appears as a leaf in $D_{i+1}$, and the claim holds. We will show that the claim also holds when $\bar{s}=s$. Because none of the derivation rules can be applied to a SAT or UNSAT node, we know that node s is not a distinguished SAT or UNSAT node, and we denote $s = \langle \mathcal {B}, T, l, u, \alpha {}, R\rangle$.

If d is $\mathsf {ReluSuccess}$ , $D_{i+1}$ has a SAT leaf and property (i) holds. Suppose d is $\mathsf {Pivot}_{1}$, $\mathsf {Pivot}_{2}$, $\mathsf {PivotForRelu}$ , $\mathsf {Update}$ , $\mathsf {Update}_{b}$ or $\mathsf {Update}_{f}$ . In any of these cases, node s has a single child in $D_{i+1}$, which we denote $s' = \langle \mathcal {B}', T', l', u', \alpha {}', R' \rangle$. By definition of these derivation rules, $l'(x_j)=l(x_j)$ and $u'(x_j)=u(x_j)$ for all j. Further, $T'$ is either identical or equivalent to T. Because node s satisfies property (ii), we get that $s'$ is a leaf that satisfies property (ii), as needed.

Suppose that d is $\mathsf {ReluSplit}$ , applied to a pair $\langle x_i,x_j\rangle \in R$. Node s has two children in $D_{i+1}$: a state $s^-$ in which the upper bounds for $x_i$ and $x_j$ have been decreased to 0 if they were previously positive, and the lower bound for $x_j$ has been increased to 0 if it was previously negative; and a state $s^+$ in which the lower bound for $x_i$ has been increased to 0 if it was previously negative, and the tableau has been extended to include the equation $x_j=x_i$. It is straightforward to see that if $\alpha {}^*(x_i)\le 0$, then property (ii) holds for $s^-$; and that if $\alpha {}^*(x_i)\ge 0$, then property (ii) holds for $s^+$. In particular, in the latter case, $\alpha ^*(x_j)=\max (0,\alpha ^*(x_i))$ combined with $\alpha ^*(x_i)\ge 0$ implies that the new equation in T, namely $x_j=x_i$, is satisfied (we assume without loss of generality that $\alpha ^*$ assigns 0 to all variables introduced by addEq). Either way, $D_{i+1}$ has a leaf for which property (ii) holds, as needed.

Next, consider the case where d is $\mathsf {deriveLowerBound}$ (the $\mathsf {deriveUpperBound}$ case is symmetrical and is omitted). Node s has a single child in $D_{i+1}$, which we denote $s' = \langle \mathcal {B}', T', l', u', \alpha {}', R' \rangle$. Because the $\mathsf {deriveLowerBound}$ and $\mathsf {deriveUpperBound}$ rules cannot be applied to the distinguished SAT node, property (ii) must hold for s. Let $x_i$ denote the variable to which $\mathsf {deriveLowerBound} {}$ was applied. By definition, $l'(x_i)\ge l(x_i)$, and all other variable bounds are unchanged between s and $s'$. Thus, it suffices to show that $\alpha {}^*(x_i)\ge l'(x_i)$. Because property (ii) holds for s, $\alpha {}^*$ satisfies T; and by the induction hypothesis, $l(x_j)\le \alpha {}^*(x_j)\le u(x_j)$ for all j. The fact that $\alpha {}^*(x_i)\ge l'(x_i)$ then follows directly from the guard condition of $\mathsf {deriveLowerBound}$ .

The remaining two cases are when d is the ${\mathsf {Failure}}_\mathsf{1}{}$ or ${\mathsf {Failure}}_\mathsf{2}$ rule. Because these rules are not applicable to the distinguished SAT node, it follows that property (ii) holds for s. Suppose towards contradiction that in this case, the ${\mathsf {Failure}}_\mathsf{1}{}$ rule is applicable to some variable $x_i$, and suppose (without loss of generality) that $\alpha {}(x_i) < l(x_i)$. By the inductive hypothesis, we know that $l(x_j)\le \alpha {}^*(x_j)\le u(x_j)$ for all j, and by property (ii) we know that $\alpha {}^*$ satisfies T. Consequently, there must be a variable $x_k$ such that $(T_{i,k}>0\ \wedge \ \alpha {}(x_k)<\alpha {}^*(x_k))$, or $(T_{i,k}<0\ \wedge \ \alpha {}(x_k)>\alpha {}^*(x_k))$. But because all variables under $\alpha {}^*$ are within their bounds, it follows that $slack^+(x_i)\ne \emptyset$, which is contradictory to the fact that the ${\mathsf {Failure}}_\mathsf{1}{}$ rule was applicable in s. Next, suppose towards contradiction that the ${\mathsf {Failure}}_\mathsf{2}$ rule is applicable to some variable $x_i$, i.e. that $l(x_i)>u(x_i)$. This immediately contradicts the fact that $l(x_i)\le \alpha {}^*(x_i)\le u(x_i)$. The claim follows. $\square$

Lemma 3

Let $\mathcal {D}$ denote a derivation starting from a derivation tree $D_0$ with a single node $s_0 = \langle \mathcal {B}_0, T_0, l_0, u_0, \alpha {}_0, R_0 \rangle$. Then, for every derivation tree $D_i$ appearing in $\mathcal {D}$, and for each node $s = \langle \mathcal {B}, T, l, u, \alpha {}, R\rangle$ appearing in $D_i$ (except for the distinguished nodes SAT and UNSAT), the following properties hold:

• (i) $R= R_0$; and

• (ii) $l(x_i)\ge l_0(x_i)$ and $u(x_i)\le u_0(x_i)$ for all i.

Proof

Property (i) follows from the fact that none of the derivation rules (except for $\mathsf {ReluSuccess}$ , ${\mathsf {Failure}}_\mathsf{1}{}$, and ${\mathsf {Failure}}_\mathsf{2}$) changes the set R. Property (ii) follows from the fact that the only rules (except for $\mathsf {ReluSuccess}$ , ${\mathsf {Failure}}_\mathsf{1}{}$, and ${\mathsf {Failure}}_\mathsf{2}$) that update lower and upper variable bounds are $\mathsf {deriveLowerBound}$ , $\mathsf {deriveUpperBound}$ , and $\mathsf {ReluSplit}$ , and that these rules can only increase lower bounds or decrease upper bounds.

We are now ready to prove that the Reluplex calculus is sound and complete.

Claim

The Reluplex calculus is sound.

Proof

We begin with the satisfiable case. Let $\mathcal {D}{}$ denote a witness for $\phi$. By definition, the final tree $D{}$ in $\mathcal {D}{}$ has a $\texttt {SAT}{}$ leaf. Let $s_0 = \langle \mathcal {B}_0, T_0, l_0, u_0, \alpha {}_0, R_0 \rangle$ denote the initial state of $D_0$, and let $s = \langle \mathcal {B}, T, l, u, \alpha {}, R\rangle$ denote a state in $D$ to which the $\mathsf {ReluSuccess}$ rule was applied (i.e., a predecessor of a SAT leaf).

By Lemma 1, $\alpha {}$ satisfies $T_0$. Also, by the guard conditions of the $\mathsf {ReluSuccess} {}$ rule, $l(x_i)\le \alpha {}(x_i)\le u(x_i)$ for all i. By property (ii) of Lemma 3, this implies that $l_0(x_i)\le \alpha {}(x_i)\le u_0(x_i)$ for all i. Consequently, $\alpha {}$ satisfies every linear inequality in $\phi$. Finally, we observe that by the conditions of the $\mathsf {ReluSuccess}$ rule, $\alpha {}$ satisfies all the ReLU constraints of s. From property (i) of Lemma 3, it follows that $\alpha {}$ also satisfies the ReLU constraints of $s_0$, which are precisely the ReLU constraints in $\phi$. We conclude that $\alpha {}$ satisfies every constraint in $\phi$, and hence $\phi$ is satisfiable, as needed.

For the unsatisfiable case, it suffices to show that if $\phi$ is satisfiable then there cannot exist a refutation for it. This is a direct result of Lemma 2: if $\phi$ is satisfiable, then there exists an assignment $\alpha {}^*$ that satisfies the initial tableau $T_0$, and for which all variables are within bounds and all ReLU constraints are satisfied. Hence, Lemma 2 implies that any derivation tree in any derivation $\mathcal {D}$ from $\phi$ must have a leaf that is not the distinguished UNSAT leaf. It follows that there cannot exist a refutation for $\phi$. $\square$

Claim

The Reluplex calculus is complete.

Proof

Having shown that the Reluplex calculus is sound, it suffices to show a strategy for deriving a witness or a refutation for every $\phi$ within a finite number of steps. As mentioned in Sect. 3, one such strategy involves two steps: (i) Eagerly apply the $\mathsf {ReluSplit}$ rule, once for each ReLU in R; and (ii) For every leaf of the resulting derivation tree, apply the simplex rules $\mathsf {Pivot}_{1}$, $\mathsf {Pivot}_{2}$, $\mathsf {Update}$ , ${\mathsf {Failure}}_\mathsf{1}{}$, and ${\mathsf {Failure}}_\mathsf{2}$, and the Reluplex rule $\mathsf {ReluSuccess}$ , in a way that guarantees a SAT or an UNSAT configuration is reached within a finite number of steps.

Let $D$ denote the derivation tree obtained after step (i). In every leaf s of $D$, all ReLU connections have been eliminated, meaning that the variable bounds and equations force each ReLU connection to be either active or inactive. This means that every such s can be regarded as a pure simplex problem, and that any solution to that simplex problem is guaranteed to satisfy also the ReLU constraints in s.

The existence of a terminating simplex strategy for deciding the satisfiability of each leaf of $D$ follows from the completeness of the simplex calculus [71]. One such widely used strategy is Bland’s Rule [71]. We observe that although the simplex $\mathsf {Success}$ rule does not exist in Reluplex, it can be directly substituted with the $\mathsf {ReluSuccess}$ rule. This is so because, having applied the $\mathsf {ReluSplit}$ rule on each of the ReLUs, any assignment that satisfies the variable bounds in s also satisfies the ReLU constraints in s.

It follows that for every $\phi$, we can produce a witness or a refutation, as needed. $\square$

1.3 Encoding ReLUs for SMT and LP solvers

We demonstrate the encoding of ReLU nodes that we used for the evaluation conducted using SMT and LP solvers. Let $y=\text {ReLU} {}{}(x)$. In the SMTLIB format, used by all SMT solvers that we tested, ReLUs were encoded using an if-then-else construct:

In LP format this was encoded using mixed integer programming. Using Gurobi’s built-in Boolean type, we defined for every ReLU connection a pair of Boolean variables, b$_{\text {on}}$ and b$_{\text {off}}$, and used them to encode the two possible states of the connection. Taking M to be a very large positive constant, we used the following assertions:

When b$_{\text {on}}$$=1$ and b$_{\text {off}}$$=0$, the ReLU connection is in the active state; and otherwise, when b$_{\text {on}}$$=0$ and b$_{\text {off}}$$=1$, it is in the inactive state.

In the active case, because b$_{\text {off}}$ $= 0$ the third and fourth equations imply that $x=y$ (observe that y is always non-negative). M is very large, and can be regarded as $\infty$; hence, because b$_{\text {on}}$$=1$, the last two equations merely imply that $x,y\le \infty$, and so pose no restriction on the solution.

In the inactive case, b$_{\text {on}}$ $=0$, and so the last two equations force $y=0$ and $x\le 0$. In this case b$_{\text {off}}$$=1$ and so the third and fourth equations pose no restriction on the solution.

Formal definitions for properties $\phi _1$,...,$\phi _{10}$

The units for the ACAS Xu DNNs’ inputs are:

• $\rho$: feet.

• $\theta ,\psi$: radians.

• $v_\text {own}, v_\text {int}$: feet per second.

• $\tau$: seconds.

$\theta$ and $\psi$ are measured counter clockwise, and are always in the range $[-\pi ,\pi ]$. In line with the discussion in Sect. 5, the family of 45 ACAS Xu DNNs are indexed according to the previous action $a_\text {prev}$ and time until loss of vertical separation $\tau$. The possible values for these two indices are:

• 1. $a_\text {prev}$: [Clear-of-Conflict, weak left, weak right, strong left, strong right].

• 2. $\tau$: [0, 1, 5, 10, 20, 40, 60, 80, 100].

We use $N_{x,y}$ to denote the network trained for the x-th value of $a_\text {prev}$ and y-th value of $\tau$. For example, $N_{2,3}$ is the network trained for the case where $a_\text {prev}=$ weak left and $\tau =5$. Using this notation, we now give the formal definition of each of the properties $\phi _1,\ldots ,\phi _{10}$ that we tested.

1.1 Property $\phi _1$

• Description: If the intruder is distant and is significantly slower than the ownship, the score of a COC advisory will always be below a certain fixed threshold.

• Tested on: all 45 networks.

• Input constraints: $\rho \ge 55947.691$, $v_\text {own}\ge 1145$, $v_\text {int}\le 60$.

• Desired output property: the score for COC is at most 1500.

1.2 Property $\phi _2$

• Description: If the intruder is distant and is significantly slower than the ownship, the score of a COC advisory will never be maximal.

• Tested on: $N_{x,y}$ for all $x\ge 2$ and for all y.

• Input constraints: $\rho \ge 55947.691$, $v_\text {own}\ge 1145$, $v_\text {int}\le 60$.

• Desired output property: the score for COC is not the maximal score.

1.3 Property $\phi _3$

• Description: If the intruder is directly ahead and is moving towards the ownship, the score for COC will not be minimal.

• Tested on: all networks except $N_{1,7}$, $N_{1,8}$, and $N_{1,9}$.

• Input constraints: $1500 \le \rho \le 1800$, $-0.06 \le \theta \le 0.06$, $\psi \ge 3.10$, $v_\text {own}\ge 980$, $v_\text {int}\ge 960$.

• Desired output property: the score for COC is not the minimal score.

1.4 Property $\phi _4$

• Description: If the intruder is directly ahead and is moving away from the ownship but at a lower speed than that of the ownship, the score for COC will not be minimal.

• Tested on: all networks except $N_{1,7}$, $N_{1,8}$, and $N_{1,9}$.

• Input constraints: $1500 \le \rho \le 1800$, $-0.06 \le \theta \le 0.06$, $\psi = 0$, $v_\text {own}\ge 1000$, $700 \le v_\text {int}\le 800$.

• Desired output property: the score for COC is not the minimal score.

1.5 Property $\phi _5$

• Description: If the intruder is near and approaching from the left, the network advises “strong right”.

• Tested on: $N_{1,1}$.

• Input constraints: $250 \le \rho \le 400$, $0.2 \le \theta \le 0.4$, $-3.141592 \le \psi \le -3.141592 + 0.005$, $100 \le v_\text {own}\le 400$, $0 \le v_\text {int}\le 400$.

• Desired output property: the score for “strong right” is the minimal score.

1.6 Property $\phi _6$

• Description: If the intruder is sufficiently far away, the network advises COC.

• Tested on: $N_{1,1}$.

• Input constraints: $12000 \le \rho \le 62000$, $(0.7 \le \theta \le 3.141592) \vee (-3.141592 \le \theta \le -0.7)$, $-3.141592 \le \psi \le -3.141592 + 0.005$, $100 \le v_\text {own}\le 1200$, $0 \le v_\text {int}\le 1200$.

• Desired output property: the score for COC is the minimal score.

1.7 Property $\phi _7$

• Description: If vertical separation is large, the network will never advise a strong turn.

• Tested on: $N_{1,9}$.

• Input constraints: $0 \le \rho \le 60760$, $-3.141592 \le \theta \le 3.141592$, $-3.141592 \le \psi \le 3.141592$, $100 \le v_\text {own}\le 1200$, $0 \le v_\text {int}\le 1200$.

• Desired output property: the scores for “strong right” and “strong left” are never the minimal scores.

1.8 Property $\phi _8$

• Description: For a large vertical separation and a previous “weak left” advisory, the network will either output COC or continue advising “weak left”.

• Tested on: $N_{2,9}$.

• Input constraints: $0 \le \rho \le 60760$, $-3.141592 \le \theta \le -0.75\cdot 3.141592$, $-0.1 \le \psi \le 0.1$, $600 \le v_\text {own}\le 1200$, $600 \le v_\text {int}\le 1200$.

• Desired output property: the score for “weak left” is minimal or the score for COC is minimal.

1.9 Property $\phi _9$

• Description: Even if the previous advisory was “weak right”, the presence of a nearby intruder will cause the network to output a “strong left” advisory instead.

• Tested on: $N_{3,3}$.

• Input constraints: $2000 \le \rho \le 7000$, $-0.4 \le \theta \le -0.14$, $-3.141592 \le \psi \le -3.141592+0.01$, $100 \le v_\text {own}\le 150$, $0 \le v_\text {int}\le 150$.

• Desired output property: the score for “strong left” is minimal.

1.10 Property $\phi _{10}$

• Description: For a far away intruder, the network advises COC.

• Tested on: $N_{4,5}$.

• Input constraints: $36000 \le \rho \le 60760$, $0.7 \le \theta \le 3.141592$, $-3.141592 \le \psi \le -3.141592+0.01$, $900 \le v_\text {own}\le 1200$, $600 \le v_\text {int}\le 1200$.

• Desired output property: the score for COC is minimal.