Strong Mixed-Integer Programming Formulations for Trained Neural Networks

Abstract

We present an ideal mixed-integer programming (MIP) formulation for a rectified linear unit (ReLU) appearing in a trained neural network. Our formulation requires a single binary variable and no additional continuous variables beyond the input and output variables of the ReLU. We contrast it with an ideal “extended” formulation with a linear number of additional continuous variables, derived through standard techniques. An apparent drawback of our formulation is that it requires an exponential number of inequality constraints, but we provide a routine to separate the inequalities in linear time. We also prove that these exponentially-many constraints are facet-defining under mild conditions. Finally, we study network verification problems and observe that dynamically separating from the exponential inequalities (1) is much more computationally efficient and scalable than the extended formulation, (2) decreases the solve time of a state-of-the-art MIP solver by a factor of 7 on smaller instances, and (3) nearly matches the dual bounds of a state-of-the-art MIP solver on harder instances, after just a few rounds of separation and in orders of magnitude less time.

A Deferred Proofs

1.1 A.1 Proof of Proposition 1

Proof

The result follows from applying Fourier–Motzkin elimination to (5a–5f) to project out the $x^0$, $x^1$, $y^0$, and $y^1$ variables; see [31, Chap. 13] for an explanation of the approach. We start by eliminating the $x^1$, $y^0$, and $y^1$ using the equations in (5a), (5b), and (5c), respectively, leaving only $x^0$.

First, if there is some input component i with $w_i=0$, then $x^0_i$ only appears in the constraints (5d–5e), and so the elimination step produces .

Second, if there is some i with $w_i < 0$, then we introduce an auxiliary variable $\tilde{x}_i$ with the equation $\tilde{x}_i = -x_i$. We then replace $w_i \leftarrow |w_i|$, $L_i \leftarrow -U_i$, and $U_i \leftarrow -L_i$, and proceed as follows under the assumption that $w > 0$.

Applying the Fourier-Motzkin procedure to eliminate $x^0_1$ gives the inequalities

along with the existing inequalities in (5a–5f) where the $x^0_1$ coefficient is zero. Repeating this procedure for each remaining component of $x^0$ yields the linear system

$$\begin{aligned} y \geqslant w \cdot x + b \end{aligned}$$

(7a)

$$\begin{aligned} y \leqslant \sum _{i \in I} w_i x_i - \sum _{i \in I} w_iL_i(1-z) + \left( b + \sum _{i \not \in I} w_iU_i\right) z \quad \forall I \subseteq {\text {supp}}(w) \end{aligned}$$

(7b)

$$\begin{aligned} y \geqslant \sum _{i \in I} w_i x_i - \sum _{i \in I} w_iU_i(1-z) + \left( b + \sum _{i \not \in I} w_iL_i\right) z \quad \forall I \subseteq {\text {supp}}(w) \end{aligned}$$

(7c)

$$\begin{aligned} (x,y,z) \in [L,U] \times \mathbb {R}_{\geqslant 0} \times [0,1]. \end{aligned}$$

(7d)

Moreover, we can show that the family of inequalities (7c) is redundant, and can therefore be removed. Fix some $I \subseteq {\text {supp}}(w)$, and take . If $h(\llbracket \eta \rrbracket \setminus I) \geqslant 0$, we can express the inequality in (7c) corresponding to the set I as a conic combination of the remaining constraints as:

Alternatively, if $h(\llbracket \eta \rrbracket \setminus I) < 0$, we can express the inequality in (7c) corresponding to the set I as a conic combination of the remaining constraints as:

To complete the proof, for any components i where we introduced an auxiliary variable $\tilde{x}_i$, we use the corresponding equation $\tilde{x}_i = -x_i$ to eliminate $x_i$ and replace it $\tilde{x}_i$, giving the result.    $\square$

1.2 A.2 Proof of Proposition 2

Proof

We fix $I = \{\kappa +1,\ldots ,\eta \}$ for some $\kappa$; this is without loss of generality by permuting the rows of the matrices presented below. Additionally, we presume that , which allows us to infer that $\breve{L} = L$ and $\breve{U} = U$. This is also without loss of generality by appropriately interchanging $+$ and − in the definition of the $\tilde{p}^k$ below. In the following, references to (6b) are taken to be references to the inequality in (6b) corresponding to the subset I.

Take the two points $p^0 = (x,y,z) = (L, 0, 0)$ and $p^1 = (U, f(U), 1)$. Each point is feasible with respect to (6a–6c) and satisfies (6b) at equality. Then for some $\epsilon > 0$ and for each $i \in \llbracket \eta \rrbracket \backslash I$, take $\tilde{p}^i = (x,y,z) = (L + \epsilon \mathbf{e}^i, 0, 0)$. Similarly, for each $i \in I$, take $\tilde{p}^i = (x,y,z) = (U - \epsilon \mathbf{e}^i, f(U - \epsilon \mathbf{e}^i), 1)$. From the strict activity assumption, there exists some $\epsilon > 0$ sufficiently small such that each $\tilde{p}^k$ is feasible with respect to (6a–6c) and satisfies (6b) at equality.

This leaves us with $\eta + 2$ feasible points satisfying (6b) at equality; the result then follows by showing that the points are affinely independent. Take the matrix

$$ \begin{pmatrix} p^1 - p^0 \\ \tilde{p}^1 - p^0 \\ \vdots \\ \tilde{p}^\kappa - p^0 \\ \tilde{p}^{\kappa +1} - p^0 \\ \vdots \\ \tilde{p}^\eta - p^0 \end{pmatrix} = \begin{pmatrix} U-L &{} f(U) &{} 1 \\ \epsilon \mathbf{e}^1 &{} 0 &{} 0 \\ \vdots &{} \vdots &{} \vdots \\ \epsilon \mathbf{e}^\kappa &{} 0 &{} 0 \\ U - L - \epsilon \mathbf{e}^{\kappa +1} &{} f(U-\epsilon \mathbf{e}^{\kappa +1}) &{} 1 \\ \vdots &{} \vdots &{} \vdots \\ U - L - \epsilon \mathbf{e}^{\eta } &{} f(U-\epsilon \mathbf{e}^{\eta }) &{} 1 \end{pmatrix} \cong \begin{pmatrix} U - L &{} f(U) &{} 1 \\ \epsilon \mathbf{e}^1 &{} 0 &{} 0 \\ \vdots &{} \vdots &{} \vdots \\ \epsilon \mathbf{e}^\kappa &{} 0 &{} 0 \\ -\epsilon \mathbf{e}^{\kappa +1} &{} -w_{\kappa +1}\epsilon &{} 0 \\ \vdots &{} \vdots &{} \vdots \\ -\epsilon \mathbf{e}^{\eta } &{} -w_\eta \epsilon &{} 0 \end{pmatrix}, $$

where the third matrix is constructed by subtracting the first row to each of row $\kappa +2$ to $\eta +1$ (i.e. those corresponding to $\tilde{p}^i-p^0$ for $i > \kappa$), and is taken to mean congruency with respect to elementary row operations. If we permute the last column (corresponding to the z variable) to the first column, we observe that the resulting matrix is upper triangular with a nonzero diagonal, and so has full row rank. Therefore, the starting matrix also has full row rank, as we only applied elementary row operations, and therefore the $\eta +2$ points are affinely independent, giving the result.    $\square$