A globally convergent primal-dual active-set framework for large-scale convex quadratic optimization

Abstract

We present a primal-dual active-set framework for solving large-scale convex quadratic optimization problems (QPs). In contrast to classical active-set methods, our framework allows for multiple simultaneous changes in the active-set estimate, which often leads to rapid identification of the optimal active-set regardless of the initial estimate. The iterates of our framework are the active-set estimates themselves, where for each a primal-dual solution is uniquely defined via a reduced subproblem. Through the introduction of an index set auxiliary to the active-set estimate, our approach is globally convergent for strictly convex QPs. Moreover, the computational cost of each iteration typically is only modestly more than the cost of solving a reduced linear system. Numerical results are provided, illustrating that two proposed instances of our framework are efficient in practice, even on poorly conditioned problems. We attribute these latter benefits to the relationship between our framework and semi-smooth Newton techniques.

Appendix: Primal-dual active-set as a semi-smooth newton method

In this appendix, we show that Algorithm 3 is equivalent to a semi-smooth Newton method under certain conditions. The following theorem utilizes the concept of a slant derivative of a slantly differentiable function [25].

Theorem 5

Let $\{(x_k,y_k,z^\ell _k,z^u_k)\}$ be generated by Algorithm 3 with Step 6 employing Algorithm 4, where we suppose that, for all $k$, $({\mathcal A}_k^\ell ,{\mathcal A}_k^u,{\mathcal I}_k,{\mathcal U}_k)$ with ${\mathcal U}_k=\emptyset$ is a feasible partition at the start of Step 3. Then, $\{(x_k,y_k,z^\ell _k,z^u_k)\}$ is the sequence of iterates generated by the semi-smooth Newton method for finding a zero of the function $\mathrm{KKT}$ defined by (2) with initial value $(x_0,y_0,z^\ell _0,z^u_0) = \mathrm{SM}({\mathcal A}^\ell _0,{\mathcal A}^u_0,{\mathcal I}_0,\emptyset )$ and slant derivative $M(a,b)$ of the slantly differentiable function $m(a,b)=\min (a,b)$ defined by

$$\begin{aligned}{}[M(a,b)]_{ij} = {\left\{ \begin{array}{ll} 0 &{}\quad \mathrm{{if }} j\notin \{i,n+i\} \\ 1 &{}\quad \mathrm{{if }} j=i, a_i \le b_j \\ 0 &{}\quad \mathrm{{if }} j=i, a_i > b_j \\ 0 &{}\quad \mathrm{{if }} j=n+i, a_i \le b_j \\ 1 &{}\quad \mathrm{{if }} j=n+i, a_i > b_j. \end{array}\right. } \end{aligned}$$

Proof

To simplify the proof, let us assume that $\ell = -\infty$ so that problem (1) has upper bounds only. This ensures that $z^\ell _k = 0$ and ${\mathcal A}^\ell _k = \emptyset$ for all $k$, so in this proof we remove all references to these quantities. The proof of the case with both lower and upper bounds follows similarly.

Under the assumptions of the theorem, the point $(x_0,y_0,z^u_0) \leftarrow \mathrm{SM}(\emptyset ,{\mathcal A}^u_0,{\mathcal I}_0,\emptyset )$ is the first primal-dual iterate for both algorithms, i.e., Algorithm 3 and the semi-smooth Newton method. Furthermore, it follows from (4)–(6) that

$$\begin{aligned} Hx_0 + c - A^T\!y_0 + z^u_0 = 0 \quad \hbox {and}\quad Ax_0 - b = 0. \end{aligned}$$

(26)

We now proceed to show that both algorithms generate the same subsequent iterate, namely $(x_1,y_1,z^u_1)$. The result then follows as a similar argument can be used to show that both algorithms generate the same iterate $(x_k,y_k,z^u_k)$ for each $k$.

Partitioning the variable indices into four sets, namely $\mathrm{I}$, $\mathrm{II}$, $\mathrm{III}$, and $\mathrm{IV}$, we find:

$$\begin{aligned} \mathrm{I}&:= \{i: i\in {\mathcal I}_0 \;\;\hbox {and}\;\; [x_0]_i \le u_i\} \implies [z^u_0]_i = 0;\end{aligned}$$

(27a)

$$\begin{aligned} \mathrm{II}&:= \{i: i\in {\mathcal A}^u_0 \;\;\hbox {and}\;\; [z^u_0]_i \le 0\} \implies [x_0]_i = u_i;\end{aligned}$$

(27b)

$$\begin{aligned} \mathrm{III}&:= \{i: i\in {\mathcal I}_0 \;\;\hbox {and}\;\; [x_0]_i > u_i\} \implies [z^u_0]_i = 0;\end{aligned}$$

(27c)

$$\begin{aligned} \mathrm{IV}&:= \{i: i\in {\mathcal A}^u_0 \;\;\hbox {and}\;\; [z^u_0]_i > 0\} \implies [x_0]_i = u_i. \end{aligned}$$

(27d)

Here, the implications after each set follow from Step 2 of Algorithm 2. Next, (16) implies

$$\begin{aligned} {\mathcal I}_1 \leftarrow \mathrm{I}\cup \mathrm{II}\;\;\hbox {and}\;\; {\mathcal A}_1 \leftarrow \mathrm{III}\cup \mathrm{IV}. \end{aligned}$$

(28)

Algorithm 3 computes the next iterate as the unique point $(x_1,y_1,z^u_1)$ satisfying

$$\begin{aligned}{}[z^u_1]_{{\mathcal I}_1} = 0, \quad [x_1]_{{\mathcal A}_1} = u_{{\mathcal A}_1}, \quad Hx_1+c-A^T\!y_1 + z^u_1 = 0, \quad \hbox {and}\quad Ax_1 - b = 0.\nonumber \\ \end{aligned}$$

(29)

Now, let us consider one iteration of the semi-smooth Newton method on the function KKT defined by (2) using the slant derivative function $M$. It follows from (27), Table 13, and the definition of $M$ that the semi-smooth Newton system may be written as

$$\begin{aligned} \begin{pmatrix}H_{\mathrm{I},\mathrm{I}} &{} H_{\mathrm{I},\mathrm{II}} &{} H_{\mathrm{I},\mathrm{III}} &{} H_{\mathrm{I},\mathrm{IV}} &{} A_{{\mathcal N},\mathrm{I}}^T\!&{} I &{} 0 &{} 0 &{} 0 \\ H_{\mathrm{II},\mathrm{I}} &{} H_{\mathrm{II},\mathrm{II}} &{} H_{\mathrm{II},\mathrm{III}} &{} H_{\mathrm{II},\mathrm{IV}} &{} A_{{\mathcal N},\mathrm{II}}^T\!&{} 0 &{} I &{} 0 &{} 0 \\ H_{\mathrm{III},\mathrm{I}} &{} H_{\mathrm{III},\mathrm{II}} &{} H_{\mathrm{III},\mathrm{III}} &{} H_{\mathrm{III},\mathrm{IV}} &{} A_{{\mathcal N},\mathrm{III}}^T\!&{} 0 &{} 0 &{} I &{} 0 \\ H_{\mathrm{IV},\mathrm{I}} &{} H_{\mathrm{IV},\mathrm{II}} &{} H_{\mathrm{IV},\mathrm{III}} &{} H_{\mathrm{IV},\mathrm{IV}} &{} A_{{\mathcal N},\mathrm{IV}}^T\!&{} 0 &{} 0 &{} 0 &{} I \\ A_{{\mathcal N},\mathrm{I}} &{} A_{{\mathcal N},\mathrm{II}} &{} A_{{\mathcal N},\mathrm{III}} &{} A_{{\mathcal N},\mathrm{IV}} &{} 0 &{} 0 &{} 0 &{} 0 &{} 0\\ 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} I &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} I &{} 0 &{} 0 \\ 0 &{} 0 &{} -I &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 &{} -I &{} 0 &{} 0 &{} 0 &{} 0 &{} 0\end{pmatrix} \begin{pmatrix}\varDelta x_\mathrm{I}\\ \varDelta x_{\mathrm{II}} \\ \varDelta x_{\mathrm{III}} \\ \varDelta x_{\mathrm{IV}} \\ -\varDelta y\\ \varDelta z_\mathrm{I}\\ \varDelta z_{\mathrm{II}} \\ \varDelta z_{\mathrm{III}} \\ \varDelta z_{\mathrm{IV}} \end{pmatrix} = -\begin{pmatrix}0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ [z^u_0]_\mathrm{II}\\ [u-x_0]_\mathrm{III}\\ 0\end{pmatrix}. \end{aligned}$$

(30)

The first five block equations of (30) combined with (26) yield

$$\begin{aligned} Ax_1 - b&= A(x_0+\varDelta x) - b = Ax_0 - b + A\varDelta x= 0 \;\;\hbox {and} \end{aligned}$$

(31a)

$$\begin{aligned} Hx_1 + c - A^T\!y_1 + z^u_1&= H(x_0+\varDelta x) + c - A^T\!(y_0+\varDelta y) + z^u_0+\varDelta z= 0, \end{aligned}$$

(31b)

while the last four blocks of equations of (30) and (27) imply

$$\begin{aligned} \varDelta z_\mathrm{I}= 0&\implies [z^u_1]_\mathrm{I}= [z^u_0+\varDelta z]_\mathrm{I}= 0 \end{aligned}$$

(32)

$$\begin{aligned} \varDelta z_\mathrm{II}= -[z^u_0]_\mathrm{II}&\implies [z^u_1]_\mathrm{II}= [z^u_0+\varDelta z]_\mathrm{II}= 0 \end{aligned}$$

(33)

$$\begin{aligned} \varDelta x_\mathrm{III}= [u-x_0]_\mathrm{III}&\implies [x_1]_\mathrm{III}= [x_0+\varDelta x]_\mathrm{III}= u_\mathrm{III}\end{aligned}$$

(34)

$$\begin{aligned} \varDelta x_\mathrm{IV}= 0&\implies [x_1]_\mathrm{IV}= [x_0+\varDelta x]_\mathrm{IV}= u_\mathrm{IV}\end{aligned}$$

(35)

so that

$$\begin{aligned}{}[z^u_1]_{{\mathcal I}_1} = 0 \;\;\hbox {and}\;\; [x_1]_{{\mathcal A}_1} = u_{{\mathcal A}_1}. \end{aligned}$$

(36)

It now follows from (29), (31), and (36) that $(x_1,y_1,z^u_1)$ generated by the semi-smooth Newton method is the same as that generated by Algorithm 3.$\square$

Table 13 Quantities relevant to evaluating the function KKT and computing the slant derivative $M$ at the point $(x_0,y_0,z^u_0)$