Hybrid algorithms with active set prediction for solving linear inequalities in a least squares sense

Abstract

Inspired by the hybrid algorithm proposed by Dax (Numer. Algor. 50, 97–114, 2009), we attempt to establish three kinds of strategies for solving the linear inequalities in a least squares sense. It can avoid inefficient iterations by active set prediction. Three different formulas are designed in terms of the identification property. A distinctive feature of our switching strategies is the adaptive estimates of the optimal active set. Extensive numerical experiments show the efficiency of hybrid algorithms, especially for the inconsistent system of linear inequalities.

Appendix:: Proofs of Corollaries 1–3

1.1 1.1 Proof of Corollary 1

Proof

Let d_k be the solution of (2.2) and by the assumption that A is full column rank, we have

$$ d_{k}=R^{-1}Q^{T}N_{k}r_{k}. $$

For the next iteration x_k+ 1 = x_k + d_k, the corresponding residual vector is

$$ r_{k+1}=b-Ax_{k+1}=r_{k}-Ad_{k}=(I-QQ^{T}N_{k})r_{k}. $$

(49)

Theorem 2 implies the following fact for sufficiently large k that there exists a positive integer k₀ such that

$$ N_{k_{0}}=N_{k}, \quad \forall \ k \geq k_{0}. $$

By simple recurrences, the result (36) holds for any positive integer n_s. □

1.2 1.2 Proof of Corollary 2

Proof

For the actual reduction δL_k defined in (38), it follows from the last equality in (19) that

$$ \begin{array}{@{}rcl@{}} \delta L_{k}&=&\Vert A(x_{k}-x_{k-1}) \Vert^{2} - (\Vert b_{\mathcal{I}_{k-1}}-A_{\mathcal{I}_{k-1}}x_{k}\Vert^{2} - \Vert (b_{\mathcal{I}_{k-1}}-A_{\mathcal{I}_{k-1}}x_{k})_{+}\Vert^{2} )\\ &\quad & - (\Vert A_{\overline{\mathcal{I}}_{k-1}}(x_{k}-x_{k-1})\Vert^{2}-\Vert (b_{\overline{\mathcal{I}}_{k-1}}-A_{\overline{\mathcal{I}}_{k-1}}x_{k})_{+}\Vert^{2}). \end{array} $$

(50)

Since $\Vert b_{\mathcal {I}_{k-1}}-A_{\mathcal {I}_{k-1}}x_{k}\Vert ^{2} \geq \Vert (b_{\mathcal {I}_{k-1}}-A_{\mathcal {I}_{k-1}}x_{k})_{+}\Vert ^{2}$, the inequality (40) can be derived by (39) and (50).

For the sufficiently large k, the identification property of fixed matrix algorithm implies that there is a positive integer k₀ such that

$$ \mathcal{I}(x_{k})=\mathcal{I}(x_{k-1}), \quad k\geq k_{0}, $$

which implies that

$$ \Vert b_{I_{k-1}}-A_{I_{k-1}}x_{k}\Vert^{2} = \Vert (b_{I_{k-1}}-A_{I_{k-1}}x_{k})_{+}\Vert^{2}, $$

and

$$ \Vert (b_{\overline{I}_{k-1}}-A_{\overline{I}_{k-1}}x_{k})_{+} \Vert^{2}=0 $$

holds for all positive integer k ≥ k₀. Then, the equality (41) follows the definitions of δE_k and δL_k. □

1.3 1.3 Proof of Corollary 3

Proof

Since x_k solves the least squares problem (18), then we have the normal equation

$$ A^{T}Ax_{k}=A^{T}(b+z_{k-1}). $$

(51)

The Moore-Penrose pseudo-inverse of A can be expressed as A⁺ = (A^TA)^− 1A^T due to the fact that A is full column rank, then it follows from (51) that

$$ x_{k}-x_{k-1}=A^{+}(z_{k-1}-z_{k-2}). $$

(52)

Let $\bar {N}_{k-1}=I-N_{k-1}$, where N_k− 1 is the sign matrix defined in Corollary 1. Consequently, $\bar {N}_{k-1}$ is the sign matrix of the vector Ax_k− 1 − b and

$$ \bar{N}_{k-1}(Ax_{k-1}-b)=(Ax_{k-1}-b)_{+}=z_{k-1}. $$

(53)

Theorem 2 shows that $\bar {N}_{k-1}=\bar {N}_{k-2}$ holds for sufficiently large k. Moreover, by combining (52) with (53), we get

$$ x_{k}-x_{k-1}=A^{+}\bar{N}_{k-1}A(x_{k-1}-x_{k-2}), $$

(54)

which yields

$$ \frac{\Vert x_{k}-x_{k-1} \Vert}{\Vert x_{k-1}-x_{k-2} \Vert} \leq \Vert A^{+}\bar{N}_{k-1}A \Vert, $$

where ∥⋅∥ denotes the spectrum norm of the matrix $A^{+}\bar {N}_{k-1}A$. The result in (42) is a direct consequence of the observation that the maximal eigenvalue of the positive semidefinite matrix $A^{+}\bar {N}_{k-1}A$ is just 1.

We assume that the sequence {x_k} satisfies (54) after a sufficiently large k₀. For simplicity, we denote $B=A^{+}\bar {N}_{k-1}A$, and we have

$$ {\Delta}_{k}=x_{k}-x_{k-1}=B^{k-k_{0}}(x_{k_{0}}-x_{k_{0}-1})=B^{k-k_{0}}{\Delta}_{0}. $$

Since $A^{+}\bar {N}_{k-1}A$ is symmetric matrix and can be diagonalizable, there exists an orthogonal matrix S such that B = S^TΛS, where Λ is a diagonal matrix consisting of the eigenvalues λ₁,...,λ_n. By denoting SΔ₀ = (ν₁,⋯ ,ν_n)^T, we have

$$ \Vert {\Delta}_{k} \Vert^{2} = \Vert S^{T} {\Lambda}^{k-k_{0}}S{\Delta}_{0} \Vert^{2}={\sum}_{i=1}^{n}(\nu_{i}\lambda_{i}^{k-k_{0}})^{2}. $$

Hence, we can obtain the ratio of contraction

$$ \frac{c_{k+1}}{c_{k}} = \frac{\Vert {\Delta}_{k+1}\Vert \Vert {\Delta}_{k-1}\Vert }{\Vert {\Delta}_{k}\Vert^{2}}. $$

By Cauchy-Schwarz inequality

$$ \left( \sum\limits_{i=1}^{n}(\nu_{i}\lambda_{i}^{k-k_{0}+1})^{2}\right)\left( \sum\limits_{i=1}^{n}(\nu_{i}\lambda_{i}^{k-k_{0}-1})^{2}\right) \geq \left( \sum\limits_{i=1}^{n}(\nu_{i}\lambda_{i}^{k-k_{0}})^{2}\right)^{2}, $$

we immediately get c_k+ 1 ≥ c_k. Hence, the contraction c_k keeps increasing until it approaches 1. □