Another look at linear programming for feature selection via methods of regularization

Abstract

We consider statistical procedures for feature selection defined by a family of regularization problems with convex piecewise linear loss functions and penalties of l ₁ nature. Many known statistical procedures (e.g. quantile regression and support vector machines with l ₁-norm penalty) are subsumed under this category. Computationally, the regularization problems are linear programming (LP) problems indexed by a single parameter, which are known as ‘parametric cost LP’ or ‘parametric right-hand-side LP’ in the optimization theory. Exploiting the connection with the LP theory, we lay out general algorithms, namely, the simplex algorithm and its variant for generating regularized solution paths for the feature selection problems. The significance of such algorithms is that they allow a complete exploration of the model space along the paths and provide a broad view of persistent features in the data. The implications of the general path-finding algorithms are outlined for several statistical procedures, and they are illustrated with numerical examples.

Appendix

Lemma 1

Suppose that $\mathcal {B}^{l+1}:=\mathcal {B}^{l}\cup\{j^{l}\}\setminus\{i^{l}\}$, where $i^{l}:=k^{l}_{{i^{l}_{*}}}$. Let be defined as in (13). Then .

Proof

First observe that

Without loss of generality, the ${i^{l}_{*}}$th column vector ${{\bf A}}_{i^{l}}$ of ${{\bf A}}_{\mathcal {B}^{l}}$ is replaced with ${{\bf A}}_{j^{l}}$ to give ${{\bf A}}_{\mathcal {B}^{l+1}}$. For the ${{\bf A}}_{\mathcal {B}^{l+1}}$,

(27)

where . Thus, we have

$$ {{\bf A}}_{\mathcal {B}^{l+1}}^{-1}{{\bf A}}_{\mathcal {B}^l}= \left [ \begin{array}{c@{\quad}c@{\quad}c@{\quad}c@{\quad}c@{\quad}c} 1&&-u^l_1/{u}^l_{i^l_*}&&\\&\ddots&\vdots&&\\&&1/u^l_{i^l_*}&&\\&&\vdots&\ddots&\\&&-u^l_M/{u}^l_{i^l_*} &&1 \end{array} \right ]. $$

(28)

Then it immediately follows that . Hence, . □

1.1 8.1 Proof of (15)

For l=0,…,J−1, consider the following difference

By the intermediate calculation in Lemma 1, we can show that the difference is $\kappa_{\lambda_{l}}{{\bf A}}^{\top}({{\bf A}}_{\mathcal {B}^{l}}^{-1} )^{\top} {{\bf e}}_{{i^{l}_{*}}}$, where

$$\begin{aligned} \kappa_{\lambda_l} :=&({{c}}_{{\tiny k^l_{{i^l_*}}}} +\lambda_l {{a}}_{{\tiny k^l_{{i^l_*}}}}) -\frac{{{c}}_{j^l}+\lambda_l{{a}}_{j^l}}{u^l_{{i^l_*}}}\\&{}+\sum_{i\in {\tiny (\mathcal {B}^{l+1}\setminus\{j^l\} )}} \frac{({{c}}_i+\lambda_l{{a}}_i){u}^l_i}{u^l_{{i^l_*}}} \\=& \frac{({{\bf c}}_{\mathcal {B}^l}+\lambda_l{{\bf a}}_{\mathcal {B}^l})^\top {{\bf A}}_{\mathcal {B}^l}^{-1} {{\bf A}}_{j^l}-({{c}}_{j^l}+\lambda_l{{a}}_{j^l})}{u^l_{{i^l_*}}} \\=&-\frac{\check{{{c}}}^l_{j^l}+\lambda_l\check{{{a}}}^l_{j^l}}{u^l_{{i^l_*}}}. \end{aligned}$$

Since $\lambda_{l}:= -\check{{{c}}}^{l}_{j^{l}}/ {\check{{{a}}}^{l}_{j^{l}}}$, $\kappa_{\lambda_{l}}=0$, which proves (15).

1.2 8.2 Proof of Theorem 3

Let ${\mathfrak {B}}^{l}:=\mathcal {B}^{l}\cup\{j^{l}\}$ for l=0,…,J−1, and ${\mathfrak {B}}^{J}:=\mathcal {B}^{J}\cup\{N+1\}$, where $\mathcal {B}^{l}$, $\mathcal {B}^{J}$, andj ^l are as defined in the simplex algorithm. We will show that, for any fixed s∈[s _l ,s _l+1) (or s≥s _J ), ${\mathfrak {B}}^{l}$ (or ${\mathfrak {B}}^{J}$) is an optimal basic index set for the LP problem in (10).

For simplicity, let j ^J:=N+1, c _N+1:=0, , and a _N+1:=1. The inverse of

$$\begin{aligned} {\mbox {$\mathbb {A}$}}_{{\mathfrak {B}}^l} =&\left [ \begin{array}{c@{\quad}c} {{\bf A}}_{\mathcal {B}^l} & {{\bf A}}_{j^l}\\ {{\bf a}}_{\mathcal {B}^l}^\top& {{a}}_{j^l} \end{array} \right ] \end{aligned}$$

is given by

for l=0,…,J.

First, we show that ${\mbox {$\mathbb {A}$}}_{{{\mathfrak {B}}^{l}}}$ is a feasible basic index set of (10) for s∈[s _l ,s _l+1], i.e.

(29)

Recalling that , $z^{l}_{j^{l}}=0$, , , and $d^{l}_{j^{l}}=1$, we have

(30)

From and

it can be shown that

Thus, (30) is a convex combination of and for s∈[s _l ,s _l+1], and hence it is non-negative. This proves the feasibility of ${\mbox {$\mathbb {A}$}}_{{\mathfrak {B}}^{l}}$ for s∈[s _l ,s _l+1] and l=0,…,J−1. For s≥s ^J, we have

Next, we prove that ${\mbox {$\mathbb {A}$}}_{{\mathfrak {B}}^{l}}$ is an optimal basic index set of (10) for s∈[s _l ,s _l+1] by showing . For i=1,…,N, the ith element of is

Similarly, for s≥s ^J,

$$\begin{aligned} {{c}}_i-\left [ \begin{array}{c} {{{\bf c}}_{\mathcal {B}^J}}\\0 \end{array} \right ]^\top {\mbox {$\mathbb {A}$}}_{{\mathfrak {B}}^J}^{-1}\left [ \begin{array}{c} {{\bf A}}_i\\{{a}}_i \end{array} \right ] =&{{c}}_i- {{\bf c}}_{\mathcal {B}^J}^\top {{\bf A}}_{\mathcal {B}^J}^{-1}{{\bf A}}_i \\=&\left\{ \begin{array} {l@{\quad}l} \check{{{c}}}^J_i &\mbox{for }i=1,\ldots,N\\0&\mbox{for }i=N+1. \end{array} \right. \end{aligned}$$

Clearly, the optimality condition holds by the non-negativity of all the elements as defined in the simplex algorithm. This completes the proof.

1.3 8.3 Proof of Theorem 4

(i) By (28), we can update the pivot rows of the tableau as follows:

(31)

If $u^{l}_{i}=0$, the ith pivot row of $\mathcal {B}^{l+1}$ is the same as the ith pivot row of . For $i={i^{l}_{*}}$, the ith pivot row of ${\mathcal {B}^{l+1}}$ is $(1/{u}^{l}_{i^{l}_{*}})$ . If $i\neq{i^{l}_{*}}$ and ${u}^{l}_{i}<0$, which imply $-(u^{l}_{i}/{u}^{l}_{{i^{l}_{*}}})>0$, the ith pivot row of since the sum of any two lexicographically positive vectors is still lexicographically positive. According to the tableau update algorithm, we have ${u}^{l}_{{i^{l}_{*}}}>0$, where ${i^{l}_{*}}$ is the index number of the lexicographically smallest pivot row among all the pivot rows for $\mathcal {B}^{l}$ with ${u}^{l}_{i}>0$. For $i\neq{i^{l}_{*}}$ and ${u}^{l}_{i}>0$, by the definition of ${i^{l}_{*}}$, $\mbox{(the ${i^{l}_{*}}$th pivot row of ${\mathcal {B}^{l}}$)}/{u}^{l}_{i^{l}_{*}}\overset{L}{<} \mbox{(the $i$th pivot row of ${\mathcal {B}^{l}}$)}/{u}^{l}_{i}$. This implies

Therefore, all the updated pivot rows are lexicographically positive.

Remark 2

If $z^{l}_{{i^{l}}}=0$, (31) implies that $z^{l}_{k^{l}_{i}}=z^{l+1}_{k^{l}_{i}}$ for $i\neq{i^{l}_{*}}$, $i\in \mathcal {M}$. and $z^{l+1}_{j^{l}}=0$. Hence . On the other hand, if $z^{l}_{i^{l}}>0$, $z^{l+1}_{j^{l}} =(z^{l}_{i^{l}}/{u}^{l}_{j^{l}})>0$ while $z^{l}_{j^{l}}=0$ since $j^{l}\notin \mathcal {B}^{l}$. This implies . Therefore, if and only if $z_{i^{l}}^{l}=0$.

(ii) When the basic index set $\mathcal {B}^{l}$ is updated to $\mathcal {B}^{l+1}$, $\check{{{c}}}^{l}_{j^{l}}<0$. Since $j^{l}\in \mathcal {B}^{l+1}$, $\check{{{c}}}^{l+1}_{j^{l}}=0$. Then, $({{\bf c}}_{j^{l}}-{{\bf c}}_{\mathcal {B}^{l+1}}^{\top} {{\bf A}}_{\mathcal {B}^{l+1}}^{-1}{{\bf A}}_{j^{l}}) -({{\bf c}}_{j^{l}}-{{\bf c}}_{\mathcal {B}^{l}}^{\top} {{\bf A}}_{\mathcal {B}^{l}}^{-1}{{\bf A}}_{j^{l}}) =(\check{{{c}}}^{l+1}_{j^{l}}-\check{{{c}}}^{l}_{j^{l}})>0$.

Similarly as the proof of (15),

$$\bigl({{\bf c}}^\top-{{\bf c}}_{\mathcal {B}^{l+1}}^\top {{\bf A}}_{\mathcal {B}^{l+1}}^{-1}{{\bf A}}\bigr)- \bigl({{\bf c}}^\top- {{\bf c}}_{\mathcal {B}^l}^\top {{\bf A}}_{\mathcal {B}^l}^{-1}{{\bf A}}\bigr) = \kappa^l{{\bf e}}_{{i^l_*}}^\top {{\bf A}}_{\mathcal {B}^l}^{-1} {{\bf A}}, $$

where . ${{\bf e}}_{{i^{l}_{*}}}^{\top} {{\bf A}}_{\mathcal {B}^{l}}^{-1}{{\bf A}}$ is the ${i^{l}_{*}}$th pivot row for $\mathcal {B}^{l}$, which is lexicographically positive. Since the j ^lth entry of ${{\bf e}}_{{i^{l}_{*}}}^{\top} {{\bf A}}_{\mathcal {B}^{l}}^{-1}{{\bf A}}$ is strictly positive, that of $({{\bf c}}^{\top}-{{\bf c}}_{\mathcal {B}^{l+1}}^{\top} {{\bf A}}_{\mathcal {B}^{l+1}}^{-1}{{\bf A}})- ({{\bf c}}^{\top}-{{\bf c}}_{\mathcal {B}^{l}}^{\top} {{\bf A}}_{\mathcal {B}^{l}}^{-1}{{\bf A}})$ must share the same sign with κ ^l. Thus, we have κ ^l>0. Then the updated cost row is given as

Clearly, the cost row for $\mathcal {B}^{l+1}$ is lexicographically greater than that for $\mathcal {B}^{l}$.