E-ENDPP: a safe feature selection rule for speeding up Elastic Net

Abstract

Lasso is a popular regression model, which can do automatic variable selection and continuous shrinkage simultaneously. The Elastic Net is one of the corrective methods of Lasso, which selects groups of correlated variables. It is particularly useful when the number of features p is much bigger than the number of observations n. However, the training efficiency of the Elastic Net for high-dimensional data remains a challenge. Therefore, in this paper, we propose a new safe screening rule, i.e., E-ENDPP, for the Elastic Net problem which can identify the inactive features prior to training. Then, the inactive features or predictors can be removed to reduce the size of problem and accelerate the training speed. Since this E-ENDPP is derived from the optimality conditions of the model, it can be guaranteed in theory that E-ENDPP will give identical solutions with the original model. Simulation studies and real data examples show that our proposed E-ENDPP can substantially accelerate the training speed of the Elastic Net without affecting its accuracy.

Appendices

Appendix A: Proof of Lemma 1

First of all, we reintroduce problem (3) in the following:

$$\begin{array}{@{}rcl@{}} \underset{\beta\in R^{p}}{min}\frac{1}{2}\|y-X\beta\|_{2}^{2}+\frac{\gamma}{2}\|\beta\|_{2}^{2}+\lambda\|\beta\|_{1}. \end{array} $$

For the above problem, let

$$\begin{array}{@{}rcl@{}} \bar{Y}=\left( \begin{array}{l} y \\ 0\end{array} \right) , \bar{X}=\left( \begin{array}{l} X \\ \sqrt{\gamma}I \end{array} \right). \end{array} $$

Then we will obtain that

$$\begin{array}{@{}rcl@{}} &&\frac{1}{2}\|y-X\beta\|_{2}^{2}+\frac{\gamma}{2}\|\beta\|_{2}^{2}+\lambda \|\beta\|_{1}\\ ~~&&=\frac{1}{2}(y-X\beta)^{T}(y-X\beta)+\frac{\gamma}{2}\beta^{T}\beta+\lambda\|\beta\|_{1}\\ ~~&&=\frac{1}{2}y^{T}y-y^{T}X\beta+\frac{1}{2}\beta^{T}(X^{T}X+\gamma I)\beta+\lambda \|\beta\|_{1} \end{array} $$

$$\begin{array}{@{}rcl@{}} ~~&&=\frac{1}{2}(y^{T}~~0)\left( \begin{array}{l}y \\0 \end{array}\right)-(y^{T}~~0) \left( \begin{array}{l}X \\ \sqrt{\gamma}I \end{array} \right) \beta\\ &&+\frac{1}{2}\beta^{T}(X^{T}~~\sqrt{\gamma}I)\left( \begin{array}{l}X \\ \sqrt{\gamma}I \end{array} \right) \beta+\lambda\|\beta\|_{1}\\ ~~&&=\frac{1}{2}\bar{Y}^{T}\bar{Y}+\frac{1}{2}\beta^{T}\bar{X}^{T}\bar{X}\beta-\bar{Y}^{T}\bar{X}\beta+\lambda\|\beta\|_{1}\\ ~~&&=\frac{1}{2}\|\bar{Y}-\bar{X}\beta\|_{2}^{2}+\lambda\|\beta\|_{1}, \end{array} $$

(24)

therefore, problem (3) can be rewritten as:

$$\begin{array}{@{}rcl@{}} \underset{\beta\in R^{p}}{min}~~\frac{1}{2}\|\bar{Y}-\bar{X}\beta\|_{2}^{2}+\lambda\|\beta\|_{1}, \end{array} $$

Lemma 1 has been proved.

Appendix B: The dual problem of the Elastic Net

For the problem (2), according to (Boyd and Vandenberghe, 2004)[4], by introducing a new set of variables $w=y-X\beta$, the problem (2) becomes

$$\begin{array}{@{}rcl@{}} \underset{w, \beta}{min}~~&&\frac{1}{2}\|w\|_{2}^{2}+\lambda\|\beta\|_{1},\\ \text{s.t.} ~~&& w=y-X\beta. \end{array} $$

(25)

By introducing the multipliers $\eta \in R^{n}$, we get the Lagrangian function as follows:

$$\begin{array}{@{}rcl@{}} L(\beta,w,\eta)=\frac{1}{2}\|w\|_{2}^{2}+\lambda\|\beta\|_{1}+\eta^{T}(y-X\beta-w). \end{array} $$

The dual function $g(\eta )$ is

$$\begin{array}{@{}rcl@{}} g(\eta)&=&\underset{\beta,w }{inf}L(\beta,w,\eta)=\eta^{T}y+\underset{w}{inf} (\frac{1}{2}\|w\|_{2}^{2} -\eta^{T}w) \\ &+&\underset{\beta}{inf} (-\eta^{T}X\beta+\lambda\|\beta\|_{1}). \end{array} $$

(26)

Note that the right side of $g(\eta )$ has three items. In order to get $g(\eta )$, we need to solve the following two optimization problems.

$$\begin{array}{@{}rcl@{}} \underset{w}{inf} &&\frac{1}{2}\|w\|_{2}^{2} -\eta^{T}w, \end{array} $$

(27)

$$\begin{array}{@{}rcl@{}} \underset{\beta}{inf} &&-\eta^{T}X\beta+\lambda\|\beta\|_{1}. \end{array} $$

(28)

Let us first consider (27). Denote the objective function as

$$\begin{array}{@{}rcl@{}} f_{1}(w)=\frac{1}{2}\|w\|_{2}^{2} -\eta^{T}w, \end{array} $$

then, let

$$\begin{array}{@{}rcl@{}} \frac{\partial f_{1}(w)}{\partial w}=w-\eta= 0\Rightarrow w=\eta, \end{array} $$

so

$$\begin{array}{@{}rcl@{}} \underset{w}{inf} f_{1}(w)=-\frac{1}{2}\eta^{T}\eta=-\frac{1}{2}\|\eta\|^{2}_{2}. \end{array} $$

Next, let us consider the problem (28). Denote the objective function as

$$\begin{array}{@{}rcl@{}} f_{2}(\beta)=-\eta^{T}X\beta+\lambda\|\beta\|_{1}, \end{array} $$

f₂(β) is convex but not smooth. So let us consider its subgradient

$$\begin{array}{@{}rcl@{}} \frac{\partial f_{2}(\beta)}{\partial \beta}=-X^{T}\eta+\lambda\frac{\partial \|\beta\|_{1}}{\partial \beta}= 0, \end{array} $$

where

$$\begin{array}{@{}rcl@{}} \frac{\partial \|\beta\|_{1}}{\partial \beta_{i}}=\left\{\begin{array}{lll} &sign([\beta^{*}]_{i}),&if [\beta^{*}]_{i}\neq0 \\ &[-1,1],&if [\beta^{*}]_{i}= 0 \end{array}\right.\quad i = 1,2,{\cdots} p. \end{array} $$

According to the necessary condition for $f_{2}(\beta )$ to attain an optimum, we have $|{X^{T}_{i}}\eta |\leq \lambda ,\quad i = 1,2,{\cdots } p$. Therefore, the optimum value of problem (28) is 0. Combining the equations above, we can get the dual problem:

$$\begin{array}{@{}rcl@{}} \underset{\eta}{max} &&g(\eta)=\eta^{T}y-\frac{1}{2} \|\eta\|_{2}^{2}, \\ \text{s.t.}~~&&|{X^{T}_{i}}\eta|\leq\lambda,\quad i = 1,2,{\cdots} p, \end{array} $$

which is equivalent to the following optimization problem:

$$\begin{array}{@{}rcl@{}} \underset{\eta}{max} &&\frac{1}{2} \|y\|_{2}^{2}-\frac{1}{2} \|\eta-y\|_{2}^{2},\\ \text{s.t.}~~&& |{X^{T}_{i}}\eta|\leq\lambda,\quad i = 1,2,{\cdots} p. \end{array} $$

(29)

By a simple re-scaling of the dual variable $\eta$, that is, let $\theta =\frac {\eta }{\lambda }$, we get the following result. The dual problem of (2) is:

$$\begin{array}{@{}rcl@{}} \underset{\theta}{max} &&\frac{1}{2}\|y\|_{2}^{2}-\frac{\lambda^{2}}{2} \|\theta-\frac{y}{\lambda}\|_{2}^{2},\\ \text{s.t.}~~&&\mid {X_{i}^{T}}\theta\mid\leq1\quad i = 1,2,{\cdots} p, \end{array} $$

(30)

where $\theta$ is the dual variable. For notational convenience, let the optimal solution of problem (30) be $\theta ^{*}(\gamma ,\lambda )$, and the optimal solution of problem (2) with parameters $\gamma$ and $\lambda$ is denoted by $\beta ^{*}(\gamma ,\lambda )$. Then, the KKT conditions are given by:

$$\begin{array}{@{}rcl@{}} y=X\beta^{*}(\gamma,\lambda)+\lambda\theta^{*}(\gamma,\lambda), \end{array} $$

(31)

$$\begin{array}{@{}rcl@{}} &&{X^{T}_{i}}\theta^{*}(\gamma,\lambda)\in \left\{ \begin{array}{lll} &sign([\beta^{*}(\gamma,\lambda)]_{i}),&if [\beta^{*}(\gamma,\lambda)]_{i}\neq0 \\ &[-1,1],&if [\beta^{*}(\gamma,\lambda)]_{i}= 0 \end{array} \right.\\&& i = 1,2,{\cdots} p, \end{array} $$

(32)

Note that $\bar {Y}=\left (\begin {array}{l} y \\ 0 \end {array} \right ) , \bar {X}=\left (\begin {array}{l} X \\ \sqrt {\gamma }I \end {array}\right )$. Let $\theta =\left (\begin {array}{l} \theta _{1} \\ \theta _{2} \end {array} \right )$, then we have the dual problem of (3) as follows,

$$\begin{array}{@{}rcl@{}} \underset{\theta}{max} &&\frac{1}{2}\|y\|_{2}^{2}-\frac{\lambda^{2}}{2} (\|\theta_{1}-\frac{y}{\lambda}\|_{2}^{2}+\|\theta_{2}\|_{2}^{2}),\\ \text{s.t.}~~&&\mid \bar{X}_{i}^{T}\theta\mid=\mid {x_{i}^{T}}\theta_{1}+\sqrt{\gamma}(\theta_{2})_{i}\mid\leq1,~ i = 1,2,{\cdots} p, \end{array} $$

(33)

where $\theta =\left (\begin {array}{l} \theta _{1} \\ \theta _{2} \end {array} \right )$ is the dual variable. So, Theorem 1 has been proved.