A sparse logistic regression framework by difference of convex functions programming

Abstract

Feature selection for logistic regression (LR) is still a challenging subject. In this paper, we present a new feature selection method for logistic regression based on a combination of the zero-norm and l ₂-norm regularization. However, discontinuity of the zero-norm makes it difficult to find the optimal solution. We apply a proper nonconvex approximation of the zero-norm to derive a robust difference of convex functions (DC) program. Moreover, DC optimization algorithm (DCA) is used to solve the problem effectively and the corresponding DCA converges linearly. Compared with traditional methods, numerical experiments on benchmark datasets show that the proposed method reduces the number of input features while maintaining accuracy. Furthermore, as a practical application, the proposed method is used to directly classify licorice seeds using near-infrared spectroscopy data. The simulation results in different spectral regions illustrates that the proposed method achieves equivalent classification performance to traditional logistic regressions yet suppresses more features. These results show the feasibility and effectiveness of the proposed method.

Appendix: The primal-dual interior-point method for solving convex problem (32)

Note that p(y = 1/x)+p(y = −1/x)=1, and thus problem 32) can be written as:

$$\begin{array}{@{}rcl@{}} \min\limits_{b,\mathbf{w},\mathbf{t}}&&\left\{G(b,\mathbf{w},\mathbf{t})-\langle \mathbf{v}^{k},\mathbf{w}\rangle: (b,\mathbf{w},\mathbf{t})\in {\Omega}\right\}\\[-2pt] =\min\limits_{b,\mathbf{w},\mathbf{t}}&&\left\{-\sum\limits_{y_{i}=1} (b+ \mathbf{w}^{T} \mathbf{x}_{i})+\sum\limits_{i}log\left( 1+e^{b+\mathbf{w}^{T} \mathbf{x}_{i}}\right)+\lambda \|\mathbf{w}\|_{2}^{2}\right.\\[-2pt] &&\left.+\mu \sum\limits_{j=1}^{n} a | w_{j}|-\mu\langle \mathbf{v}^{k},\mathbf{w}\rangle: (b,\mathbf{w},\mathbf{t})\in {\Omega}\right\}\\[-2pt] =\min\limits_{b,\mathbf{w},\mathbf{t}}&&\left\{-\sum\limits_{y_{i}=1} \left( b+\mathbf{w}^{T} \mathbf{x}_{i}\right)+\sum\limits_{i}log\left( 1+e^{b+\mathbf{w}^{T} x_{i}}\right)+\lambda \|\mathbf{w}\|_{2}^{2}\right.\\[-2pt] &&\left.+\mu\sum\limits_{j=1}^{n}t_{j}-\mu\langle \mathbf{v}^{k},\mathbf{w}\rangle : (b,\mathbf{w},\mathbf{t})\in {\Omega}\right\} \end{array} $$

(42)

where:

$${\Omega}\,=\,\left\{(b,\mathbf{w},\mathbf{t})\in R^{2n+1}:-\alpha w_{j}\leq t_{j},\alpha w_{j}\leq t_{j},j=1...n \right\} $$

Let x = (b, w, t) with x∈R ²ⁿ⁺¹and

$$\begin{array}{@{}rcl@{}} F(\mathbf{x})&=&\!\!-\sum\limits_{y_{i}=1}\! \left( b+\mathbf{w}^{T} \mathbf{x}_{i}\right)\,+\,\sum\limits_{i}log\left( 1+e^{b+\mathbf{w}^{T}\mathbf{x}_{i}}\right)+\lambda \|\mathbf{w} \|_{2}^{2}\\ &&+\mu\sum\limits_{j=1}^{n}t_{j}-\mu\langle \mathbf{v}^{k},\mathbf{w}\rangle\ \end{array} $$

(43)

Then the problem (42) is equivalent to

$$\begin{array}{@{}rcl@{}} \min\limits_{b,\mathbf{w},\mathbf{t}}&& F(b,\mathbf{w},\mathbf{t})\\ \textit{ s.t.}&& -a w_{j}\leq t_{j},a w_{j}\leq t_{j},j=1...n \label {03} \end{array} $$

(44)

Introducing Lagrangian multiplier s with components s _i (s _i ≥0, the Lagrangian function for the problem (44) can be expressed as

$$\begin{array}{@{}rcl@{}} F(\mathbf{x})-\mathbf{s}^{T}A\mathbf{x}, s\geq 0, \mathbf{s} \in R^{2n} \end{array} $$

(45)

where

$$\begin{array}{@{}rcl@{}} A=\left( \begin{array}{lcr} 0_{n\times1} &\alpha*I_{n\times n}&I_{n\times n}\\ 0_{n\times 1} &-\alpha*I_{n\times n}&I_{n\times n} \end{array} \right) \end{array} $$

(46)

where I _n×n is unit matrix and 0 _n×1 stands for a real n×1 matrix. The first-order necessary optimality conditions for (44) is

$$\begin{array}{@{}rcl@{}} \nabla F(\mathbf{x})-A^{T}{s}=0 \\ {s}^{T}A\mathbf{x}=0 ,\mathbf{s}\geq 0 \end{array} $$

(47)

$$\begin{array}{@{}rcl@{}} Ax\geq 0 \end{array} $$

(48)

where

$$\begin{array}{@{}rcl@{}} \nabla F(\mathbf{x})=\left( \begin{array}{lcr} -{\sum}_{y_{i}=1}1+{\sum}_{i}\frac{e^{b+\mathbf{w}^{T} x_{i}}}{1+e^{b+\mathbf{w}^{T} x_{i}}}\\ -{\sum}_{y_{i}=1}\mathbf{x}_{i}+{\sum}_{i}\dfrac {e^{b+\mathbf{w}^{T} \mathbf{x}_{i}}}{1+e^{b+\mathbf{w}^{T} \mathbf{x}_{i}}}\mathbf{x}_{i}+2\lambda \mathbf{w}-\mu \mathbf{v}^{k}\\ \mu \mathbf{\xi}_{n\times1} \end{array} \right) \end{array} $$

(49)

where ξ _n×1 denotes a real n×1 matrix. Let A x = z, the above system (47) can be written as

$$\begin{array}{@{}rcl@{}} \nabla F(\mathbf{x})-A^{T}\mathbf{s}=0 \\ A\mathbf{x}=\mathbf{z},\mathbf{z}\geq 0 \end{array} $$

(50)

$$\begin{array}{@{}rcl@{}} \mathbf{s}^{T}\mathbf{z}=0 ,\mathbf{s}\geq 0 \end{array} $$

(51)

Accordingto the primal-dual interior-point algorithm, we replace s ^T z = 0 by s _i z _i = μ(μ>0) in the system (50), and then obtain

$$\begin{array}{@{}rcl@{}} \nabla F(\mathbf{x})-A^{T}\mathbf{s}=0 \\ A\mathbf{x}=\mathbf{z} ,\mathbf{z} \geq 0 \end{array} $$

(52)

$$\begin{array}{@{}rcl@{}} s_{i}z_{i}=\mu, s_{i}\geq 0 \end{array} $$

(53)

where z _i is the i-th component of the variable z. The above system (52) is a perturbation of the first-order optimality conditions for (47). For a certain μ, the Newton method is used to solve this system, and then we decease μ to 0. Finally, we get the approximation solution for the system (47). Moreover, at each iteration the Newton direction is obtained by solving:

$$\begin{array}{@{}rcl@{}} \nabla^{2}F(\mathbf{x})\triangle \mathbf{x}-A^{T}{\Delta} \mathbf{s}=-\nabla F(\mathbf{x})+A^{T}\mathbf{s} \\ A{\Delta} y-{\Delta} \mathbf{z}=\mathbf{z}-A\mathbf{x},\mathbf{z}\geq 0 \end{array} $$

(54)

$$\begin{array}{@{}rcl@{}} s_{i}{\Delta} z+{\Delta} s_{i}z=\mu-s_{i}z_{i}, s_{i} \geq 0 \end{array} $$

(55)