Sparse semi-supervised support vector machines by DC programming and DCA

Highlights

• Two DC (Difference of Convex) Algorithms based approaches are investigated for feature selection in S3VM using the zero norm.

• The first is DC approximation approach that approximates the zero norm by a DC function. Five approximations are considered.

• The second is an exact reformulation approach based on exact penalty techniques in DC programming.

• All the resulting problems are DC programs for which six versions of DCA are developed.

• A careful empirical experiment on several benchmark datasets are performed.

Abstract

This paper studies the problem of feature selection in the context of Semi-Supervised Support Vector Machine (S3VM). The zero norm, a natural concept dealing with sparsity, is used for feature selection purpose. Due to two nonconvex terms (the loss function of unlabeled data and the ℓ₀ term), we are faced with a NP hard optimization problem. Two continuous approaches based on DC (Difference of Convex functions) programming and DCA (DC Algorithms) are developed. The first is DC approximation approach that approximates the ℓ₀-norm by a DC function. The second is an exact reformulation approach based on exact penalty techniques in DC programming. All the resulting optimization problems are DC programs for which DCA are investigated. Several usual sparse inducing functions are considered, and six versions of DCA are developed. Empirical numerical experiments on several Benchmark datasets show the efficiency of the proposed algorithms, in both feature selection and classification.

Introduction

In machine learning, supervised learning is a task of inferring a predictor function (classifier) from a labeled training dataset. Each example in training set consists of an input object and a label. The objective is to build a predictor function which can be used to identify the label of new examples with the highest possible accuracy. Nevertheless, in most of real word applications, a large portion of training data are unlabeled, and supervised learning cannot be used in these contexts. To deal with this difficulty, recently, in the machine learning community, there has been an attracting increasing attention in using semi-supervised learning methods. In contrast to supervised methods, semi-supervised learning methods take into account both labeled and unlabeled data to construct prediction models.

We are interested in semi-supervised classification, more precisely, in the so-called Semi-Supervised Support Vector Machines (S3VM). Among the semi-supervised classification methods, the large margin approach S3VM, which extends the Support Vector Machine (SVM) to semi-supervised learning concept, is certainly the most popular [5], [6], [7], [8], [9], [10], [11], [12], [13], [37], [39], [41], [47], [54], [59]. An extensive overview of semi-supervised classification methods can be found in [61]. S3VM was originally proposed by Vapnik and Sterin in 1977 [57] under the name of transductive support vector machine. Later, in 1999, Bennett and Demiriz [2] proposed the first optimization formulation of S3VM which is described as follows.

Given a training set which consists of m labeled points $\left\{\right(x_i,y_i)\in {\mathbb{R}}^n\times \{-1,1\},i=1,\dots ,m\}$ and p unlabeled points $\{x_i\in {\mathbb{R}}^n,i=(m+1),\dots ,(m+p\left)\right\}$. We are to find a separating hyperplane $P=\{x\mid x\in {\mathbb{R}}^n,x^{T}w=b\}$, far away from both the labeled and unlabeled points. Hence, the optimization problem of S3VM takes the form

$$\underset{w,b}{\min }{\Vert w\Vert }_2^2+\alpha \sum _{i=1}^{m}L\left(y_i(〈w,x_i〉+b)\right)+\beta \sum _{i=m+1}^{m+p}L\left(|〈w,x_i〉+b|\right).$$

Here, the first two terms define a standard SVM while the third one incorporates the loss function of unlabeled data points. The loss function of labeled and unlabeled data points are weighted by penalty parameters $\alpha >0$ and $\beta >0$. Usually, in classical SVM one uses the hinge loss function $L\left(u\right)=\max \{0,1-u\}$ which is convex. On the contrary, the problem (1) is nonconvex, due to the nonconvexity of the third term.

There are two broad strategies for solving the optimization problem (1) of S3VM: the combinatorial methods (Mixed Integer Programming [2], Branch and Bound algorithm [6]) and the continuous optimization methods such as self-labeling heuristic $S^3{\mathit{VM}}^{\mathit{light}}$ [15], gradient descent [5], deterministic annealing [53], semi-definite programming [4], DC programming [9]. Combinatorial methods are not available for massive data sets in real applications (high dimension and large data set). Thus, major efforts have focused on efficient local algorithms. For more complete reviews of S3VM methods, the reader is referred to [7], [61] and references therein.

On the other hand, feature selection is one of the fundamental problems in machine learning. In many areas of application such as text classification, web mining, gene expression, micro-array analysis, and combinatorial chemistry, image analysis, data sets contain a large number of features, many of which are irrelevant or redundant. Feature selection is often applied to high dimensional data prior to classification learning. The main goal is to select a subset of features of a given data set while preserving or improving the discriminative ability of a classifier. Several feature-selection methods for SVMs have been proposed in the literature (see e.g. [3], [11], [17], [19], [29], [30], [33], [40], [46], [49], [62], [63]).

This paper deals with the feature selection in the context of S3VM. We are to find a separating hyperplane far away from both the labeled and unlabeled points, that uses the least number of features. As standard approaches for feature selection in SVM, here we use the zero norm, a natural concept dealing with sparsity for feature selection purpose. The zero norm of a vector, denoted ℓ₀ or $\Vert .{\Vert }_0$, is defined as the number of its nonzero components. Similar to SVM, we replace the term $\Vert w{\Vert }_2^2$ in (1) by the ℓ₀-norm and then formulate the feature selection S3VM problem as follows:

$$\underset{w,b}{\min }\Vert w{\Vert }_0+\alpha \sum _{i=1}^{m}L\left(y_i(〈w,x_i〉+b)\right)+\beta \sum _{i=m+1}^{m+p}L\left(|〈w,x_i〉+b|\right).$$

While S3VM has been widely studied, there exist few methods in the literature for feature selection in S3VM. Due to the discontinuity of the ${\ell }_0$ term and the nonconvexity of the third term, we are facing “double” difficulties in (2) (it is well known that the problem of minimizing the zero-norm is NP-Hard [1]).

During the last two decades, research is very active in models and methods optimization involving the zero norm. Works can be divided into three categories according to the way to treat the zero norm: convex approximation (the ℓ₀-norm is replaced by a convex function, for instance the ${\ell }_1$-norm [55] or the conjugate function [43]), nonconvex approximation (a continuous nonconvex function is used instead to the ℓ₀-norm, usual sparse inducing functions are introduced in [3], [11], [32], [45], [63]), and nonconvex exact reformulation (with the binary variables $u_i=0$ if $w_i=0$ and $u_i=1$ otherwise, the original problem is formulated as a combinatorial optimization problem which is equivalently reformulated as a continuous nonconvex program via exact penalty techniques, see [33], [56]). An extensive overview of these approaches can be found in [33]. When the objective function (besides the ℓ₀-term) is convex, convex approximation techniques result to a convex optimization problem which is so far “easy” to solve. Unfortunately, for S3VM, the problem (2) remains nonconvex with any approximation – convex or nonconvex, of the ℓ₀-norm.

In this work, we tackle the problem (2) by nonconvex approaches based on DC (Difference of Convex functions) programming and DCA (DC Algorithms), powerful tools in nonconvex programming framework. Our motivating arguments to use the ℓ₀-norm are multiple. Firstly, even though using the ℓ₁ is the simplest way to deal with the sparsity, the ℓ₁ can encourage the sparsity in only some cases with restrictive assumptions (see [14]). In particular, for feature selection purpose, the ℓ₁ penalty has been shown to be, in certain cases, inconsistent and biased [60]. Secondly, the ℓ₀-norm is the most natural and suitable concept for modeling the sparsity, and nonconvex approximations of the ℓ₀-norm are, in general, deeper than the ℓ₁-norm, and can then produce better sparsity. Especially, for feature selection in SVM, solutions of the ℓ₀-norm penalty problem have been shown to be much sparser than those of ℓ₁-norm approach in several previous works [3], [29], [33], [40]. Thirdly, although we are faced with a “double” difficulties in the problem (2) because of the ℓ₀-norm and the third term, the power of DCA can be exploited to efficiently solve this hard problem, knowing that DCA has been successfully developed in a variety of works in Machine Learning (see e.g. [9], [20], [27], [28], [38], [48], [50] and the list of references in [36]), in particular for feature selection in SVM [29], [30], [33], [34], [35], [40], [46].

Paper׳s contributions: We develop an unified approach based on DC programming and DCA for solving the nonconvex optimization problem (2). The first work in this research direction has been published in the conference paper [22] where we considered the piecewise exponential [3] for the ℓ₀-norm and presented a DC formulation, its corresponding DCA for solving the resulting problem as well as some preliminary numerical experiments. In this paper, we carefully explore and exploit DCA based approaches to deal with the zero norm and sparse S3VM. Firstly, several DC approximations of ℓ₀-norm will be used to (2): logarithm function [63], Smoothly Clipped Absolute Deviation (SCAD) [11], piecewise exponential [3], DC polyhedral [45] and piecewise linear approximation [32]. Secondly, we inspire the same technique in [33], [56] to equivalently formulate (2) as a combinatorial optimization problem. Then, thanks to the new result on exact penalty techniques recently developed in [31], we reformulate the resulting problem as a continuous optimization problem and investigate DCA to solve it. Finally, we provide an empirical experimentation, on several Benchmark datasets, of all proposed algorithms to study their efficiency in both feature selection and classification.

The remainder of the paper is organized as follows. DC programming and DCA are briefly presented in Section 2 while Section 3 is devoted to the development of DCA for solving the feature selection S3VM problem (2). Computational experiments are reported in Section 4 and finally Section 5 concludes the paper.