On the role of sparsity in feature selection and an innovative method LRMI

Abstract

Feature selection is used in many applications in machine learning and bioinformatics. As a popular approach, feature selection can be implemented in the filter-manner based on the sparse solution of the l₁ regularization. Most study of the l₁ regularization concentrates on investigating the iteration solution of the problem or focuses on adapting sparsity to different applications. It is necessary to explore more deeply about how the sparsity learned with the l₁ regularization contributes to feature selection. In this paper, we make an effort to analyze the role of the l₁ regularization in feature selection from the perspective of information theory. We discover that the l₁ regularization contributes to minimizing the redundancy in feature selection. To avoid the complex computation of the l₁ optimization, we propose a novel feature selection algorithm, i.e. the Laplacian regularization based on mutual information (LRMI), which realizes the minimization of the redundancy in a new way, and incorporates the l₂ norm to achieve automatic grouping. Extensive experimental results demonstrate the superiority of LRMI over several traditional l₁ regularization based feature selection algorithms with less time consumption.

Introduction

In computer vision, machine learning and data mining, the object is usually represented as high-dimensional feature vectors. During the processes of various applications, high-dimensional data usually consumes an enormous amount of computing time and storage space. Moreover, most of the existing machine learning methods, such as classification, regression or other tasks, are more adaptive to low-dimensional data. The computation of high-dimensional data can become much more complex and challenging. As a solution to this issue, feature selection (also known as variable selection) [1], [2], [3], [4] is performed to choose a representative subset from the high dimensional features. The subset is expected to eliminate redundancy and bear sufficient information about the original high-dimensional feature set for specific learning tasks.

According to different evaluation metrics, feature selection algorithms can roughly be classified into three categories, i.e., filter-manner, wrapper, and embedded methods [4]. The filter-manner methods, such as Variance or Fisher score [5], [6], rely on general characteristics of the data to evaluate and select feature subsets without the interference of the learning algorithm. The wrapper models take the performance of the learning algorithms as the evaluation criterion. In other words, the purpose of the wrapper methods is to select the most suitable feature subset for the learning task. For embedded methods, the process of feature selection and the training of learning model are integrated and accomplished simultaneously.

The theory of sparse representation [7], [8] and compressed sensing [9] has been integrated in embedded feature selection. These feature selection methods have been widely used in face authentication and detection [10], [11], face attributes classification [12], mass spectrometry data classification [13] and gene expression [14]. They usually resort to sparsity solution based on the regularization techniques such as the l₁ norm constraint or the l₁ penalty term. Hence, sparse representation is also called as the l₁ regularization, the l₁ penalty or the l₁ norm in some cases.

The theoretical understanding of these methods is mostly focused on the sparsity ensured by the l₁ regularization with concerns on two major aspects. One is about various solutions to the l₁ regularization, the other is about multiple applications of sparsity-inducing feature selection. The mechanism about the sparsity learned with the l₁ regularization is not fully revealed in feature selection. What role does the l₁ regularization play in the process of feature selection? Is it really necessary to adopt the l₁ regularization to select features subset? By understanding the real merit of the l₁ regularization in feature selection, could more efficient methods be developed?

After solving the objective function containing the l₁ regularization, a coefficient vector is obtained as the sparsity value for each feature. The larger the value is, the more important the corresponding feature is. However, in the validation experiments in our previous work [15], [16], we discover that features corresponding to smaller weights can also obtain good results in classification tasks. It is very attractive to explore the behind mechanism of such phenomena that seems to be odd to the presumption about the sparsity-inducing feature selection.

Inspired by such observation, we dedicate to reevaluate the role of sparsity in feature selection from the perspective of information theory. We analyze the variable dependency among features selected with the l₁ regularization sparsity. We discover that the l₁ regularization serves to minimize redundancy among selected features. It is known that the sparsity obtained with the l₁ regularization are very time-consuming. The solution often resorts to solving either an NP-hard problem or an alternative problem that involves a costly iterative optimization [17]. For this reason, we propose a novel feature selection method, i.e. the Laplacian regularization based on mutual information (LRMI), to solve the above problem from two aspects. Firstly, the mutual information is adopted to measure the feature correlation to realize the redundancy minimization. Secondly, we integrate the essential structures of the features to identify important variables by introducing the l₂ norm in the objective function. Besides its special group effect [18], the l₂ norm leads to a closed-form solution and avoids the time-consuming optimization of the l₁ regularization. In addition, different from the filter-manner of the other mutual information based feature selection methods [19], [20], [21], LRMI realizes feature selection in the embedded manner. To evaluate our approach, we conduct extensive experiments on several public data sets. The experimental results prove the effectiveness of our method. We also develop the proposed model to a parallel realization to improve its time efficiency.

As an extension to our previous work [16], the main contributions of this paper are as follows:

• We use information theory to evaluate the role of the l₁ regularization in feature selection and discover that sparsity intrinsically acts as the item of minimal redundancy in feature selection.

• We propose a new feature selection method based on mutual information instead of to the l₁ regularization, which not only realizes the minimum redundancy among features but also utilizes the structural information of the feature. The new method has a closed-form solution and avoids the time-consuming optimization iteration of the l₁ regularization.

• By experiments in several public data sets, we demonstrate the superiority of the approach over several mainstream l₁ regularization algorithms.

• We develop a parallel computing realization of the proposed model. The training time of our method is only 50 percent of that of the traditional methods.

The remainder of this paper is organized as follows. Section 2 describes the related work. In Section 3, we discuss the role of the l₁ regularization sparsity in feature selection. We present a novel feature selection model in Section 5. The experimental results are summarized in Section 5. We draw the conclusions in Section 6.