Sparse discriminative feature weights learning

Abstract

Sparse representation, a locality-based data representation method, leads to promising results in many scientific and engineering fields. Meanwhile in the study of feature selection, locality preserving is widely recognized as an effective measurement criterion. In this paper, we introduce l₁-norm driven sparse representation into feature selection, and propose a novel joint feature weights learning algorithm, named sparse discriminative feature weights (SDFW). SDFW assigns the highest score to the feature that has the smallest difference between within-class reconstruction residual and between-class reconstruction residual in the space of selected features. It possesses the following advantages: (1) compared with feature selection methods based on k nearest neighbors, SDFW automatically (vs. manually) determines neighborhood for individual sample; (2) compared with conventional heuristic feature search which selects features individually, SDFW selects feature subset in batch mode. Extensive experiments on different data types demonstrate the effectiveness of SDFW.

Introduction

In the past decades, dimensionality reduction has attracted much attention for its immense potentials in many applications, e.g., computer vision [1], object recognition [2] and data mining. It can decrease storage requirements, significantly speed up subsequent learning process, and enhance classifiers׳ generalization capability. Generally speaking, dimensionality reduction techniques can be classified into two categories: feature selection and feature extraction [3]. Compared with feature extraction, feature selection merely selects a subset of features from the original features, thus it preserves the semantics of the original features.

According to the availability of label information, feature selection algorithms can be classified into two groups: unsupervised feature selection [4], [5], [6] and supervised feature selection [7], [8], [9], [10]. Unsupervised feature selection algorithms, e.g., Laplacian score [9], discriminative feature selection [11], select the features which best preserve the local data information. Supervised feature selection algorithms, e.g., ReliefF [8], robust feature selection [12], and trace ratio [13], usually select discriminative features according to labels of the training data. In this paper, we focus on supervised feature selection.

For feature selection, researchers mainly focus on searching strategies and measurement criteria.

From the perspective of searching strategies, feature selection can be viewed as a binary optimization problem, which is NP hard. To alleviate this difficulty, heuristic feature selection evaluates the importance of each feature individually, and adds features into the feature subset one by one until the user-defined size is reached. However this greedy way neglects the interaction and dependency among different features. Recently sparse model for joint feature selection algorithms [6], [12], [14], [15] has been developed. For example, Ref. [6] conduct spectral regression and l₁-norm minimization in two separated steps. To facilitate feature selection, the $l_{2,1}$-norm regularized model [12], [11], [16], [17] imposes the row-sparse constraint on the regularization term, and consequently they select features associated with the non-zero rows of the feature selection matrix. These efforts have demonstrated the effectiveness of joint feature selection.

In study of measurement criteria, researchers regard manifold structure preserving as a useful criterion for measuring the importance of each feature. For example, He et al. [9] estimate the importance of a feature by its power of locality preserving based on k nearest neighbor graph. Kira et al. [8] evaluate feature weights according to their ability to discriminate between neighboring samples. Yang et al. [11] select the most discriminative features, where manifold structure is considered.

In this paper we incorporate sparse representation, discriminative information, joint feature weights learning into a unified framework to select feature subset in batch mode. The proposed feature selection algorithm is thus coined sparse discriminative feature weights (SDFW) learning, which has the following advantages.

• Because of the importance of discriminative information [7] in data analysis, we exploit discriminative information based on sparse representation for feature selection. SDFW characterizes the scatter of samples by the difference between within-class reconstruction residual and between-class reconstruction residual, thus it adaptively seeks features that are efficient for discrimination.

• SDFW iteratively estimates the feature weights and sparse representation coefficients alternatively by optimizing the objective function. Compared with traditional methods, SDFW seeks the nearest neighbors of each sample in the weighted space, not the original space. Moreover SDFW provides the potential to discover a possible intrinsic relationship between the feature selection and the sparse representation.

• SDFW jointly assigns each feature a real-valued number to indicate its importance. Different from traditional heuristic feature search, SDFW selects the most discriminative feature subset from the original features in batch mode by learning a nonnegative feature weight vector.

• Since SDFW, as a joint feature weights learning method, exploits the feature correlation, we do not apply any de-correlation transformation for data preprocess in our algorithm. Furthermore, different from whitening transformation, which removes feature correlation in the original data space, our method takes adaptive mechanism to learn feature weights in the space of selected features.

The remaining of this paper is organized as follows. We briefly review sparse representation and it steered optimized projections for classification in Section 2. In Section 3 we present the proposed formulation followed by a new iterative algorithm to optimizing our objective function. The experiments on four data sets are conducted in Section 4. Finally, we draw a conclusion in Section 5.