Unsupervised feature selection via Diversity-induced Self-representation

Abstract

Feature selection is to select a subset of relevant features from the original feature set. In practical applications, regarding the unavailability of an amount of the labels is still a challenging problem. To overcome this problem, unsupervised feature selection algorithms have been developed and achieve promising performance. However, most existing approaches consider only the representativeness of features, but the diversity of features which may lead to the high redundancy and the losses of valuable features are ignored. In this paper, we propose a Diversity-induced Self-representation (DISR) based unsupervised feature selection method to effectively select the features with both representativeness and diversity. Specifically, based on the inherent self-representation property of features, the most representative features can be selected. Meanwhile, to preserve the diversity of selected features and reduce the redundancy of the original features as soon as possible, we introduce a novel diversity term, which adjusts the weights of selected features by incorporating the similarities between features. We then present an efficient algorithm to solve the optimization problem by using the inexact Augmented Lagrange Method (ALM). Finally, both clustering and classification tasks are used to evaluate the proposed method. Empirical results on the synthetic dataset and nine real-world datasets demonstrate the superiority of our method compared with state-of-the-art algorithms.

Introduction

The high-dimensional data is ubiquitous in many areas, such as computer vision, pattern recognition and data mining. It not only significantly increases the computation and storage cost, but also induces overfitting and incomprehensible models. To overcome these issues, feature selection techniques have been considered as a type of effective method to reduce the dimensionality by removing irrelevant and redundant features. The aim of feature selection is to obtain a subset of features by removing the noise and redundancy in original features, so that the more intrinsic representation of data and the better performance are achieved [1].

According to the availability of the label information, feature selection approaches can be categorized into supervised methods [2], [3], [4], [5] and unsupervised methods [6], [7], [8], [9], [10], [11]. Compared with supervised methods, unsupervised feature selection methods aim at selecting relevant features without label information. Since the labeling of samples is usually expensive, we cannot always obtain the labels beforehand. Thus, unsupervised feature selection holds great potential in real-world applications.

The early unsupervised feature sePh.D.lection algorithms consider feature ranking techniques as the principle criteria for feature selection [8], [12], [13], [14], [15]. One of the main limitations of these methods is that they treat features independently without considering possible correlations among features. To address this problem, a series of algorithms [10], [16], [17] have been developed. A typical method is spectral clustering based algorithms, which can select a feature subset with preserving the underlying structure between clusters. Spectral clustering based methods explore the cluster structure of data using matrix factorization for spectral analysis, and then select features via sparsity regularization models. Nevertheless, they heavily rely on the learned graph Laplacian. Noises in features may lead to their unreliability. Recently, the self-representation technique has shown significant potential in many tasks, such as subspace clustering [18], [19] and active learning [20], [21]. Motivated by this, some researchers consider the feature selection from the perspective of self-representation property of features [22], i.e., each feature can be well represented by the linear combination of its relevant features. However, they mainly consider selecting the representative features while ignoring the diversity among them. Both representativeness and diversity properties are very important for selecting the effective features: (1) the ideal selected features are supposed to represent the whole original features. That is to say, the highly irrelevant features are discarded, and meanwhile the most relevant features are preserved. (2) The ideal selected features should be diverse enough to capture not only important (representative) but also comprehensive information from features. By considering the diversity property, we can capture more information of data, because features usually describe different aspects of data. (3) The diversity property implies that the very similar features should not be selected simultaneously, so that the redundancy can be greatly reduced. Therefore, there is a great need for integrating both representativeness and diversity properties of features for feature selection.

In this paper, considering both representativeness and diversity properties of features, we propose a novel method, called Diversity-induced Self-representation (DISR) for unsupervised feature selection. Specifically, based on self-representation property, i.e., each feature can be well approximated by the linear combination of its relevant features, the most representative features can be selected. Meanwhile, by incorporating the similarities between features to adjust the weights of being selected, we introduce a diversity term to reduce the redundancy. Then, an efficient optimization algorithm is provided by using the inexact Augmented Lagrange Method (ALM).

Finally, we evaluate our method in both clustering and classification tasks. Experimental results on the synthetic dataset and nine real-world datasets show that the proposed DISR has a better performance than other compared methods.

To summarize, the main contributions of this paper are as follows:

• A novel Diversity-induced Self-representation (DISR) for unsupervised feature selection algorithm is proposed. The algorithm considers both the representativeness and diversity properties of features, and hence it can select more valuable features.

• A diversity term is introduced to the method. This term is the measure of diversity of all the features. Thus, the diversity of features is used to guide for feature selection.

• An iterative algorithm based on inexact ALM is proposed to efficiently solve the optimization model. Experimental results demonstrate that the superiority of our algorithm compared with state-of-the-art algorithms.

The rest of the paper is organized as follows. A brief review of the related works on unsupervised feature selection is given in Section 2. Section 3 introduces the proposed DISR algorithm, and Section 4 describes the optimization of our algorithm in details. Extensive experiments on synthetic and real-world datasets are presented in Section 5. Finally, Section 6 concludes this paper.