Unsupervised feature selection with adaptive multiple graph learning

Highlights

• A framework of jointly multiple graph learning and feature selection is proposed.

• An effective algorithm is proposed for optimizing the objective function.

• The experimental results show that our algorithm outperforms the state-of-the-art methods.

Abstract

Unsupervised feature selection methods try to select features which can well preserve the intrinsic structure of data. To represent such structure, conventional methods construct various graphs from data. In most cases, those different graphs often contain some consensus and complementary information. To make full use of such information, we construct multiple base graphs and learn an adaptive consensus graph from these base graphs for feature selection. In our method, we integrate the multiple graph learning and the feature selection into a unified framework, which can jointly characterize the structure of the data and select the features to preserve such structure. The underlying optimization problem is hard to solve, and we solve it via a block coordinate descent schema, whose convergence is guaranteed. The extensive experiments well demonstrate the effectiveness of our proposed framework.

Introduction

Feature selection is a fundamental problem in machine learning and has attracted considerable attention in the past decades [1], [2], [3], [4], [5]. In many real-world applications, the data contain a large number of features, which may cause the curse of dimensionality. Moreover, some features are often contaminated by noises, which may deteriorate the performance of machine learning methods. To address these problems, feature selection is applied to select a small number of informative and non-redundant features for the machine learning methods. Since feature selection usually leads to better learning performance, it has been successfully applied in many real applications such as text categorization [6], [7], image processing [8], and bioinformatics [9]. According to the availability of the labels, these methods can be categorized into supervised [10], [11], semi-supervised [12] and unsupervised algorithms [5], [13]. Due to the absence of the label information, the unsupervised feature selection is a more challenging problem.

Since the label information is absent in unsupervised learning, feature selection method can only make use of the information of data itself, or equivalently speaking, the intrinsic structure of data [14], [15], [16], [17], [18]. Conventional methods often construct a graph from the data to represent such structure. According to the way of constructing the graph, the existing methods can be roughly categorized into two classes: (1) using a pre-defined graph, which often pre-defines a graph and selects features to preserve such graph structure [19], [20], [21]; (2) leaning an adaptive graph, which learns an adaptive graph simultaneously in the process of feature selection [5], [17], [22].

Note that both the two classes only use one single graph (either a pre-defined graph or an adaptive graph) to represent the structure of data. However, in real applications, the structure may be too complex to be captured by one single graph. Given a data set, various graphs can be constructed based on different distance metrics, such as Euclidean distance and cosine similarity. In most cases, these different graphs contain some consensus and complementary information. Those conventional methods which only use one single graph may ignore such useful information. Moreover, some data are naturally performed in multiple graph structures. For example, relationships of research papers contain several graphs such as co-author graph and citation graph. When handling these data, the aforementioned methods may fail to fully utilize the given graphs.

To address these problems, in this paper, we characterize the intrinsic structure with adaptive multiple graph learning. Multiple graph learning tries to learn a consensus graph from multiple base graphs. For example, Nie et al. [23] proposed a parameter-free multiple graph learning method which learned the weight of each graph automatically; Zhan et al. [24] learned a consensus graph with minimizing disagreement between different views and constraining the rank of the Laplacian matrix. In our framework, firstly, if the data contains multiple graphs, we directly use them as base graphs; if otherwise, we construct some pre-defined graphs from data. Then we learn the consensus graph from these base graphs simultaneously in the process of feature selection. On the one hand, we use the result of feature selection to guide the multiple graph learning, and on the other hand, we apply the learned graph to select the informative features. When learning the consensus graph, since the scale of the graphs may vary dramatically, we first normalize all base graph adjacency matrices as transition matrices and learn a consensus transition matrix from them. Since the transition matrix has a clear probabilistic interpretation, we use the Kullback-Leibler divergence for consensus measuring. When selecting the features, we impose a weight on each feature and try to transform the weighted data into a subspace which can well preserve the intrinsic structure represented by the consensus graph. This procedure repeats until convergence.

To integrate the multiple graph learning and feature selection into a unified framework as introduced before, we carefully design a non-convex objective function. To optimize it, we propose a block coordinate descent algorithm and prove its convergence. We conduct extensive experiments on benchmark data sets by comparing our algorithm with several state-of-the-art unsupervised feature selection methods, and the experimental results show that ours outperforms these state-of-the-art methods.

The paper is organized as follows. Section 2 describes some related work. Section 3 presents in detail the main algorithm of our method. Section 4 shows the experimental results, and Section 5 concludes the paper.