Semi-supervised learning via sparse model

Abstract

Graph-based Semi-Supervised Learning (SSL) methods are the widely used SSL methods due to their high accuracy. They can well meet the manifold assumption with high computational cost, but don't meet the cluster assumption. In this paper, we propose a Semi-supervised learning via SPArse (SSPA) model. Since SSPA uses sparse matrix multiplication to depict the adjacency relations among samples, SSPA can approximate low dimensional manifold structure of samples with lower computational complexity than these graph-based SSL methods. Each column of this sparse matrix corresponds to one sparse representation of a sample. The rational is that the inner product of sparse representations can also be sparse under certain constraint. Since the dictionary in the SSPA model can depict the distribution of the entire samples, the sparse representation of a sample encodes its spatial location information. Therefore, in the SSPA model the manifold structure of samples is computed via their locations in the intrinsic geometry of the distribution instead of their feature vectors. In order to meet the cluster assumption, we propose an structured dictionary learning algorithm to explicitly reveal the cluster structure of the dictionary. We develop the SSPA algorithms with the structured dictionary besides non-structured one, and experiments show that our methods are efficient and outperform state-of-the-art graph-based SSL methods.

Introduction

Semi-supervised learning (SSL) uses a large number of unlabeled samples together with a small number of labeled samples to build a better learner. It has attracted great research interest in both theory and practice in the past decade [1]. There exist various approaches in SSL to exploit unlabeled samples, showing incremental performance improvements [2]. Among them, graph-based methods, also known as manifold methods [3], [4], are widely used due to their good performances [3], [5], [6].

Through establishing the graph structure to represent adjacency relations among samples, the graph-based SSL methods can approximate the intrinsically low-dimensional manifold structure well [7]. Existing ways for graph construction include K-nearest-neighbor(KNN) method and ε-ball based method [5], [6]. In general, they construct the graph structure in two steps: (1) choosing adjacency samples needed to be connected, and (2) determining the edge weights [6]. $l_1$ graph, recently proposed by [4], is constructed at one run by solving each sample's sparse representation based on the rest of the entire sample set. It can surpass KNN method and ε-ball based method due to its characteristics of the high discriminative power, sparsity, and adaptive neighborhood [8]. However, these graph-based SSL methods have common drawbacks: high computational cost always comes together with the graph construction. When the adjacency relations among samples are computed to construct the graph structure, the complexity is $O\left({\mathit{mn}}^2\right)$ [3], where n is the number of data samples and m is the dimensionality of the sample feature.

The graph-based SSL methods do not meet the cluster assumption. The cluster assumption states that if points are located in the same cluster, they are likely to belong to the same class. The success of SSL relies highly on certain semi-supervised assumptions (SSA) on the samples' distribution, which includes manifold and cluster assumptions [1]. The manifold assumption indicates that the high dimensional data lie on a low-dimensional manifold. Graph-based methods have not explored the cluster information of input distribution, so they may connect data points of different classes in cluster boundaries by the Euclidean distances of their feature vectors.

In order to overcome the above shortcomings of the graph-based SSL methods, we propose a new Semi-supervised learning method based on SPArse model, named SSPA. With the SSPA model, following advantages are achieved: Firstly adjacency relations among samples can be achieved by sparse matrix multiplication with a complexity of $O\left({\mathit{sn}}^2\right)$ [9], where s is the average number of nonzero elements in each sparse representation and far less than the feature dimensionality m, so $O\left({\mathit{sn}}^2\right)⪯¡O\left({\mathit{mn}}^2\right)$. Secondly, we explore the fact that the dictionary in SSPA model has the same cluster information as the training data samples, and thus we propose a structured dictionary learning algorithm to learn a structured dictionary which has explicit cluster information. With the structured dictionary the structured sparse model can be used in SSPA to gain better accuracy.

The main contribution of the work is that we propose a new SSPA model, with two kinds of dictionaries: non-structured [10] and structured. The SSPA model has a lower computational complexity than the graph-based SSL methods. With the structured dictionary the SSPA model can utilize the cluster information to improve the classification accuracy. The experimental results demonstrate that the proposed methods are efficient and outperform state-of-the-art graph-based SSL methods.