Flexible constrained sparsity preserving embedding

Highlights

• Two non-linear semi-supervised embeddings are proposed.

• These methods elegantly integrate sparsity preserving and constrained embedding.

• The second framework provides a non-linear embedding and its out-of-sample extension.

• Classification performance after embedding is assessed on eight image datasets.

• KNN and SVM classifiers are used after getting the embedding.

• Experimental results on eight public image datasets show the outperformance of the methods.

Abstract

In this paper, two semi-supervised embedding methods are proposed, namely Constrained Sparsity Preserving Embedding (CSPE) and Flexible Constrained Sparsity Preserving Embedding (FCSPE). CSPE is a semi-supervised embedding method which can be considered as a semi-supervised extension of Sparsity Preserving Projections (SPP) integrated with the idea of in-class constraints. Both the labeled and unlabeled data can be utilized within the CSPE framework. However, CSPE does not have an out-of-sample extension since the projection of the unseen samples cannot be obtained directly. In order to have an inductive semi-supervised learning, i.e. being able to handle unseen samples, we propose FCSPE which can simultaneously provide a non-linear embedding and an approximate linear projection in one regression function. FCSPE simultaneously achieves the following: (i) the local sparse structures is preserved, (ii) the data samples with a same label are mapped onto one point in the projection space, and (iii) a linear projection that is the closest one to the non-linear embedding is estimated. Experimental results on eight public image data sets demonstrate the effectiveness of the proposed methods as well as their superiority to many competitive semi-supervised embedding techniques.

Introduction

In many real world applications, such as face recognition and text categorization, the data are usually provided in a high dimension space. In many real-world problems, collecting a large number of labeled samples is practically impossible. The reasons are twofold. Firstly, these labeled samples can be very few. Secondly, acquiring labels requires expensive human labor. To deal with this problem, semi-supervised embedding methods can be used to project the data in the high-dimensional space into a space with fewer dimensions.

A lot of methods for dimension reduction have proposed. Principal Component Analysis [1] (PCA) and Multidimensional Scaling [2] (MDS) are two classic linear unsupervised embedding methods. Linear Discriminant Analysis [1] (LDA) is a supervised method. In 2000, Locally Linear Embedding [3] (LLE) and Isometric Feature Mapping (ISOMAP) [4] were separately proposed in science which laid a foundation of manifold learning. Soon afterward, Belkin et al. proposed Laplacian Eigenmaps [5] (LE). He et al. proposed both Locality Preserving Projection [6] (LPP), essentially a linearized version of LE, and Neighborhood Preserving Embedding [7] (NPE), a linearized version of LLE. LPP and NPE can be interpreted in a general graph embedding framework with different choices of graph structure. Most of these methods are unsupervised methods. Afterwards, sparse representation [8], [9], [10] based methods have attracted extensive attention. Lai et al. proposed a 2-D feature extraction method called sparse 2-D projections for image feature extraction [11]. In [12], a robust tensor learning method called sparse tensor alignment (STA) is then proposed for unsupervised tensor feature extraction based on the alignment framework. In [13], multilinear sparse principal component analysis (MSPCA) inherits the sparsity from the sparse PCA and iteratively learns a series of sparse projections that capture most of the variation of the tensor data.

Sparsity Preserving Projection (SPP) is an unsupervised learning method [10]. It can be considered as an extension to NPE since the latter has a similar objective function. However, SPP utilizes sparse representation over the whole data to obtain the affinity matrix.

In the last decade, semi-supervised learning algorithms have been developed to effectively utilize a large amount of unlabeled samples as well as the limited number of labeled samples for real world applications [14], [15], [16], [17], [18], [19], [20], [21], [22]. In the past years, many graph-based methods for semi-supervised learning have been developed [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35].

Constrained Laplacian Eigenmaps [36] (CLE) is a semi-supervised embedding method. CLE constrains the solution space of Laplacian Eigenmaps only to contain embedding results that are consistent with the labels. Labeled points belonging to the same class are merged together, labeled points belonging to different classes are separated, and similar points are close to one another. Similarly, Constrained Graph Embedding [37] (CGE) tries to project the data points from a same class onto one single point in the projection space with a constraint matrix.

Flexible Manifold Embedding [38] (FME) is a label propagation method. FME simultaneously estimates the non-linear embedding of unlabeled samples and the linear regression over these non-linear representations. In [39], the authors propose a whole learning process that can provide the data graph and a linear regression within a same framework.

SPP is a successful unsupervised learning method. To extend SPP to a semi-supervised embedding method, we introduce the idea of in-class constraints in CGE into SPP and propose a new semi-supervised method for data embedding named Constrained Sparsity Preserving Embedding (CSPE). The weakness of CSPE is that it can not handle the new coming samples which means a cascade regression should be performed after the non-linear mapping is obtained by CSPE over the whole training samples. Inspired by FME, we add a regression term in the objective function to obtain an approximate linear projection simultaneously when non-linear embedding is estimated and proposed Flexible Constrained Sparsity Preserving Embedding (FCSPE). So in this paper, two semi-supervised embedding methods namely CSPE and FCSPE are proposed. Compared to the existing works, the proposed CSPE retains the advantages of both CGE and SPP. On the other hand, the proposed FCSPE simultaneously estimates the non-linear mapping over the training samples and the linear projection for solving the out-of-sample problem, which is usually not provided by existing graph-based semi-supervised non-linear mapping methods.

This paper is organized as follows. Section 2 reviews the related methods including LPP, SPP, CGE and FME. Section 3 introduces the two proposed semi-supervised methods. Section 4 presents performance evaluations on six face image databases: Yale, ORL, FERET, PIE, Extended Yale B and LFW (the original version and the aligned version), one handwriting image database USPS and an object image database COIL-20. Section 5 presents some concluding remarks.