Recursive Dimension Reduction for semisupervised learning

Abstract

Semisupervised Dimension Reduction (SDR) Using Trace Ratio Criterion (TR-FSDA) is an effective iterative SDR algorithm, which introduces a flexible regularization term ${\Vert \mathbf{F}-{\mathbf{X}}^T\mathbf{W}\Vert }^2$ to relax such a hard linear constraint in SDA that the low-dimensional representation $\mathbf{F}$ is constrained to lie in the linear subspace spanned by the data matrix $\mathbf{X}$. We, however, observe that TR-FSDA may take some meaningless features in the iteration and cannot be always guaranteed to converge. In this paper, we propose a novel method for SDR, referred to as Recursive Dimension Reduction for Semisupervised Learning (RDS). Instead of solving the non-trivial TR problem using the iterative algorithm of TR-FSDA, we solve the objective function of TR-FSDA using a newly-developed recursive procedure. In each iteration, only a projection vector and a one-dimensional data representation are produced by solving a standard Rayleigh Quotient problem. Our algorithm escapes from the convergence guarantee, since it directly solves the objective and requires no any iterative strategy in finding each of the projection vectors. The experiments on four face databases, one object database, one shape image database, and one Handwritten Digit database demonstrate the effectiveness of RDS.

Introduction

DIMENSION reduction (DR) has attracted much attention in pattern recognition, computer vision, etc., since it can effectively address the so-called “curse of dimensionality” problem. Two of the most well-known DR algorithms are Principal Component Analysis (PCA) [1] and Fisher Linear Discriminant (FLD) [2]. In addition, by considering different aspects of DR, different algorithms have been developed [1], [2], [3], [4], [5], [6], [7], [8], [40], [41], [42], [43], e.g., to yield nonnegative projection [40], [41] by using nonnegative matrix factorization and to improve discrimination by using the parallel vector field embedding algorithm [42], [43].

PCA and FLD are two linear subspace learning algorithms, which, however, cannot discover the essential data structures that are nonlinear. Recently, a number of manifold-based learning techniques, e.g., Isometric Feature Mapping (ISOMAP) [4], Local Linear Embedding (LLE) [5], and Laplacian Eigenmap (LE) [6] are developed to resolve this problem. The central idea of manifold learning is to find an intrinsic low-dimensional embedding of data. However, these classical approaches cannot map new coming samples. In order to solve this problem, He et al., proposed Locality Preserving Projections (LPP) [7]. Yang et al. [8] proposed a “classification-oriented” technique, called Unsupervised Discriminant Projection (UDP). Yan et al. [9] recently proposed a general formulation known as graph embedding providing a unified formulation of a broad set of DR techniques.

For supervised learning, we may require collecting a large number of data points. Directly labeling these data is not only time-consuming but also expensive. Furthermore, supervised learning algorithms may not obtain the promising results when label information is not sufficient. Therefore, semisupervised learning plays an important role to solve these problems. In the literature, many semisupervised learning algorithms for classification have been developed, e.g., Transductive SVM (TSVM) [10], [11], and graph-based semi-supervised learning algorithms [12], [13], [14], [15]. Among them, GFHF [12] and LGC [13] are two label propagation approaches designed for predicting the labels of unlabeled data in the training set, which, however, cannot deal with new coming data. The linear LapRLS [14] can be viewed as an “out-of-sample” extension of LGC/GFHF. The recent years have witnessed numerous researching activities on semisupervised DR [16], [17], [18], [19], [20] for different tasks. However, these algorithms suffer from such a constraint that the low-dimensional data representation $\mathbf{F}$ is constrained to lie within the linear space spanned by all the training samples. To relax this hard linear constraint, Nie et al. developed Flexible Manifold Embedding (FME) [21], which is a multidimensional extension of LapRLS. In the research [22], Nie et al. proposed Semisupervised Dimensionality Reduction via Virtual Label Regression (VLR), which can be viewed as a formulation of two-step FME with outlier detection.

Considering that in Semisupervised Discriminant Analysis (SDA) [19], [20], the manifold smoothness term is introduced in the objective function of FLD and the low-dimensional data representation $\mathbf{F}$ is constrained to be in the space spanned by all the training samples, Semisupervised Dimension Reduction Using Trace Ratio Criterion (TR-FSDA) is proposed to relax this constraint by modeling the mismatch between $\mathbf{F}$ and $h(\mathbf{X})={\mathbf{X}}^T\mathbf{W}$ [23]. To solve the resulted non-trivial Trace Ratio (TR) optimization problem, an iterative algorithm is designed to simultaneously find $\mathbf{F}$ and $\mathbf{W}$. TR-FSDA has demonstrated the promising results for different recognition tasks.

In this paper, we target to solve the problem for semisupervised DR using a well-designed recursive procedure. Our algorithm is based on TR-FSDA. Despite the exhibited performance advantage, TR-FSDA suffers from two restrictions. Firstly, the meaningful discriminant projection vectors may not be correctly found. TR-FSDA adopts an iterative algorithm to address the non-trivial TR optimization problem. At each iteration, it solves a problem similar to Maximum Margin Criterion (MMC) [24] which optimizes the problem ${\max }_{\mathbf{W},{\mathbf{W}}^T\mathbf{W}=\mathbf{I}}tr({\mathbf{W}}^T\mathbf{A}\mathbf{W}-{\mathbf{W}}^T\mathbf{B}\mathbf{W})$, where $tr(\cdot )$ denotes the trace, and $\mathbf{A}$ and $\mathbf{B}$ represent graph relationships that represent different types of information of data [9], respectively. As claimed in [24], the most meaningful discriminant projection vectors should be selected as the eigenvectors corresponding to the eigen values more than or equal to zeros of matrix $\mathbf{A}-\mathbf{B}$. TR-FSDA does not take into account this problem and arbitrarily chooses the number of the discriminant projection axes at each iteration, such that the used meaningless discriminant projection vectors from the previous iterations may affect the generation of optimal discriminant projection vectors in the following iterations. Second, the convergence cannot be guranteed. At the $t\mathrm{th}$ iteration, TR-FSDA requires the calculation of matrix ${\mathbf{Z}}_t={({\lambda }_t+{\lambda }_t{\overline{\mathbf{L}}}_a-{\overline{\mathbf{L}}}_b)}^{-1}$, where ${\lambda }_t$ is the TR value calculated from the projection matrix of the previous step, and the definitions on ${\overline{\mathbf{L}}}_a$ and ${\overline{\mathbf{L}}}_b$ can be found in [23]. ${\mathbf{Z}}_t$ must be positive to ensure the convergence of TR-FSDA, which is not always true in real applications and depends on the values of the parameters involved in TR-FSDA. In order to solve this problem, Huang et al., [23] proposed to ignore the parameter combination that causes the non-convergence of TR-FSDA. However, doing so is very absurd, since the parameters are used to balance different terms to improve the recognition results. Thus, TR-FSDA may achieve undesired results. To address these problems, a new recursive procedure is designed to calculate $\mathbf{F}$ and $\mathbf{W}$. At each iteration, our approach is to solve a Rayleigh Quotient rather than non-trivial TR optimal problem and thus has no problem existed in TR-FSDA.