Learning from normalized local and global discriminative information for semi-supervised regression and dimensionality reduction

Abstract

Semi-supervised dimensionality reduction is one of the important topics in pattern recognition and machine learning. During the past decade, Laplacian Regularized Least Square (LapRLS) and Semi-supervised Discriminant Analysis (SDA) are the two widely-used semi-supervised dimensionality reduction methods. In this paper, we show that SDA and LapRLS can be unified into a constrained manifold regularized least square framework. The manifold term, however, cannot fully utilize the underlying discriminative information. We thus introduce a new and effective semi-supervised dimensionality reduction method, called Learning from Local and Global Information (LLGDI), to solve the problem. The proposed LLGDI method adopts a set of local classification functions to preserve both local geometrical and discriminative information of dataset. It also adopts a global classification function to preserve the global discriminative information, and an uncorrelated constraint to calculate the projection matrix for simultaneously solving regression and dimensionality reduction problem. As a result, the LLGDI method is able to preserve local discriminative, manifold information as well as the global discriminative information. Theoretical analysis and extensive simulations presented in the paper show the effectiveness of the LLGDI algorithm. The results also demonstrate LLGDI can achieve superior performance compared with other existing methods.

Introduction

Dealing with high-dimensional data has always been a major problem with the research of pattern recognition and machine learning. Typical applications include face recognition, document categorization, and image retrieval. Thus, finding a low-dimensional representation of high-dimensional space is of great practical importance. The goal of dimensionality reduction is to reduce the complexity of input space and embed high-dimensional space into a low-dimensional space while keeping most of the desired intrinsic information [16], [18], [28], [36], [38], [39], [40], [41], [42], [43]. Among all the dimensionality reduction techniques, Principle Compent Analysis (PCA) [19] and Linear Discriminant Analysis (LDA) [1] are the two most popular methods. PCA pursues the direction of maximum variance for optimal reconstruction, while LDA, a supervised method, finds the optimal projection V maximizing the between-class scatter matrix S_b and minimizing the within-class scatter matrix S_w in a low-dimensional subspace. Due to the utilization of label information, LDA can achieve better classification results compared with PCA given sufficient labeled samples are provided [1], [4].

In general, supervised methods can deliver better performance than unsupervised methods, but obtaining sufficient number of labeled data for training can be problematic because labeling large number of samples is costly and laborious. On the other hand, unlabeled samples are abundant and can easily be obtained in numerous real world cases. Compared to supervised learning approaches that only rely on labeled training data, the idea of semi-supervised learning is to incorporate labeled and unlabeled data together to improve learning performance [2], [3], [12], [13], [14], [44], [45]. In brief, semi-supervised learning can be perceived as a framework that can provide efficient alternative to labeling unlabeled data. Well-known semi-supervised learning methods include Gaussian Field and Harmonic Fuction (GFHF) [45], Learning from Local and Global Consistency (LLGC) [44] and Special Label Propagation (SLP) [12]. These methods work in a transductive way by propagating label information from labeled set into unlabeled set through label propagation. This approach is efficient but it cannot predict class labels of new-coming samples. This drawback usually results in the out-of-sample problem. In contrast, semi-supervised dimensionality reduction methods not only reduce the dimensionality, but also naturally solve the out-of-sample problem. Thus semi-supervised methods can usually deliver better results when dealing with real-world applications.

The two widely-used semi-supervised methods are Semi-supervised Discriminant Analysis (SDA) [3] and Laplacian Regularized Least Square (LapRLS) [9]. These two methods share the same concept of dimensionality reduction, i.e. they first construct a graph Laplacian matrix to approximate the manifold structure by using both labeled and unlabeled samples. They then perform dimensionality reduction by adding the graph Laplacian matrix as a regularized term to the original objective function of LDA and Regularized Least Square (RLS). As a result, the discriminative structure embedded in the labeled samples and the geometrical structure embedded in labeled and unlabeled data can be preserved. In fact Lap-RLS is essentially derived from the perspective of regression instead of classification. Lap-RLS can be perceived as a training method that is aimed at training a linear classification model by regressing labeled set on the class label, while SDA is a subspace learning method which is aimed for solving classification problems. Though they both are stemmed from different supervised methods, we in this paper show that both SDA and Lap-RLS can be unified under a regularized least square framework. As a result, both of them are able to solve regression as well as subspace learning problems.

The connection and theoretical similarities between SDA and LapRLS can be elaborated under the least square framework. It should be noted that the regression term in LapRLS and the least square framework is supervised, which mean these two methods utilize a labeled set to train a linear classification function. Since the number of labeled data is relatively small compared with unlabeled data, training a linear classification function under a small sample size can be ineffective [21]. Another issue of semi-supervised method is the utilization of data samples to construct a graph that is used for characterizing the local structure of data manifold. In SDA and Lap-RLS, local structure is preserved by using a manifold regularized term defined on the affinity matrix of Gaussian function. But these Laplacian matrixes cannot capture the discriminative information of classes. This is essential when handling classification problems. In addition, the Gaussian function based affinity matrix is found to be over sensitive to the Gaussian variance; only a slight variation on the variance may affect the results significantly. Thus, Gaussian function based affinity matrix is not a popular method for handling complicated image classification and visualization problems. Instead of using Gaussian function for graph construction, several methods including Locally Linear Reconstruction [22], [23], Local Regression and Global Alignment [30], [31] and Local Spline Regression [26], [27] have then been proposed.

In this paper, we introduce a newly developed method, Learning from Local and Global Discriminative Information (LLGDI), for solving the above semi-supervised dimensionality reduction problems. The proposed LLGDI aims to train a classification function by utilizing all availanle data points. Specifically, our proposed method first relaxes the original supervised regression term making it a loss term and a global regression regularized term. The loss term measures the inconsistency between the predicted and initial labels on a labeled set, while the global regression regularized term aims to train the classification function as well as to calculate the projection matrix for out-of-sample problem. In addition, in order to characterize both manifold and discriminative structure embedded in a dataset, LLGDI employs a set of local classification function for each data point to predict the label of its neighboring points. In this way, both local and global discriminative information of a dataset can be preserved by using the LLGDI method. Also, in order to handle the subspace learning problem, we have also introduced an uncorrelated constraint into the objective function of LLGDI. As a result, both regression and subspace learning problems can be solved at the same time.

The main contributions of this work are as follows. First, we address the SDA method into a least square framework and establish the connections between SDA and LapRLS. Second, in order to relaxe the limitations of the least square framework of SDA, we develop a new method, called LLGDI. The new method can preserve the local geometrical and discriminative information of a dataset by using a normalized local discriminative manifold regularization term. Third, we extend the LLGDI method to perform dimensionality reduction by including a relaxed uncorrelated constraint to the objective function. As a result, both regression and subspace learning problems can be solved simultaneously. Finally, the relationship between LLGDI and other state-of-the-art methods are analyzed. Theoretical analysis shows that many other semi-supervised methods are different the special cases of the LLGDI method.

This paper is organized as follows. In Section 2, the notations and a brief review of LDA, MR and SDA are detailed. In Section 3, the equivalence between SDA and Lap-RLS under a constrained regularized least square framework is derived. Section 4 presents the proposed LLGDI method for semi-supervised regression and dimensionality reduction through the introduction of a normalized local discriminative manifold regularization term. Discussion on the relationship between LLGDI and other state-of-the-art semi-supervised methods is also included. Section 5 demonstrates the extensive simulations and the final conclusions are drawn in Section 6.