Supervised locality pursuit embedding for pattern classification

Abstract

In pattern recognition research, dimensionality reduction techniques are widely used since it may be difficult to recognize multidimensional data especially if the number of samples in a data set is not large comparing with the dimensionality of data space. Locality pursuit embedding (LPE) is a recently proposed method for unsupervised linear dimensionality reduction. LPE seeks to preserve the local structure, which is usually more significant than the global structure preserved by principal component analysis (PCA) and linear discriminant analysis (LDA). In this paper, we investigate its extension, called supervised locality pursuit embedding (SLPE), using class labels of data points to enhance its discriminant power in their mapping into a low dimensional space. We compare the proposed SLPE approach with traditional LPE, PCA and LDA methods on real-world data sets including handwritten digits, character data set and face images. Experimental results demonstrate that SLPE is superior to other three methods in terms of recognition accuracy.

Introduction

In pattern recognition task, raw data acquired by sensors like cameras or scanners are often served as the input module to a recognition system as they are or after simple preprocessing. Being straightforward, the drawbacks of such approach are: a large data dimensionality makes recognition difficult and time-consuming on one hand; and the effect known as the curse of dimensionality unavoidably lowers the accuracy rate on the other hand. Therefore, a kind of dimensionality reduction is needed to eliminate unfavorable consequences of using multidimensional data for recognition. The underlying structure of multidimensional data can be characterized by a small number of parameters in many cases. It is also important to reduce the dimensionality of such data for visualizing the intrinsic structure.

In the past decades, there have been many methods proposed for dimensionality reduction [1], [2], [3], [4], [5], [6], [7], [15]. Two canonical forms of them are principal component analysis (PCA) and multidimensional scaling (MDS). Both of them are eigenvector methods aimed at modeling linear variability in the multidimensional space. PCA computes the linear projections of the greatest variance from the top eigenvectors of the data covariance matrix. MDS, however, computes the low dimensional embedding that best preserves pair wise distances between data points. The results of MDS will be equivalent to PCA if the similarity is Euclidean distance. Both methods are simple to implement and not prone to local minima. However, for the data on a nonlinear sub-manifold embedded in the feature space, the results given by PCA preserve only the global structure. In many cases, local structure is emphasized especially when using nearest neighbor classifier.

Locally linear embedding [6], [7] and Laplacian Eigenmap [3] are nonlinear local approaches proposed recently to discover the nonlinear structure of the manifold. The essence of the two methods is to map nearby points on a manifold to nearby points in a low dimensional space. Isomap [5] is a nonlinear global approach based on MDS and seeks to preserve the intrinsic geometry of the data. These nonlinear methods have achieved impressive results both on some benchmark artificial datasets and some real applications [8], [13]. Nevertheless, the nonlinearity makes them computationally expensive. In addition, the mappings derived from them are defined on the training set and how to evaluate a novel test data remains unclear.

Recently, an unsupervised linear dimensionality reduction method, locality pursuit embedding (LPE), was proposed and applied to real datasets [9], [10], [11], [12], [26]. LPE aims to preserve the local structure of the multidimensional structure instead of global structure preserved by PCA. In addition, LPE shares some similar properties compared with LLE such as a locality preserving character. However, their objective functions are totally different. LPE is the optimal linear approximation to the eigenfunctions of the Laplace Beltrami operator on the manifold [26].

In this paper, we describe a supervised variant of LPE, called the supervised locality pursuit embedding (SLPE) algorithm. Unlike LPE, SLPE projects high dimensional data to the embedded low space taking class membership relations into account. This allows obtaining well-separated clusters in the embedded space. It is worthwhile to highlight the discriminant power of SLPE by using class information besides inheriting the properties of LPE. Therefore, SLPE demonstrates powerful recognition performance when applied to some pattern recognition tasks.

The rest of the paper is organized as follows: Section 2 describes locality pursuit embedding versus PCA and LDA. The proposed supervised locality pursuit embedding is described in Section 3. In Section 4, we apply SLPE to some real datasets including handwritten digits, character dataset and face datasets to test its performance compared with PCA, LDA (linear discriminant analysis) and LPE. Finally, we provide some concluding remarks and suggestions for future work in Section 5.