Kernel class-wise locality preserving projection

Abstract

In the recent years, the pattern recognition community paid more attention to a new kind of feature extraction method, the manifold learning methods, which attempt to project the original data into a lower dimensional feature space by preserving the local neighborhood structure. Among them, locality preserving projection (LPP) is one of the most promising feature extraction techniques. However, when LPP is applied to the classification tasks, it shows some limitations, such as the ignorance of the label information. In this paper, we propose a novel local structure based feature extraction method, called class-wise locality preserving projection (CLPP). CLPP utilizes class information to guide the procedure of feature extraction. In CLPP, the local structure of the original data is constructed according to a certain kind of similarity between data points, which takes special consideration of both the local information and the class information. The kernelized (nonlinear) counterpart of this linear feature extractor is also established in the paper. Moreover, a kernel version of CLPP namely Kernel CLPP (KCLPP) is developed through applying the kernel trick to CLPP to increase its performance on nonlinear feature extraction. Experiments on ORL face database and YALE face database are performed to test and evaluate the proposed algorithm.

Introduction

In pattern recognition, feature extraction techniques are widely employed to reduce the dimensionality of data and to enhance the discriminatory information [7]. Most feature extraction methods have focused on finding the linear transformation to project the data from a high-dimensional input space into a lower dimensional feature space, and the feature vector in the feature space contains all the necessary discriminative information. In the past several decades, many dimensionality reduction techniques have been proposed. The most well-known feature extraction methods may be principal component analysis (PCA) [1] and linear discriminant analysis (LDA) [2]. PCA seeks a linear optimal transformation matrix to minimize the mean squared error criterion, and the optimal matrix is constituted by the largest eigenvectors (called principal components) of the sample covariance. The purpose of PCA is to keep the information in terms of variance as much as possible. Linear discriminant analysis (also called Fisher’s linear discriminant) is another popular linear dimensionality reduction method. In many applications, LDA has proven to be much more effective than PCA. In the previous work, PCA was generalized to the nonlinear curves such as principal curves [6] and its extension such as principal surfaces [3]. Principal curves and principal surfaces are the nonlinear generalizations of principal components and subspaces respectively. It has turned out that discretized principal curves are essentially equivalent to self-organizing maps (SOM) [13], [16]. SOM is a nonparametric latent variable model with a topological constraint, such as lines, squares, or hexagonal grids and its mapping is similar to a discrete self-similarity principle for a principal manifold. SOM is a data driven dimensionality reduction method. SOM is regarded as an approximation of the principal surface. SOM was extended to Visualisation-induced SOM (ViSOM) [27]. ViSOM represents a discrete principal curve or surface, and ViSOM produces a smooth and graded mesh in the data space and captures the nonlinear manifold of data [25]. Moreover, other nonlinear manifold algorithms have been proposed, such as Locally Linear Embedding (LLE) [17] and Isomap [22]. LLE regards dimensionality reduction as geometrical perspective, while Isomap utilizes geodesic distances to represent connected graphs and relationship among data. Both LLE and Isomap preserve the neighborhoods and geometric relationships of the data. Isomap and LLE map easily the training data points in the reduced dimensional space, but it is hard to locate the test data points. Locality preserving projection [8] locates easily the new data point in the reduced representation space. But LPP is a linear dimensionality reduction method and has no the sufficient nonlinear discriminant power for linearly non-separable classes. Kernel methods have been widely used to overcome the limitation of some linear feature extraction and classification. Kernel-based learning feature extraction methods, such as Kernel Principal Component Analysis (KPCA) [19], [11], Kernel Discriminant Analysis (KDA) [14], [26] and Support Vector Machine (SVM) [10], [24], were widely used in the pattern recognition and machine learning areas [4], [23]. Other research topics relative to kernel-based learning have attracted researchers more and more attention [12], [15], [18], [20]. LPP was successfully applied in the pattern recognition and information retrieval areas. For example, LPP based feature extraction method namely Laplacianfaces was proposed for face recognition [9]. LPP constructs the adjacency graph by doing the nearest neighbor search, and the original data are mapped to the low dimensional space for feature extraction. LPP performs well on many practical applications, such as audio, video, text documents retrieval. Moreover, all of these methods are completely unsupervised with regard to the class labels of the data, and have little to do with discriminative features optimal for classification. In this paper, a novel local structure based feature extraction method based on the idea of LPP, namely class-wise locality preserving projection (CLPP), is proposed to enhance the class structure of the data. Unlike the unsupervised learning scheme of LPP, CLPP follows the supervised learning scheme, i.e. it uses the class information to model the manifold structure. In CLPP, the local structure of the original data is constructed according to a certain way of constructing the nearest neighbor graph, which takes special consideration of both the local information and the class information. Moreover, we improve CLPP on nonlinear feature extraction with kernel trick to develop Kernel CLPP algorithm and make CLPP a robust technique for the feature extraction tasks.

The rest of this paper is organized as follows. The detailed formulation and development of KCLPP are described in Section 2. An evaluation of the proposed method on two databases is reported in Section 3. Finally, Section 4 draws a conclusion and opens a prospective for future work.