Supervised locality pursuit embedding for pattern classification
Introduction
In pattern recognition task, raw data acquired by sensors like cameras or scanners are often served as the input module to a recognition system as they are or after simple preprocessing. Being straightforward, the drawbacks of such approach are: a large data dimensionality makes recognition difficult and time-consuming on one hand; and the effect known as the curse of dimensionality unavoidably lowers the accuracy rate on the other hand. Therefore, a kind of dimensionality reduction is needed to eliminate unfavorable consequences of using multidimensional data for recognition. The underlying structure of multidimensional data can be characterized by a small number of parameters in many cases. It is also important to reduce the dimensionality of such data for visualizing the intrinsic structure.
In the past decades, there have been many methods proposed for dimensionality reduction [1], [2], [3], [4], [5], [6], [7], [15]. Two canonical forms of them are principal component analysis (PCA) and multidimensional scaling (MDS). Both of them are eigenvector methods aimed at modeling linear variability in the multidimensional space. PCA computes the linear projections of the greatest variance from the top eigenvectors of the data covariance matrix. MDS, however, computes the low dimensional embedding that best preserves pair wise distances between data points. The results of MDS will be equivalent to PCA if the similarity is Euclidean distance. Both methods are simple to implement and not prone to local minima. However, for the data on a nonlinear sub-manifold embedded in the feature space, the results given by PCA preserve only the global structure. In many cases, local structure is emphasized especially when using nearest neighbor classifier.
Locally linear embedding [6], [7] and Laplacian Eigenmap [3] are nonlinear local approaches proposed recently to discover the nonlinear structure of the manifold. The essence of the two methods is to map nearby points on a manifold to nearby points in a low dimensional space. Isomap [5] is a nonlinear global approach based on MDS and seeks to preserve the intrinsic geometry of the data. These nonlinear methods have achieved impressive results both on some benchmark artificial datasets and some real applications [8], [13]. Nevertheless, the nonlinearity makes them computationally expensive. In addition, the mappings derived from them are defined on the training set and how to evaluate a novel test data remains unclear.
Recently, an unsupervised linear dimensionality reduction method, locality pursuit embedding (LPE), was proposed and applied to real datasets [9], [10], [11], [12], [26]. LPE aims to preserve the local structure of the multidimensional structure instead of global structure preserved by PCA. In addition, LPE shares some similar properties compared with LLE such as a locality preserving character. However, their objective functions are totally different. LPE is the optimal linear approximation to the eigenfunctions of the Laplace Beltrami operator on the manifold [26].
In this paper, we describe a supervised variant of LPE, called the supervised locality pursuit embedding (SLPE) algorithm. Unlike LPE, SLPE projects high dimensional data to the embedded low space taking class membership relations into account. This allows obtaining well-separated clusters in the embedded space. It is worthwhile to highlight the discriminant power of SLPE by using class information besides inheriting the properties of LPE. Therefore, SLPE demonstrates powerful recognition performance when applied to some pattern recognition tasks.
The rest of the paper is organized as follows: Section 2 describes locality pursuit embedding versus PCA and LDA. The proposed supervised locality pursuit embedding is described in Section 3. In Section 4, we apply SLPE to some real datasets including handwritten digits, character dataset and face datasets to test its performance compared with PCA, LDA (linear discriminant analysis) and LPE. Finally, we provide some concluding remarks and suggestions for future work in Section 5.
Section snippets
LPE versus PCA and LDA
More formally, let us consider a set of M sample images taking values in an n-dimensional image space X={x1,x2,…,xM}, and assume that each image belongs to one of c classes {C1,C2,…,Cc}. PCA seeks a linear transformation mapping the original n-dimensional image space into an r-dimensional feature space, where r<n. Then the transformed new feature vectors are defined as follows:
The total scatter matrix of original sample images ST is defined aswhere M
Supervised locality pursuit embedding
From the above analysis, both PCA and LPE are unsupervised learning methods. They do not take the class membership relation into account. Imprecisely speaking, one of the differences between them lies in the global or local preserving property. The locality preserving property leads to the fact that LPE outperforms PCA in [9], [10], [11], [12], [26]. While PCA and LDA are all global methods, LDA utilizes the class information to enhance its discriminant ability. That is the reason why LDA
Experimental results
In the Section 3, the results shown in Fig. 1 indicate that SLPE can have more discriminant power than PCA, LDA and LPE. In this section, several experiments are carried out on different datasets to show the accuracy of our proposed SLPE for pattern classification.
Discussion and future work
A general framework for supervised LPE was proposed in this paper. Experiments on a number of data sets demonstrated that SLPE is a powerful feature extraction method, which when coupled with simple classifiers can yield very promising recognition results. SLPE takes the class membership information into account besides holding the locality preserving property of LPE. Since both local structure and discriminant information are important for classification, SLPE outperforms the traditional LPE,
Acknowledgements
The authors wish to acknowledge that this work is supported by Fundamental Project of Shanghai under grant number 03DZ14015.
References (29)
- et al.
A fuzzy neural network for pattern classification and feature selection
Fuzzy Sets Syst.
(2002) - et al.
Locality pursuit embedding
Pattern Recognit.
(2004) Principal Component Analysis
(1986)- et al.
Data visualization by multidimensional scaling: a deterministic annealing approach
Pattern Recognit.
(1999) - M. Belkin, P. Niyogi, Laplacian eigenmaps and spectral techniques for embedding and clustering, Proceedings of the...
- et al.
Learning the parts of objects by non-negative matrix factorization
Nature
(1999) - et al.
Independent component representations for face recognition
Proceedings of the SPIE
(1998) - et al.
Nonlinear dimensionality reduction by locally linear embedding
Science
(2000) A global geometric framework for nonlinear dimensionality reduction
Science
(2000)- Zhonglong Zheng, Jie Yang, Extended LLE with Gabor Wavelet for Face Recognition, The 17th Australian Joint Conference...
Face recognition using laplacianfaces
IEEE Trans. PAMI
Cited by (14)
Group sparse autoencoder
2017, Image and Vision ComputingCitation Excerpt :The ℓ2,1 is approximated as a l2-norm of every column in the low-rank representation (LRR) of the original matrix. It is well known in the machine learning community that supervised feature extraction usually leads to better classification [17]. For instance, incorporating Fisher criterion for determining projections in subspace methods reduces the intra-class variability and increases the inter-class variability, thereby increasing discrimination capabilities.
Exemplar based Laplacian Discriminant Projection
2011, Expert Systems with ApplicationsFacial feature localization based on an improved active shape model
2008, Information SciencesCitation Excerpt :Some original sample images of SJTU dataset are shown in Fig. 4. After preprocessing by a face detecting system [26], the detected face images serve as the inputs of the proposed ASM algorithm. Three kinds of improvements upon the original ASM are proposed in this paper.
Deep Classified Autoencoder for Lithofacies Identification
2022, IEEE Transactions on Geoscience and Remote SensingSemi-supervised deep autoencoder for seismic facies classification
2021, Geophysical ProspectingLaplacian MinMax discriminant analysis and its applications
2010, Tien Tzu Hsueh Pao/Acta Electronica Sinica