Elsevier

Neurocomputing

Volume 74, Issue 9, April 2011, Pages 1478-1484
Neurocomputing

Letters
Kernel-view based discriminant approach for embedded feature extraction in high-dimensional space

https://doi.org/10.1016/j.neucom.2011.01.004Get rights and content

Abstract

Derived from the traditional manifold learning algorithms, local discriminant analysis methods identify the underlying submanifold structures while employing discriminative information for dimensionality reduction. Mathematically, they can all be unified into a graph embedding framework with different construction criteria. However, such learning algorithms are limited by the curse-of-dimensionality if the original data lie on the high-dimensional manifold. Different from the existing algorithms, we consider the discriminant embedding as a kernel analysis approach in the sample space, and a kernel-view based discriminant method is proposed for the embedded feature extraction, where both PCA pre-processing and the pruning of data can be avoided. Extensive experiments on the high-dimensional data sets show the robustness and outstanding performance of our proposed method.

Introduction

It is well known that many machine learning and data mining problems deal with the high-dimensional data representation and analysis. In the past few decades, numerous dimension reduction and feature extraction methods have been devoted to find the resultful feature representation of the original data. In the literature, principal component analysis (PCA) [1] and linear discriminant analysis (LDA) [2], [3] have been the most popular techniques. Moreover, they can be carried out in a reproducing kernel Hilbert space (RKHS) by making use of the well-known “kernel trick” [4], and the kernel discriminant method and its variants [5], [6], [7] are drawn in the recent years.

Different from the statistical feature extraction techniques that consider the global Euclidean structure, the theoretical basis of manifold learning methods, e.g., ISOMAP [8], LLE [9] and Laplacian eigenmap (LE) [10], depends on the observation that the high-dimensional data may reside on an intrinsic nonlinear manifold with much low dimensionality. Particularly, locality preserving projection (LPP) [11] is presented based on the LE idea. In order to find the discriminative submanifold structures embedded on the original data, some locally supervised learning techniques [12], [13], [14], [15], [16] are proposed, in the light of the locality preserving conception.

On the other hand, many real-world applications, such as image retrieval and pattern recognition, handle the high-dimensional data that bring the curse-of-dimensionality for the local discriminant analysis techniques. Generally, the existing local methods handle the high dimensionality of data in the following two ways:

  • Since the maximum margin criterion (MMC) has been successfully applied to classical LDA [17], it is employed to replace the ratio discriminant formulation with the subtraction one. The drawback of such an idea is that the computational expense usually depends on the dimensionality of data and it is hard to carry out if the original data lie on a high-dimensional manifold. A simple solution to this problem is to resize the data into a smaller size. Such an idea, however, would destroy the integrity of the original data.

  • By other means, PCA is usually used to reduce dimension primarily in such environments. Though the PCA pre-processing step can be considered to generate a new coordinate system, the local manifold structure cannot be preserved if a pruning of PCA energy is adopted.

In view of this intrinsic limitation, we propose a kernel-view based discriminant approach, namely KVDA, for embedded feature extraction of high-dimensional data. Different from other techniques, it is insensitive to the high dimensionality of data and PCA stage is unnecessary. It is noticeable that though local discriminative learning is conducted via a kernel approach in our work, the original sample space is involved rather than the RKHS.

The remainder of this paper is organized as follows. The problem statement is given in Section 2. The proposed KVDA is described in Section 3. A comprehensive set of comparison experiments on feature extraction and classification is given in Section 4, followed by the conclusion in Section 5. For convenience, the important notations used in the paper are listed in Table 1.

Section snippets

Problem statement

To discover the action on the data structure, the difference between PCA and local manifold learning methods for dimensionality reduction is considered. Taking LPP for example, we show the difference between PCA and LPP for the real data sets, namely, Ionosphere and Monks3 [18], in Fig. 1, where the representation of PCA and LPP for the data sets in the first two significant dimensions is illustrated. Evidently, LPP tends to preserve the local localities, while PCA makes the global distribution

Kernel-view based discriminant approach

In order to depict the supervised manifold structure hidden in high-dimensional data, two graphs, i.e., the intrinsic graph G and the penalty graph Gp, are generally constructed. Suppose that XRd×n denotes the data matrix consisting of n samples, S and Sp are respectively the adjacency matrices of the intrinsic and penalty graphs, their corresponding Laplacian matrices are indicated by L and Lp. Without loss of generality, discriminant embedding aims to find an optimal subspace, where the

Experiment

In this section, we evaluate the performance of the proposed method for feature extraction from the high-dimensional data, which simply takes three respective intraclass and extraclass neighbors to construct the intrinsic and penalty graphs. Nevertheless, it is noticeable that other graph criterions are also applicable. Several state-of-the-art methods, including PCA [1], PCA+LDA [3], DLDA [24], RLDA [25], SLPP [11], MFA [12], PCA+LSDA [26], LGMP [27], are involved to give the comparison. In

Conclusion

To overcome the limitation of existing local discriminative learning methods, a kernel-view based discriminant approach is developed to extract the embedded features in high-dimensional space. Different from other local methods, we consider the discriminant subspace learning as a kernel analysis approach so that the proposed method is insensitive to high dimensionality of data and the PCA pre-processing that may destroy the discriminative submanifold structure is unnecessary. Nevertheless, the

Acknowledgments

This work was supported by the Natural Science Foundation of China (Nos. 60873092 and 90820306), and the research grant funded by the research committee of University of Macau.

Miao Cheng received the B.E. degree in electronic information engineering and Ph.D. degree in computer science from Chongqing University, Chongqing, China. At present, he is a postdoctoral research fellow at the Department of Computer and Information Science, University of Macau. His current research interests include computer vision, data mining, machine learning and signal processing. He is a member of IEEE.

References (33)

  • J. Tenenbaum et al.

    A global geometric framework for nonlinear dimensionality reduction

    Science

    (2000)
  • S. Roweis et al.

    Nonlinear dimensionality reduction by locally linear embedding

    Science

    (2000)
  • M. Belkin, P. Niyogi, Laplacian eigenmaps and spectral techniques for embedding and clustering, in: Neural Information...
  • X. He et al.

    Face recognition using laplacianfaces

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2005)
  • S. Yan et al.

    Graph embedding and extensions: a general framework for dimensionality reduction

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2007)
  • X.L. Li et al.

    Discriminant locally linear embedding with high-order tensor data

    IEEE Trans Syst. Man Cybern. Part B: Cybern.

    (2008)
  • Cited by (0)

    Miao Cheng received the B.E. degree in electronic information engineering and Ph.D. degree in computer science from Chongqing University, Chongqing, China. At present, he is a postdoctoral research fellow at the Department of Computer and Information Science, University of Macau. His current research interests include computer vision, data mining, machine learning and signal processing. He is a member of IEEE.

    Bin Fang received the B.Eng. degree in electrical engineering from Xi’an Jiaotong University, Xi’an, China, the M.Sc. degree in electrical engineering from Sichuan University, Chengdu, China, and the Ph.D. degree in electrical engineering from the University of Hong Kong, Hong Kong, China. He is currently a Professor in the Department of Computer Science, Chongqing University, China. His research interests include computer vision, pattern recognition, medical image processing, biometrics applications, and document analysis. He is a senior member of IEEE.

    Chi-Man Pun received his B.Sc. and M.Sc. degrees in software engineering from the University of Macau in 1995 and 1998, respectively, and Ph.D. degree in computer science and Engineering from the Chinese University of Hong Kong in 2002. He is currently an Associate Professor at the Department of Computer and Information Science of the University of Macau. He has investigated several funded research projects and published more than 50 refereed scientific papers in international journals, books and conference proceedings. Dr. Pun has also been invited to serve as referee/reviewer and/or committee member for international journals and conferences. His research interests include content-based multimedia indexing and retrieval; digital watermarking; multimedia databases; image/video compression, analysis and processing; pattern recognition and computer vision, intelligent multimedia systems and applications. He is also a Senior Member of the IEEE.

    Yuan Yan Tang received the B.S. degree in electrical and computer engineering from Chongqing University, Chongqing, China, the M.Eng. degree in electrical engineering from the Graduate School of Post and Telecommunications, Beijing, China, and the Ph.D. degree in computer science from Concordia University, Montreal, QC, Canada. He is presently a Professor in the Department of Computer Science, Chongqing University, and a Professor in the Department of Computer and Information Science, University of Macau, and an Adjunct Professor in Computer Science, Concordia University. He is an Honorary Lecturer at the University of Hong Kong and an Advisory Professor at many institutes in China. His current interests include wavelet theory and applications, pattern recognition, image processing, document processing, artificial intelligence, parallel processing, Chinese computing, and VLSI architecture. He has published more than 250 technical papers and is the author/coauthor of 21 books/book chapters on subjects ranging from electrical engineering to computer science. Prof. Tang has served as the General Chair, Program Chair, and Committee Member for many international conferences. He was the General Chair of the 19th International Conference on Pattern Recognition (ICPR’06). He is the Founder and Editor-in-Chief of the International Journal on Wavelets, Multiresolution, and Information Processing (IJWMIP) and an Associate Editor of several international journals related to pattern recognition and artificial intelligence. He is an IEEE and IAPR Fellow.

    View full text