Skip to main content
SpringerLink
Log in

An efficient KPCA algorithm based on feature correlation evaluation

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Classic kernel principal component analysis (KPCA) is less computationally efficient when extracting features from large data sets. In this paper, we propose an algorithm, that is, efficient KPCA (EKPCA), that enhances the computational efficiency of KPCA by using a linear combination of a small portion of training samples, referred to as basic patterns, to approximately express the KPCA feature extractor, that is, the eigenvector of the covariance matrix in the feature extraction. We show that the feature correlation (i.e., the correlation between different feature components) can be evaluated by the cosine distance between the kernel vectors, which are the column vectors in the kernel matrix. The proposed algorithm can be easily implemented. It first uses feature correlation evaluation to determine the basic patterns and then uses these to reconstruct the KPCA model, perform feature extraction, and classify the test samples. Since there are usually many fewer basic patterns than training samples, EKPCA feature extraction is much more computationally efficient than that of KPCA. Experimental results on several benchmark data sets show that EKPCA is much faster than KPCA while achieving similar classification performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Schölkopf B, Smola A, Muller KR (1998) Nonlinear component analysis as a kernel eigenvalue problem. Neural Comput 10(5):1299–1319

    Article  Google Scholar 

  2. Muller KR, Mika S, Ratsch G, Tsuda K et al (2001) An introduction to kernel-based learning algorithms. IEEE Trans Neural Netw 12(2):181–201

    Article  Google Scholar 

  3. Shawe-Taylor J, Cristianini N (2004) Kernel methods for pattern analysis. Cambridge University Press, Cambridge

    Book  Google Scholar 

  4. Scholkopf B, Mika S, Burges CJC, Knirsch P et al (1999) Input space versus feature space in kernel-based methods. IEEE Trans Neural Netw 10(5):1000–1017

    Article  Google Scholar 

  5. Honeine P (2012) Online kernel principal component analysis: a reduced-order model. IEEE Trans Pattern Anal Mach Intell 34(9):1814–1826

    Article  Google Scholar 

  6. Abrahamsen TJ, Hansen LK (2011) A cure for variance inflation in high dimensional kernel principal component analysis. J Mach Learn Res 12(6):2027–2044

    MathSciNet  MATH  Google Scholar 

  7. Taouali O, Elaissi I, Messaoud H (2012) Online identification of nonlinear system using reduced kernel principal component analysis. Neural Comput Appl 21:161–169

    Article  Google Scholar 

  8. Rasmussen PM, Abrahamsen TJ, Madsen KH, Hansen LK (2012) Nonlinear denoising and analysis of neuroimages with kernel principal component analysis and pre-image estimation. NeuroImage 60(3):1807–1818

    Article  Google Scholar 

  9. Honeine P, Richard C (2011) Preimage problem in kernel-based machine learning. Signal Process Mag IEEE 28(2):77–88

    Article  Google Scholar 

  10. Mika S, Schölkopf B, Smola A, Müller KR et al (1999) Kernel PCA and de-noising in feature spaces. Adv Neural Inform Process Syst 11(1):536–542

    Google Scholar 

  11. Zheng WS, Lai JH, Yuen PC (2010) Penalized preimage learning in kernel principal component analysis. IEEE Trans Neural Netw 21(4):551–570

    Article  Google Scholar 

  12. Kim KI, Franz MO, Scholkopf B (2005) Iterative kernel principal component analysis for image modeling. IEEE Trans Pattern Anal Mach Intell 27(9):1351–1366

    Article  Google Scholar 

  13. Liu C (2004) Gabor-based kernel PCA with fractional power polynomial models for face recognition. IEEE Trans Pattern Anal Mach Intell 26(5):572–581

    Article  Google Scholar 

  14. De la Torre F, Nguyen MH (2008) Parameterized kernel principal component analysis: Theory and applications to supervised and unsupervised image alignment. In: IEEE conference on computer vision and pattern recognition, 2008. CVPR 2008, pp. 1–8

  15. Li J, Li X, Tao D (2008) KPCA for semantic object extraction in images. Pattern Recogn 41(10):3244–3250

    Article  MATH  Google Scholar 

  16. Hotta K (2012) Local co-occurrence features in subspace obtained by KPCA of local blob visual words for scene classification. Pattern Recogn 45(10):3687–3694

    Article  Google Scholar 

  17. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86

    Article  Google Scholar 

  18. Yang J, Zhang D, Frangi AF (2004) Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Trans Pattern Anal Mach Intell 26(1):131–137

    Article  Google Scholar 

  19. Zhang D, Zhou ZH (2005) (2D) 2PCA: two-directional two-dimensional PCA for efficient face representation and recognition. Neurocomputing 69(1):224–231

    Article  Google Scholar 

  20. Xu Y, Zhang D, Yang JY (2010) A feature extraction method for use with bimodal biometrics. Pattern Recogn 43(3):1106–1115

    Article  MATH  Google Scholar 

  21. Vapnik V (1999) The nature of statistical learning theory. Springer, Berlin

    Google Scholar 

  22. Boser BE, Guyon IM, Vapnik VN (1992) A training algorithm for optimal margin classifiers. In: Proceedings of the fifth annual workshop on Computational learning theory, pp 144–152

  23. Lee D, Jung KH, Lee J (2009) Constructing sparse kernel machines using attractors. IEEE Trans Neural Netw 20(4):721–729

    Article  Google Scholar 

  24. Rosipal R, Girolami M (2001) An expectation-maximization approach to nonlinear component analysis. Neural Comput 13(3):505–510

    Article  MATH  Google Scholar 

  25. Zheng W, Zou C, Zhao L (2005) An improved algorithm for kernel principal component analysis. Neural Process Lett 22(1):49–56

    Article  Google Scholar 

  26. Schraudolph NN, Gunter S, Vishwanathan S (2007) Fast iterative kernel PCA. Presented at the advances in neural information processing systems

  27. Moerland P (2000) An on-line EM algorithm applied to kernel PCA, Technical report, IDIAP2000

  28. Gnecco G, Sanguineti M (2009) Accuracy of suboptimal solutions to kernel principal component analysis. Comput Optim Appl 42(2):265–287

    Article  MathSciNet  MATH  Google Scholar 

  29. Xu Y, Zhang D, Song F, Yang JY et al (2007) A method for speeding up feature extraction based on KPCA. Neurocomputing 70(4):1056–1061

    Article  Google Scholar 

  30. Kim SW, Oommen BJ (2004) On using prototype reduction schemes to optimize kernel-based nonlinear subspace methods. Pattern Recogn 37(2):227–239

    Article  MATH  Google Scholar 

  31. Chin TJ, Suter D (2007) Incremental kernel principal component analysis. IEEE Trans Image Process 16(6):1662–1674

    Article  MathSciNet  Google Scholar 

  32. Alzate C, Suykens J (2008) Kernel component analysis using an epsilon-insensitive robust loss function. IEEE Trans Neural Netw 19(9):1583–1598

    Article  Google Scholar 

  33. Xu Y, Lin C, Zhao W (2010) Producing computationally efficient KPCA-based feature extraction for classification problems. Electron Lett 46(6):452–453

    Article  Google Scholar 

  34. Wright J, Yang AY, Ganesh A, Sastry SS et al (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31(2):210–227

    Article  Google Scholar 

  35. Wang S-J, Yang J, Sun M-F, Peng X-J et al (2012) Sparse tensor discriminant color space for face verification. Neural Netw Learn Syst IEEE Trans 23(6):876–888

    Article  Google Scholar 

  36. Wang S, Yang J, Zhang N, Zhou C (2011) Tensor discriminant color space for face recognition. IEEE Trans Image Process 20(9):2490–2501

    Article  MathSciNet  Google Scholar 

  37. Wang S-J, Zhou C-G, Zhang N, Peng X-J et al (2011) Face recognition using second-order discriminant tensor subspace analysis. Neurocomputing 74(12):2142–2156

    Article  Google Scholar 

  38. Wang S-J, Chen H-L, Peng X-J, Zhou C-G (2011) Exponential locality preserving projections for small sample size problem. Neurocomputing 74(17):3654–3662

    Article  Google Scholar 

  39. Ishii T, Ashihara M, Abe S (2008) Kernel discriminant analysis based feature selection. Neurocomputing 71(13):2544–2552

    Article  Google Scholar 

  40. Kim JS, Scott CD (2010) L2 kernel classification. IEEE Trans Pattern Anal Mach Intell 32(10):1822–1831

    Article  Google Scholar 

  41. Tian J, Li M, Chen F (2009) A hybrid classification algorithm based on coevolutionary EBFNN and domain covering method. Neural Comput Appl 18(3):293–308

    Article  Google Scholar 

  42. Vert JP, Tsuda K, Schölkopf B (2004) A primer on kernel methods. In: Schölkopf B, Tsuda K, Vert JP (eds) Kernel methods in computational biology, MIT Press, Cambridge, pp 35–70. http://www.amazon.com/Kernel-Methods-Computational-Biology-Molecular/dp/0262195097#reader_0262195097

  43. Yang ZR (2006) A novel radial basis function neural network for discriminant analysis. IEEE Trans Neural Netw 17(3):604–612

    Article  Google Scholar 

  44. Xu Y, Zhu Q, Wang J (2011) Breast cancer diagnosis based on a kernel orthogonal transform. Neural Comput Appl 21(8):1865–1870

    Article  Google Scholar 

Download references

Acknowledgments

This article is partly supported by NSFC under grants Nos. 61263032, 61071179, 11061014, and 61202276, Jiangxi Provincial Natural Science Foundation of China under Grant 2010GQS0027, as well as the Science and Technology Foundation of Jiangxi Educational Committee of China (GJJ12309).

Author information

Authors and Affiliations

  1. School of Basic Science, East China Jiaotong University, Nanchang, 330013, China

    Zizhu Fan, Baogen Xu & Pengzhi Tang

  2. Department of Computing, The Hong Kong Polytechnic University, Kowloon, Hong Kong, China

    Jinghua Wang

Authors
  1. Zizhu Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinghua Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Baogen Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pengzhi Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zizhu Fan.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fan, Z., Wang, J., Xu, B. et al. An efficient KPCA algorithm based on feature correlation evaluation. Neural Comput & Applic 24, 1795–1806 (2014). https://doi.org/10.1007/s00521-013-1424-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-013-1424-9

Keywords

Search

Navigation