Skip to main content
SpringerLink
Log in

Speed up kernel discriminant analysis

  • Regular Paper
  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract

Linear discriminant analysis (LDA) has been a popular method for dimensionality reduction, which preserves class separability. The projection vectors are commonly obtained by maximizing the between-class covariance and simultaneously minimizing the within-class covariance. LDA can be performed either in the original input space or in the reproducing kernel Hilbert space (RKHS) into which data points are mapped, which leads to kernel discriminant analysis (KDA). When the data are highly nonlinear distributed, KDA can achieve better performance than LDA. However, computing the projective functions in KDA involves eigen-decomposition of kernel matrix, which is very expensive when a large number of training samples exist. In this paper, we present a new algorithm for kernel discriminant analysis, called Spectral Regression Kernel Discriminant Analysis (SRKDA). By using spectral graph analysis, SRKDA casts discriminant analysis into a regression framework, which facilitates both efficient computation and the use of regularization techniques. Specifically, SRKDA only needs to solve a set of regularized regression problems, and there is no eigenvector computation involved, which is a huge save of computational cost. The new formulation makes it very easy to develop incremental version of the algorithm, which can fully utilize the computational results of the existing training samples. Moreover, it is easy to produce sparse projections (Sparse KDA) with a L 1-norm regularizer. Extensive experiments on spoken letter, handwritten digit image and face image data demonstrate the effectiveness and efficiency of the proposed algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Baudat G., Anouar F.: Generalized discriminant analysis using a kernel approach. Neural. Comput. 12, 2385–2404 (2000)

    Article  Google Scholar 

  2. Burges C.J.C.: A tutorial on support vector machines for pattern recognition. Data. Min. Knowl. Discov. 2(2), 121–167 (1998)

    Article  Google Scholar 

  3. Cai, D., He, X., Han, J.: Efficient kernel discriminant analysis via spectral regression. In Proc. Int. Conf. on Data Mining (ICDM’07) (2007)

  4. Cai, D., He, X., Han, J.: Spectral regression: A unified approach for sparse subspace learning. In: Proceedings International Conference on Data Mining (ICDM’07) (2007)

  5. Cai, D., He, X., Han,J.: Spectral regression: a unified subspace learning framework for content-based image retrieval. In: Proceedings of the 15th ACM International Conference on Multimedia, Augsburg, Germany (2007)

  6. Cai D., He X., Han J.: SRDA: an efficient algorithm for large scale discriminant analysis. IEEE Trans. Knowl. Data. Eng. 20(1), 1–12 (2008)

    Article  Google Scholar 

  7. Cai, D., He, X., Zhang, W.V., Han, J.: Regularized locality preserving indexing via spectral regression. In: Proceedings of the 16th ACM Conference on Conference on Information and Knowledge Management (CIKM’07), pp. 741–750 (2007)

  8. Chakrabarti, K., Mehrotra, S.: Local dimensionality reduction: a new approach to indexing high dimensional spaces. In: Proceedings 2000 International Conference Very Large Data Bases (VLDB’00) (2000)

  9. Chakrabarti S., Roy S., Soundalgekar M.V.: Fast and accurate text classification via multiple linear discriminant projections. VLDB J. 12(2), 170–185 (2003)

    Article  Google Scholar 

  10. Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines, 2001. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.

  11. d’Aspremont, A., Chaoui, L.E., Jordan, M.I., Lanckriet, G.R.G.: A direct formulation for sparse PCA using semidefinite programming. In: Advances in Neural Information Processing Systems 17, (2004)

  12. Dietterich T.G., Bakiri G.: Solving multiclass learning problems via error-correcting output codes. J. Artif. Intell. Res. 2, 263–286 (1995)

    MATH  Google Scholar 

  13. Efron B., Hastie T., Johnstone I., Tibshirani R.: Least angle regression. Ann. Stat. 32(2), 407–499 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  14. Fanty, M.A., Cole, R.: Spoken letter recognition. In: Advances in Neural Information Processing Systems 3 (1990)

  15. Fukunaga, K.: Introduction to Statistical Pattern Recognition. Academic Press, 2nd edn (1990)

  16. Gaede V., Günther O.: Multidimensional access methods. ACM Comput. Surv. 30(2), 170–231 (1998)

    Article  Google Scholar 

  17. Golub, G.H., Loan, C.F.V. Matrix computations. Johns Hopkins University Press, 3rd edn (1996)

  18. Hastie T., Tibshirani R., Friedman J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer-Verlag, New York (2001)

    MATH  Google Scholar 

  19. Hull, J.J.: A database for handwritten text recognition research. IEEE Trans. Pattern Anal. Mach. Intell., 16(5) (1994)

  20. Jin, H., Ooi, B.C., Shen, H.T., Yu, C., Zhou, A.: An adaptive and efficient dimensionality reduction algorithm for high-dimensional indexing. In: Proceedigs 2003 International Conference on Data Engineering (ICDE’03) (2003)

  21. Micchelli, C.A.: Algebraic aspects of interpolation. In: Proceedings of Symposia in Applied Mathematics, vol. 36, pp. 81–102 (1986)

  22. Mika, S., Rätsch, G., Weston, J., Schölkopf B., Müller, K.-R.: Fisher discriminant analysis with kernels. In: Proceedings of IEEE Neural Networks for Signal Processing Workshop (NNSP), (1999)

  23. Mika, S., Smola, A., Schölkopf, B.: An improved training algorithm for kernel fisher discriminants. In: Proceedings AISTATS 2001. Morgan Kaufmann (2001)

  24. Moghaddam, B., Weiss, Y., Avidan S.: Spectral bounds for sparse PCA: Exact and greedy algorithms. In: Advances in Neural Information Processing Systems 18 (2005)

  25. Moghaddam, B., Weiss, Y., Avidan S.: Generalized spectral bounds for sparse LDA. In: ICML ’06: Proceedings of the 23rd International Conference on Machine learning, pp. 641–648 (2006)

  26. Park C.H., Park H.: Nonlinear discriminant analysis using kernel functions and the generalized singular value decomposition. SIAM J. Matrix Anal. Appl 27(1), 87–102 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  27. Schölkopf B., Smola A.J.: Learning with Kernels. MIT Press, Cambridge (2002)

    Google Scholar 

  28. Shen, H.T., Ooi, B.C., Zhou, X.: Towards effective indexing for very large video sequence database. In: Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, pp. 730–741, (2005)

  29. Shen, H.T., Zhou, X., Cui, B.: Indexing text and visual features for www images. In: 7th Asia Pacific Web Conference (APWeb2005), (2005)

  30. Shen H.T., Zhou X., Zhou A.: An adaptive and dynamic dimensionality reduction method for high-dimensional indexing. VLDB J. 16(2), 219–234 (2007)

    Article  Google Scholar 

  31. Stewart, G.W.: Matrix algorithms volume I: basic decompositions. SIAM (1998)

  32. Stewart, G.W.: Matrix algorithms volume II: eigensystems. SIAM (2001)

  33. Tao D., Li X., Wu X., Maybank S.J.: General tensor discriminant analysis and gabor features for gait recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(10), 1700–1715 (2007)

    Article  Google Scholar 

  34. Tao D., Li X., Wu X., Maybank S.J.: Geometric mean for subspace selection. IEEE Trans. Pattern Anal. Mach. Intell. 31(2), 260–274 (2009)

    Article  Google Scholar 

  35. Tao D., Tang X., Li X., Rui Y.: Kernel direct biased discriminant analysis: a new content-based image retrieval relevance feedback algorithm. IEEE Trans. Multimed. 8(4), 716–727 (2006)

    Article  MATH  Google Scholar 

  36. Vapnik V.N.: Statistical Learning Theory. Wiley, New York (1998)

    MATH  Google Scholar 

  37. Zou, H., Hastie, T., Tibshirani, R.: Sparse Principle Component Analysis. Technical Report, Statistics Department, Stanford University (2004)

Download references

Author information

Authors and Affiliations

  1. State Key Lab of CAD and CG, College of Computer Science, Zhejiang University, Hangzhou, China

    Deng Cai & Xiaofei He

  2. Department of Computer Science, University of Illinois at Urbana Champaign, Champaign, IL, USA

    Jiawei Han

Authors
  1. Deng Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaofei He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiawei Han
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deng Cai.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cai, D., He, X. & Han, J. Speed up kernel discriminant analysis. The VLDB Journal 20, 21–33 (2011). https://doi.org/10.1007/s00778-010-0189-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-010-0189-3

Keywords

Search

Navigation