Skip to main content
SpringerLink
Log in

Superlinear Convergence of a General Algorithm for the Generalized Foley–Sammon Discriminant Analysis

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

Linear Discriminant Analysis (LDA) is one of the most efficient statistical approaches for feature extraction and dimension reduction. The generalized Foley–Sammon transform and the trace ratio model are very important in LDA and have received increasing interest. An efficient iterative method has been proposed for the resulting trace ratio optimization problem, which, under a mild assumption, is proved to enjoy both the local quadratic convergence and the global convergence to the global optimal solution (Zhang, L.-H., Liao, L.-Z., Ng, M.K.: SIAM J. Matrix Anal. Appl. 31:1584, 2010). The present paper further investigates the convergence behavior of this iterative method under no assumption. In particular, we prove that the iteration converges superlinearly when the mild assumption is removed. All possible limit points are characterized as a special subset of the global optimal solutions. An illustrative numerical example is also presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Berlin (2006)

    MATH  Google Scholar 

  2. Duchene, L., Leclerq, S.: An optimal transformation for discriminant and principal component analysis. IEEE Trans. Pattern Anal. Mach. Intell. 10, 978–983 (1988)

    Article  MATH  Google Scholar 

  3. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification. Wiley-Interscience, New York (2001)

    MATH  Google Scholar 

  4. Fisher, R.A.: The use of multiple measurements in taxonomic problems. Annu. Eugen. 7, 179–188 (1936)

    Article  Google Scholar 

  5. Foley, D., Sammon, J.: An optimal set of discriminant vectors. IEEE Trans. Comput. 24, 281–289 (1975)

    Article  MATH  Google Scholar 

  6. Fukunaga, K.: Introduction to Statistical Pattern Classification. Academic Press, San Diego (1990)

    Google Scholar 

  7. Hastie, T., Tibshirani, R., Friedman, J.H.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, Berlin (2001)

    MATH  Google Scholar 

  8. Martinez, A.M., Kak, A.C.: PCA versus LDA. IEEE Trans. Pattern Anal. Mach. Intell. 23, 228–223 (2001)

    Article  Google Scholar 

  9. Martinez, A.M., Zhu, M.: Where are linear feature extraction methods applicable? IEEE Trans. Pattern Anal. Mach. Intell. 27, 1934–1944 (2005)

    Article  Google Scholar 

  10. McLanchlan, G.J.: Discriminant Analysis and Statistical Pattern Recognition. Wiley, New York (2004)

    Google Scholar 

  11. Theodoridis, S., Koutroumbas, K.: Pattern Recognition. Academic Press, New York (1999)

    Google Scholar 

  12. Guo, Y.-F., Li, S.-J., Yang, J.-Y., Shu, T.-T., Wu, L.-D.: A generalized Foley–Sammon transform based on generalized Fisher discriminant criterion and its application to face recognition. Pattern Recognit. Lett. 24, 147–158 (2003)

    Article  MATH  Google Scholar 

  13. Wang, H., Yan, S.-C., Xu, D., Tang, X., Huang, T.: Trace ratio vs. ratio trace for dimensionality reduction. In: Proc. International Conf. on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

    Google Scholar 

  14. Zhang, L.-H., Liao, L.-Z., Ng, M.K.: Fast algorithms for the generalized Foley–Sammon discriminant analysis. SIAM J. Matrix Anal. Appl. 31, 1584–1605 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  15. Zhang, L.-H.: Uncorrected trace ratio LDA for undersampled problems. Pattern Recognit. Lett. 32, 476–484 (2011)

    Article  Google Scholar 

  16. Ngo, T.T., Bellalij, M., Saad, Y.: The trace ratio optimization problem for dimensionality reduction. SIAM J. Matrix Anal. Appl. 31, 2950–2971 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  17. Nie, F., Xiang, S., Jia, Y., Zhang, C.: Semi-supervised orthogonal discriminant analysis via label propagation. Pattern Recognit. 42, 2615–2627 (2009)

    Article  MATH  Google Scholar 

  18. Howland, P., Jeon, M., Park, H.: Structure preserving dimension reduction for clustered text data based on the generalized singular value decomposition. SIAM J. Matrix Anal. Appl. 25, 165–179 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  19. Howland, P., Park, H.: Generalizing discriminant analysis using the generalized singular value decomposition. IEEE Trans. Pattern Anal. Mach. Intell. 26, 995–1006 (2004)

    Article  Google Scholar 

  20. Howland, P., Park, H.: Equivalence of several two-stage methods for linear discriminant analysis. In: Proceedings of the Fourth SIAM International Conference on Data Mining, Kissimmee, FL, pp. 69–77 (2004)

    Google Scholar 

  21. Ng, M.K., Liao, L.-Z., Zhang, L.-H.: On sparse linear discriminant analysis for high-dimensional data. Numer. Linear Algebra Appl. 18, 223–235 (2010)

    Article  MathSciNet  Google Scholar 

  22. Park, H., Drake, B.L., Lee, S., Park, C.H.: Fast linear discriminant analysis using QR decomposition and regularization. Technical Report GT-CSE-07-21 (2007)

  23. Ye, J.-P.: Characterization of a family of algorithms for generalized discriminant analysis on undersampled problems. J. Mach. Learn. Res. 6, 483–502 (2005)

    MathSciNet  MATH  Google Scholar 

  24. Ye, J.-P., Xiong, T.: Computational and theoretical analysis of null space and orthogonal linear discriminant analysis. J. Mach. Learn. Res. 7, 1183–1204 (2006)

    MathSciNet  MATH  Google Scholar 

  25. Ye, J.-P., Janardan, R., Park, C., Park, H.: An optimization criterion for generalized discriminant analysis on undersampled problems. IEEE Trans. Pattern Anal. Mach. Intell. 26, 982–994 (2004)

    Article  Google Scholar 

  26. Ye, J.-P., Xiong, T., Li, Q., Janardan, R., Bi, J.-B., Cherkassky, V., Kambhamettu, C.: Efficient model selection for regularized linear discriminant analysis. In: The Fifteenth ACM International Conference on Information and Knowledge Management (CIKM), pp. 532–539 (2006)

    Google Scholar 

  27. Yan, S., Xu, D., Zhang, B., Zhang, H.: Graph embedding: A general framework for dimensionality reduction. In: Proceedings of Conference on Computer Vision and Pattern Recognition, pp. 830–837 (2005)

    Google Scholar 

  28. Krzanowski, W.J., Jonathan, P., McCarthy, W.V., Thomas, M.R.: Discriminant analysis with singular covariance matrices: methods and applications to spectroscopic data. Appl. Stat. 44, 101–115 (1995)

    Article  MATH  Google Scholar 

  29. Friedman, J.: Regularized discriminant analysis. J. Am. Stat. Assoc. 84, 165–175 (1989)

    Article  Google Scholar 

  30. Stewart, G.W.: Matrix Algorithms: Vol. II, Eigensystems. SIAM, Philadelphia (2001)

    Book  Google Scholar 

  31. Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. Johns Hopkins University Press, Baltimore (1996)

    MATH  Google Scholar 

  32. Parlett, B.N.: The Rayleigh quotient iteration and some generalizations for nonnormal matrices. Math. Comput. 28, 679–693 (1974)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Applied Mathematics, Shanghai University of Finance and Economics, 777 Guoding Road, Shanghai, 200433, People’s Republic of China

    Lei-Hong Zhang

  2. Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Kowloon, Hong Kong, People’s Republic of China

    Li-Zhi Liao & Michael K. Ng

Authors
  1. Lei-Hong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Li-Zhi Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael K. Ng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei-Hong Zhang.

Additional information

The authors would like to thank two anonymous referees and the editor for their helpful comments and suggestions on the earlier version of this paper. Research of the second author was supported in part by FRG grants from Hong Kong Baptist University and the Research Grant Council of Hong Kong. Research of the third author was supported in part by RGC grants 7035/04P, 7035/05P and HKBU FRGs.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, LH., Liao, LZ. & Ng, M.K. Superlinear Convergence of a General Algorithm for the Generalized Foley–Sammon Discriminant Analysis. J Optim Theory Appl 157, 853–865 (2013). https://doi.org/10.1007/s10957-011-9832-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-011-9832-4

Keywords

Search

Navigation