Skip to main content
SpringerLink
Log in

On optimizing the sum of the Rayleigh quotient and the generalized Rayleigh quotient on the unit sphere

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

Given symmetric matrices B,D∈ℝn×n and a symmetric positive definite matrix W∈ℝn×n, maximizing the sum of the Rayleigh quotient x ⊤ D x and the generalized Rayleigh quotient \(\frac{\mathbf{x}^{\top}B \mathbf{x}}{\vphantom{\mathrm{I}^{\mathrm{I}}}\mathbf{x}^{\top}W\mathbf{x}}\) on the unit sphere not only is of mathematical interest in its own right, but also finds applications in practice. In this paper, we first present a real world application arising from the sparse Fisher discriminant analysis. To tackle this problem, our first effort is to characterize the local and global maxima by investigating the optimality conditions. Our results reveal that finding the global solution is closely related with a special extreme nonlinear eigenvalue problem, and in the special case D=μW (μ>0), the set of the global solutions is essentially an eigenspace corresponding to the largest eigenvalue of a specially-defined matrix. The characterization of the global solution not only sheds some lights on the maximization problem, but motives a starting point strategy to obtain the global maximizer for any monotonically convergent iteration. Our second part then realizes the Riemannian trust-region method of Absil, Baker and Gallivan (Found. Comput. Math. 7:303–330, 2007) into a practical algorithm to solve this problem, which enjoys the nice convergence properties: global convergence and local superlinear convergence. Preliminary numerical tests are carried out and empirical evaluation of its performance is reported.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. We will demonstrate how to compute these points in Sect. 3.

  2. The matrix S w can be assumed to be positive definite as the singularity in S w (i.e., the undersampled problem) can be handled by, for example, the regularization [17].

  3. In Matlab environment, the generic Riemannian trust-region package for the optimization of functions defined on Riemannian manifolds is available at: http://www.math.fsu.edu/~cbaker/GenRTR/.

  4. Because the general RTR method proposed in [4, 5] is stated to minimize a cost function on a general manifold, we will solve , instead of .

  5. http://math.nist.gov/MatrixMarket/.

References

  1. Abraham, R., Marsden, J.E., Ratiu, T.: Manifolds, Tensor Analysis, and Applications, 2nd edn. Applied Mathematical Sciences, vol. 75. Springer, New York (1988)

    Book  MATH  Google Scholar 

  2. Absil, P.-A., Gallivan, K.A.: Accelerated line-search and trust-region methods. SIAM J. Numer. Anal. 47, 997–1018 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  3. Absil, P.-A., Baker, C.G., Gallivan, K.A.: A truncated-CG style method for symmetric generalized eigenvalue problems. J. Comput. Appl. Math. 189, 274–285 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  4. Absil, P.-A., Baker, C.G., Gallivan, K.A.: Trust-region methods on Riemannian manifolds. Found. Comput. Math. 7, 303–330 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  5. Absil, P.-A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton (2008)

    MATH  Google Scholar 

  6. Adler, R.L., Dedieu, J.-P., Margulies, J.Y., Martens, M., Shub, M.: Newton’s method on Riemannian manifolds and a geometric model for the human spine. IMA J. Numer. Anal. 22, 359–390 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  7. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Berlin (2006)

    MATH  Google Scholar 

  8. Bonnans, J.F., Shapiro, A.: Perturbation Analysis of Optimization Problems. Springer, New York (2000)

    MATH  Google Scholar 

  9. Chu, M.T., Driessel, K.R.: The projected gradient method for least squares matrix approximations with spectral constraints. SIAM J. Numer. Anal. 27, 1050–1060 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  10. Conn, A.R., Gould, N.I.M., Toint, P.L.: Trust-Region Methods. SIAM, Philadelphia (2000)

    Book  MATH  Google Scholar 

  11. Duchene, L., Leclerq, S.: An optimal transformation for discriminant and principal component analysis. IEEE Trans. Pattern Anal. Mach. Intell. 10, 978–983 (1988)

    Article  MATH  Google Scholar 

  12. Dundar, M.M., Fung, G., Bi, J., Sandilya, S., Rao, B.: Sparse fisher discriminant analysis for computer aided detection. In: Proceedings of SIAM International Conference on Data Mining (2005)

    Google Scholar 

  13. Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20, 303–353 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  14. Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96, 1348–1360 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  15. Fisher, R.A.: The use of multiple measurements in taxonomic problems. Annu. Eugen. 7, 179–188 (1936)

    Google Scholar 

  16. Foley, D., Sammon, J.: An optimal set of discriminant vectors. IEEE Trans. Comput. 24, 281–289 (1975)

    Article  MATH  Google Scholar 

  17. Friedman, J.: Regularized discriminant analysis. J. Am. Stat. Assoc. 84, 165–175 (1989)

    Article  Google Scholar 

  18. Fukunaga, K.: Introduction to Statistical Pattern Classification. Academic Press, San Diego (1990)

    Google Scholar 

  19. Fung, E., Ng, M.: On sparse fisher discriminant method for microarray data analysis. Bioinformation 2, 230–234 (2007)

    Article  Google Scholar 

  20. Gao, X.B., Golub, G.H., Liao, L.-Z.: Continuous methods for symmetric generalized eigenvalue problems. Linear Algebra Appl. 428, 676–696 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  21. Golub, G.H., Liao, L.-Z.: Continuous methods for extreme and interior eigenvalue problems. Linear Algebra Appl. 415, 31–51 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  22. Golub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn. Johns Hopkins University Press, Baltimore (1996)

    MATH  Google Scholar 

  23. Helmke, U., Moore, J.B.: Optimization and Dynamical Systems. Springer, London (1994)

    Book  Google Scholar 

  24. Howland, P., Jeon, M., Park, H.: Structure preserving dimension reduction for clustered text data based on the generalized singular value decomposition. SIAM J. Matrix Anal. Appl. 25, 165–179 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  25. Hunter, D.R., Li, R.: Variable selection using MM algorithms. Ann. Stat. 33, 1617–1642 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  26. Kelley, C.T.: Iterative Methods for Linear and Nonlinear Equations. SIAM, Philadelphia (1995)

    Book  MATH  Google Scholar 

  27. Lehoucq, R.B., Sorensen, D.C.: Deflation techniques for an implicitly re-started Arnoldi iteration. SIAM J. Matrix Anal. Appl. 17, 789–821 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  28. Lehoucq, R.B., Sorensen, D.C., Yang, C.: ARPACK Users’ Guide: Solution of Large-Scale Eigenvalue Problems with Implicitly Restarted Arnoldi Methods. SIAM, Philadelphia (1998)

    Book  Google Scholar 

  29. Ng, M.K., Liao, L.-Z., Zhang, L.-H.: On sparse linear discriminant analysis for high-dimensional data. Numer. Linear Algebra Appl. 18, 223–235 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  30. Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, New York (2006)

    MATH  Google Scholar 

  31. Parlett, B.N.: The Rayleigh quotient iteration and some generalizations for nonnormal matrices. Math. Comput. 28, 679–693 (1974)

    Article  MathSciNet  MATH  Google Scholar 

  32. Parlett, B.N.: The Symmetric Eigenvalue Problem. Classics Appl. Math., vol. 20. SIAM, Philadelphia (1998)

    Book  MATH  Google Scholar 

  33. Primolevo, G., Simeone, O., Spagnolini, U.: Towards a joint optimization of scheduling and beamforming for MIMO downlink. In: IEEE Ninth International Symposium on Spread Spectrum Techniques and Applications, pp. 493–497 (2006)

    Chapter  Google Scholar 

  34. Saad, Y.: Numerical Methods for Large Eigenvalue Problems. Algorithms and Architectures for Advanced Scientific Computing. Manchester University Press, Manchester (1992)

    MATH  Google Scholar 

  35. Steihaug, T.: The conjugate gradient method and trust regions in large scale optimization. SIAM J. Numer. Anal. 20, 626–637 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  36. Toint, P.L.: Towards an efficient sparsity exploiting newton method for minimization. In: Duff, I.S. (ed.) Sparse Matrices and Their Uses, pp. 57–88. Academic Press, London (1981)

    Google Scholar 

  37. Wu, M.C., Zhang, L.S., Wang, Z.X., Christiani, D.C., Lin, X.H.: Sparse linear discriminant analysis for simultaneous testing for the significance of a gene set/pathway and gene selection. Bioinformatics 25, 1145–1151 (2009)

    Article  Google Scholar 

  38. Ye, J.-P., Janardan, R., Park, C., Park, H.: An optimization criterion for generalized discriminant analysis on undersampled problems. IEEE Trans. Pattern Anal. Mach. Intell. 26, 982–994 (2004)

    Article  Google Scholar 

  39. Zhang, L.-H.: Uncorrected trace ratio LDA for undersampled problems. Pattern Recognit. Lett. 32, 476–484 (2011)

    Article  Google Scholar 

  40. Zhang, L.-H., Liao, L.-Z., Ng, M.K.: Fast algorithms for the generalized Foley-Sammon discriminant analysis. SIAM J. Matrix Anal. Appl. 31, 1584–1605 (2010)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The author would like to thank the Editor and two anonymous referees for careful reading, helpful comments and suggestions that have improved the presentation of the paper.

Author information

Authors and Affiliations

  1. Department of Applied Mathematics, Shanghai University of Finance and Economics, 777 Guoding Road, Shanghai, 200433, People’s Republic of China

    Lei-Hong Zhang

Authors
  1. Lei-Hong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei-Hong Zhang.

Additional information

This work was supported by the National Natural Science Foundation of China NSFC-11101257.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, LH. On optimizing the sum of the Rayleigh quotient and the generalized Rayleigh quotient on the unit sphere. Comput Optim Appl 54, 111–139 (2013). https://doi.org/10.1007/s10589-012-9479-6

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-012-9479-6

Keywords

Search

Navigation