Skip to main content
SpringerLink
Log in

A self-consistent-field iteration for MAXBET with an application to multi-view feature extraction

  • Published:
Advances in Computational Mathematics Aims and scope Submit manuscript

Abstract

As an extension of the traditional principal component analysis, the multi-view canonical correlation analysis (MCCA) aims at reducing m high dimensional random variables \(\boldsymbol {s}_{i}\in \mathbb {R}^{n_{i}}~(i=1,2,\ldots ,m)\) by proper projection matrices \(X_{i}\in \mathbb {R}^{n_{i}\times \ell }\) so that the m reduced ones \(\boldsymbol {y}_{i}=X_{i}^{\mathrm {T}}\boldsymbol {s}_{i}\in \mathbb {R}^{\ell }\) have the “maximal correlation.” Various measures of the correlation for yi (i = 1,2,…,m) in MCCA have been proposed. One of the earliest criteria is the sum of all traces of pair-wise correlation matrices between yi and yj subject to the orthogonality constraints on Xi, i = 1,2,…,m. The resulting problem is to maximize a homogeneous quadratic function over the product of Stiefel manifolds and is referred to as the MAXBET problem. In this paper, the problem is first reformulated as a coupled nonlinear eigenvalue problem with eigenvector dependency (NEPv) and then solved by a novel self-consistent-field (SCF) iteration. Global and local convergences of the SCF iteration are studied and proven computational techniques in the standard eigenvalue problem are incorporated to yield more practical implementations. Besides the preliminary numerical evaluations on various types of synthetic problems, the efficiency of the SCF iteration is also demonstrated in an application to multi-view feature extraction for unsupervised learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Absil, P.-A., Baker, C. G., Gallivan, K. A.: Trust-region methods on Riemannian manifolds. Found. Comput. Math. 7(3), 303–330 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  2. Absil, P. -A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton (2008)

  3. Bai, Z., Demmel, J. W., Dongarra, J., Ruhe, A., Vorst, H., Van Der Vorst, H.: Templates for the Solution of Algebraic Eigenvalue Problems: A Practical Guide. SIAM, Philadelphia (2000)

  4. Bai, Z., Lu, D., Vandereycken, B.: Robust Rayleigh quotient minimization and nonlinear eigenvalue problems. SIAM J. Sci. Comput. 40, A3495–A3522 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bendel, R. B., Mickey, M. R.: Population correlation matrices for sampling experiments. Comm Statist. Simul. Comput. 7(2), 163–182 (1978)

    Article  MATH  Google Scholar 

  6. Ten Berge, J. M. F.: Generalized approaches to the MAXBET problem and the MAXDIFF problem, with applications to canonical correlations. Psychometrika 53(4), 487–494 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  7. Ten Berge, J. M. F., Knol, D. L.: Orthogonal rotations to maximal agreement for two or more matrices of different column orders. Psychometrika 49 (4), 49–55 (1984)

    Article  Google Scholar 

  8. Cai, Y., Zhang, L. -H., Bai, Z., Li, R. -C.: On an eigenvector-dependent nonlinear eigenvalue problem. SIAM J. Matrix Anal. Appl. 39(3), 1360–1382 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  9. Chu, M. T., Watterson, J. L.: On a multivariate eigenvalue problem, Part I: Algebraic theory and a power method. SIAM J. Sci. Comput. 14(4), 1089–1106 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  10. Cunningham, J. P., Ghahramani, Z.: Linear dimensionality reduction: Survey, insights, and generalizations. J. Mach. Lear. Res. 16, 2859–2900 (2015)

    MathSciNet  MATH  Google Scholar 

  11. Davies, P. I., Higham, N. J.: Numerically stable generation of correlation matrices and their factors. BIT 40, 640–651 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  12. Davis, C., Kahan, W.: The rotation of eigenvectors by a perturbation. III. SIAM J. Numer. Anal. 7, 1–46 (1970)

    Article  MathSciNet  MATH  Google Scholar 

  13. Van de Geer, J. P.: Linear relations among k sets of variables. Psychometrika 49, 70–94 (1984)

  14. Goemans, M. X., Williamson, D. P.: Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. J. ACM 42, 1115–1145 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  15. Golub, G. H., Van Loan, C. F.: Matrix Computations, 4th edn. Johns Hopkins University Press, Baltimore (2013)

  16. Guan, Y., Chu, M. T., Chu, D.: SVD-based algorithms for the best rank-1 approximation of a symmetric tensor. SIAM J. Matrix Anal. Appl. 39, 1095–1115 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  17. Hanafi, M., Ten Berge, J. M. F.: Global optimality of the successive Maxbet algorithm. Psychometrika 68, 97–103 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  18. Hardoon, D. R., Szedmak, S., Shawe-Taylor, J.: Canonical correlation analysis: An overview with application to learning methods. Neural Comput. 16, 2639–2664 (2004)

    Article  MATH  Google Scholar 

  19. Horst, P.: Relations among m sets of measures. Psychometrika 26, 129–149 (1961)

    Article  MathSciNet  MATH  Google Scholar 

  20. Hotelling, H.: Relations between two sets of variates. Biometrika 28, 321–377 (1936)

    Article  MATH  Google Scholar 

  21. Kettenring, J. R.: Canonical analysis of several sets of variables. Biometrika 58(1), 433–451 (1971)

    Article  MathSciNet  MATH  Google Scholar 

  22. Knyazev, A. V.: Toward the optimal preconditioned eigensolver: Locally optimal block preconditioned conjugate gradient method. SIAM J. Sci. Comput. 23(2), 517–541 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  23. Knyazev, A. V., Argentati, M. E.: Rayleigh-Ritz majorization error bounds with applications to FEM. SIAM J Matrix Anal Appl 31(3), 1521–1537 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  24. Kovač-Striko, J., Veselić, K.: Some remarks on the spectra of Hermitian matrices. Linear Algebra Appl. 145, 221–229 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  25. Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), pp. 2169–2178 (2006)

  26. Li, F. F., Fergus, R., Perona, P.: Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Comput. Vis. Image Underst. 106, 59–70 (2007)

    Article  Google Scholar 

  27. Li, R. -C.: New perturbation bounds for the unitary polar factor. SIAM J. Matrix Anal. Appl. 16, 327–332 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  28. Li, R. -C.: Accuracy of computed eigenvectors via optimizing a Rayleigh quotient. BIT 44(3), 585–593 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  29. Li, R.-C.: Rayleigh quotient based optimization methods for eigenvalue problems. In: Bai, Z., Gao, W., Su, Y. (eds.) Matrix Functions and Matrix Equations, volume 19 of Series in Contemporary Applied Mathematics, pp 76–108. World Scientific, Singapore (2015)

  30. Li, Z., Nie, F., Chang, X., Yang, Y.: Beyond trace Weighted Harmonic mean of trace ratios for multiclass discriminant analysis. IEEE Trans. Knowl. Data Engrg. 29(10), 2100–2110 (2017)

    Article  Google Scholar 

  31. Lingoes, J. C., Borg, I.: A direct approach to individual differences scaling using increasingly complex transformations. Psychometrika 43, 491–519 (1978)

    Article  MathSciNet  MATH  Google Scholar 

  32. Liu, X. -G., Wang, X. -F., Wang, W.-G.: Maximization of matrix trace function of product S,tiefel manifolds. SIAM J Matrix Anal. Appl. 36 (4), 1489–1506 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  33. Martin, R. M.: Electronic structure: Basic theory and practical methods. Cambridge University Press, Cambridge (2004)

  34. Moré, J. J., Sorensen, D. C.: Computing a trust region step. SIAM J. Sci. Statist. Comput. 4(3), 553–572 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  35. Nielsen, A. A.: Multiset canonical correlations analysis and multispectral, truly multitemporal remote sensing data. IEEE Trans. Image Process. 11(3), 293–305 (2002)

    Article  Google Scholar 

  36. Nocedal, J.: Numerical Optimization, 2nd edn. Springer, New York (2006)

  37. Parlett, B. N.: The symmetric eigenvalue problem. SIAM. Philadelphia (1998)

  38. Pourahmadi, M., Wang, X.: Distribution of random correlation matrices: Hyperspherical parameterization of the Cholesky factor. Statist. Probab. Lett. 106(3), 5–12 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  39. Saad, Y., Chelikowsky, J. R., Shontz, S. M.: Numerical methods for electronic structure calculations of materials. SIAM Rev. 52(1), 3–54 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  40. Seber, G. A. F.: A Matrix Handbook for Statisticians. Wiley, New Jersey (2007)

    Book  Google Scholar 

  41. Stewart, G. W.: Matrix Algorithms, Vol. II: Eigensystems. SIAM, Philadelphia (2001)

  42. Veleva, E.: Generation of correlation matrices. AIP Conf. Proc. 1895(120008), 1–7 (2017)

    Google Scholar 

  43. von Neumann, J.: Some matrix-inequalities and metrization of matrix- space. Tomck. Univ Some Rev. 1, 286–300 (1937)

    MATH  Google Scholar 

  44. Wang, L., Li, R.-C.: A scalable algorithm for large-scale unsupervised multi-view partial least squares. IEEE Trans. Big Data. https://doi.org/10.1109/TBDATA.2020.3014937, to appear (2020)

  45. Wang, L., Zhang, L. -H., Bai, Z., Li, R. -C.: Orthogonal canonical correlation analysis and applications. Optim. Methods Softw. 35(4), 787–807 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  46. Wang, Z., Ruan, Q., An, G.: Projection-optimal local Fisher discriminant analysis for feature extraction. Neural Comput. Appl. 26, 589–601 (2015)

    Article  Google Scholar 

  47. Yang, W. H., Zhang, L. -H., Song, R. Y.: Optimality conditions of the nonlinear programming on Riemannian, manifolds. Pac. J. Optim. 10, 415–434 (2014)

    MathSciNet  MATH  Google Scholar 

  48. Yang, X., Liu, W., Liu, W., Tao, D.: A survey on canonical correlation analysis. IEEE Trans. Knowl. Data Engrg. 33(6), 2349–2368 (2020)

    Article  Google Scholar 

  49. Yu, Y., Zhang, L. -H., Zhang, S.: Simultaneous clustering of multiview biomedical data using manifold optimization. Bioinformatics 35(20), 4029–4037 (2019)

    Article  Google Scholar 

  50. Zhang, L. -H.: Riemannian Newton method for the multivariate eigenvalue problem. SIAM J. Matrix Anal. Appl. 31(5), 2972–2996 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  51. Zhang, L.-H.: Riemannian trust-region method for the maximal correlation problem. Numer. Funct. Anal. Optim. 33(3), 338–362 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  52. Zhang, L. -H., Li, R.-C.: Maximization of the sum of the trace ratio on the Stiefel, manifold, I Theory. Sci. China Math. 57(12), 2495–2508 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  53. Zhang, L. -H., Li, R.-C.: Maximization of the sum of the trace ratio on the Stiefel manifold, II Computation. Sci. China Math. 58, 1549–1566 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  54. Zhang, L. -H., Ma, X., Shen, C.: A structure-exploiting nested Lanczos-type iteration for the multi-view canonical correlation analysis. SIAM J. Sci. Comput. 43(4), A2685–A2713 (2021)

    Article  MATH  Google Scholar 

  55. Zhang, L. -H., Li, Wang, Bai, Z., Li, R.-C.: A self-consistent-field iteration for orthogonal canonical correlation analysis. IEEE Trans. Pattern Anal. Mach Intell. 44(2), 890–904 (2022)

    Article  Google Scholar 

  56. Zhang, L. -H., Yang, W. H., Shen, C., Ying, J.: An eigenvalue-based method for the unbalanced Procrustes problem. SIAM J Matrix Anal. Appl. 41(3), 957–983 (2020)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors are grateful to the anonymous referees for their useful comments and suggestions to improve the presentation of this paper.

Funding

The work of the third author was supported partially by NSF DMS-2009689 and NIH R01AG075582; the work of the fourth author was supported partially by the National Natural Science Foundation of China NSFC-12071332; and the work of the fifth author was supported partially by NSF DMS-1719620 and DMS-2009689.

Author information

Authors and Affiliations

  1. School of Mathematics, Shanghai University of Finance and Economics, 777 Guoding Road, Shanghai, 200433, China

    Xijun Ma & Lei-Hong Zhang

  2. College of Science, University of Shanghai for Science and Technology, Shanghai, 200093, China

    Chungen Shen

  3. Department of Mathematics, University of Texas at Arlington, Arlington, 76019-0408, TX, USA

    Li Wang & Ren-Cang Li

  4. School of Mathematical Sciences, Soochow University, Suzhou, 215006, Jiangsu, China

    Lei-Hong Zhang

  5. Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Kowloon, Hong Kong, Hong Kong, China

    Ren-Cang Li

Authors
  1. Xijun Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chungen Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Li Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lei-Hong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ren-Cang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei-Hong Zhang.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Communicated by: Raymond H. Chan

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ma, X., Shen, C., Wang, L. et al. A self-consistent-field iteration for MAXBET with an application to multi-view feature extraction. Adv Comput Math 48, 13 (2022). https://doi.org/10.1007/s10444-022-09929-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10444-022-09929-3

Keywords

Mathematics Subject Classification (2010)

Search

Navigation