Skip to main content
SpringerLink
Log in

Operator-Valued Formulas for Riemannian Gradient and Hessian and Families of Tractable Metrics in Riemannian Optimization

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

We provide explicit formulas for the Levi-Civita connection and Riemannian Hessian for a Riemannian manifold that is a quotient of a manifold embedded in an inner product space with a non-constant metric function. Together with a classical formula for projection, this allows us to evaluate Riemannian gradient and Hessian for several families of metrics on classical manifolds, including a family of metrics on Stiefel manifolds connecting both the constant and canonical ambient metrics with closed-form geodesics. Using these formulas, we derive Riemannian optimization frameworks on quotients of Stiefel manifolds, including flag manifolds, and a new family of complete quotient metrics on the manifold of positive-semidefinite matrices of fixed rank, considered as a quotient of a product of Stiefel and positive-definite matrix manifold with affine-invariant metrics. The method is procedural, and in many instances, the Riemannian gradient and Hessian formulas could be derived by symbolic calculus. The method extends the list of potential metrics that could be used in manifold optimization and machine learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Absil, P.A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton (2007)

    MATH  Google Scholar 

  2. Absil, P.A., Mahony, R., Trumpf, J.: An extrinsic look at the Riemannian Hessian. In: F. Nielsen, F. Barbaresco (eds.) Geometric Science of Information–First International Conference, GSI 2013, Paris, France, Proceedings, Lecture Notes in Computer Science, Vol. 8085, pp. 361–368. Springer, Berlin (2013)

  3. Adler, R.L., Dedieu, J., Margulies, J.Y., Martens, M., Shub, M.: Newton’s method on Riemannian manifolds and a geometric model for the human spine. IMA J. Numer. Anal. 22(3), 359–390 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  4. Aitken, A.C.: On least squares and linear combination of observations. Proc. R. Soc. Edinb. 55, 42–48 (1936)

    Article  MATH  Google Scholar 

  5. Bartels, R.H., Stewart, G.W.: Solution of the matrix equation ax + xb = c [f4]. Commun. ACM 15(9), 820–826 (1972)

    Article  MATH  Google Scholar 

  6. Bhatia, R., Rosenthal, P.: How and why to solve the operator equation AX - XB = Y. Bull. Lond. Math. Soc. 29, 1–21 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  7. Bonnabel, S., Sepulchre, R.: Riemannian metric and geometric mean for positive semidefinite matrices of fixed rank. SIAM J. Matrix Anal. Appl. 31(3), 1055–1070 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  8. Boumal, N., Mishra, B., Absil, P.A., Sepulchre, R.: Manopt, a Matlab toolbox for optimization on manifolds. J. Mach. Learn. Res. 15, 1455–1459 (2014)

    MATH  Google Scholar 

  9. Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20(2), 303–353 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  10. Gabay, D.: Minimizing a differentiable function over a differential manifold. J. Optim. Theory Appl. 37(2), 177–219 (1982)

    Article  MathSciNet  MATH  Google Scholar 

  11. Hüper, K., Markina, I., Silva Leite, F.: A Lagrangian approach to extremal curves on Stiefel manifolds. J. Geom. Mech. 13(1), 55–72 (2021). https://doi.org/10.3934/jgm.2020031

    Article  MathSciNet  MATH  Google Scholar 

  12. Jeuris, B., Vandebril, R., Vandereycken, B.: A survey and comparison of contemporary algorithms for computing the matrix geometric mean. Electron. Trans. Numer. Anal. 39, 379–402 (2012)

    MathSciNet  MATH  Google Scholar 

  13. Journée, M., Bach, F., Absil, P.A., Sepulchre, R.: Low-rank optimization on the cone of positive semidefinite matrices. SIAM J. Optim. 20(5), 2327–2351 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  14. Kasai, H., Mishra, B.: Low-rank tensor completion: a Riemannian manifold preconditioning approach. In: M.F. Balcan, K.Q. Weinberger (eds.) Proceedings of The 33rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 48, pp. 1012–1021. New York (2016). http://proceedings.mlr.press/v48/kasai16.html

  15. Manton, J.H.: Optimization algorithms exploiting unitary constraints. IEEE Trans. Signal Process. 50(3), 635–650 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  16. Miolane, N., Brigant, A.L., Mathe, J., Hou, B., Guigui, N., Thanwerdas, Y., Heyder, S., Peltre, O., Koep, N., Zaatiti, H., Hajri, H., Cabanes, Y., Gerald, T., Chauchat, P., Shewmake, C., Kainz, B., Donnat, C., Holmes, S., Pennec, X.: Geomstats: A Python package for Riemannian geometry in machine learning (2020). arXiv:2004.04667

  17. Mishra, B., Sepulchre, R.: Riemannian preconditioning. SIAM J. Optim. 26(1), 635–660 (2016). https://doi.org/10.1137/140970860

    Article  MathSciNet  MATH  Google Scholar 

  18. Nguyen, D.: Project ManNullRange. https://github.com/dnguyend/ManNullRange (2020)

  19. Nguyen, D.: Project SimpleFlag. https://github.com/dnguyend/SimpleFlag (2021)

  20. Nguyen, D.: Closed-form geodesics and optimization for Riemannian logarithms of Stiefel and flag Manifolds. J. Optim. Theory Appl. 194, 142–166 (2022). https://doi.org/10.1007/s10957-022-02012-3

    Article  MathSciNet  MATH  Google Scholar 

  21. Nishimori, Y., Akaho, S., Plumbley, M.D.: Riemannian optimization method on the flag manifold for independent subspace analysis. In: J. Rosca, D. Erdogmus, J.C. Príncipe, S. Haykin (eds.) ICA 2006: Independent Component Analysis and Blind Signal Separation, Proceedings, Lecture Notes in Computer Science, vol. 3889, pp. 295–302. Springer (2006)

  22. O’Neill, B.: Semi-Riemannian Geometry with Applications to Relativity, Pure and Applied Mathematics, vol. 103. Academic Press Inc, New York, NY (1983)

    MATH  Google Scholar 

  23. Pennec, X., Fillard, P., Ayache, N.: A Riemannian framework for tensor computing. Int. J. Comput. Vis. 66, 41–66 (2006)

    Article  MATH  Google Scholar 

  24. Smith, S.T.: Covariance, subspace, and intrinsic Cramér-Rao bounds. IEEE Trans. Signal Process. 53(5), 1610–1630 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  25. Sra, S., Hosseini, R.: Conic geometric optimization on the manifold of positive definite matrices. SIAM J. Optim. 25(1), 713–739 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  26. Townsend, J., Koep, N., Weichwald, S.: Pymanopt:a Python toolbox for optimization on manifolds using automatic differentiation. J. Mach. Learn. Res. 17(137), 1–5 (2016)

    MathSciNet  MATH  Google Scholar 

  27. Vandereycken, B., Absil, P.A., Vandewalle, S.: A Riemannian geometry with complete geodesics for the set of positive semidefinite matrices of fixed rank. IMA J. Numer. Anal. 33(2), 481–514 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  28. Ye, K., Wong, K.S.W., Lim, L.H.: Optimization on flag manifolds. Math. Program. 194, 621–660 (2022). https://doi.org/10.1007/s10107-021-01640-3

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

We are thankful to the editor and reviewers for their careful reading and helpful comments. Their work is essential to the publication of this paper. It is our sole responsibility for any remaining mistake. We would like to thank Professor Overton for his interest and very kind advice, our friend John Tillinghast and our family for their support in this project.

Author information

Authors and Affiliations

  1. Darien, CT, USA

    Du Nguyen

Authors
  1. Du Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Du Nguyen.

Ethics declarations

Conflict of interest

The author declares that he has no conflict of interest. No funds, grants, or other support were received.

Additional information

Communicated by Alexandru Kristály.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nguyen, D. Operator-Valued Formulas for Riemannian Gradient and Hessian and Families of Tractable Metrics in Riemannian Optimization. J Optim Theory Appl 198, 135–164 (2023). https://doi.org/10.1007/s10957-023-02242-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-023-02242-z

Keywords

Mathematics Subject Classification

Search

Navigation