Skip to main content
Log in

Convergence rate analysis for the higher order power method in best rank one approximations of tensors

  • Published:
Numerische Mathematik Aims and scope Submit manuscript

Abstract

A popular and classical method for finding the best rank one approximation of a real tensor is the higher order power method (HOPM). It is known in the literature that the iterative sequence generated by HOPM converges globally, while the convergence rate can be superlinear, linear or sublinear. In this paper, we examine the convergence rate of HOPM in solving the best rank one approximation problem of real tensors. We first show that the iterative sequence of HOPM always converges globally and provide an explicit eventual sublinear convergence rate. The sublinear convergence rate estimate is in terms of the dimension and the order of the underlying tensor space. Then, we examine the concept of nondegenerate singular vector tuples and show that, if the sequence of HOPM converges to a nondegenerate singular vector tuple, then the global convergence rate is R-linear. We show that, for almost all tensors (in the sense of Lebesgue measure), all the singular vector tuples are nondegenerate, and so, the HOPM “typically” exhibits global R-linear convergence rate. Moreover, without any regularity assumption, we establish that the sequence generated by HOPM always converges globally and R-linearly for orthogonally decomposable tensors with order at least 3. We achieved this by showing that each nonzero singular vector tuple of an orthogonally decomposable tensor with order at least 3 is nondegenerate.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Here, a property holds for almost all tensors means that the set of tensors for which the property does not hold is a set of Lebesgue measure zero.

  2. In [34], this notion was referred as completely orthogonally decomposable tensors.

  3. While finalizing a first version of this work, Professor Bernd Sturmfels kindly pointed out to the authors that the total number of nonzero real singular vector tuples as well as the characterization of the set of nonzero singular vector tuples for an orthogonally decomposable tensor, has also been recently derived in [53] using algebraic geometry tools. It is worth noting that our derivation is more elementary. Moreover, our main concern here is the nondegeneracy, which is not considered in [53].

References

  1. Ammar, A., Chinesta, F., Falcó, A.: On the convergence of a greedy rank-one update algorithm for a class of linear systems. Arch. Comput. Methods Eng. 17, 473–486 (2010)

    Article  MathSciNet  Google Scholar 

  2. Anandkumar, A., Ge, R., Hsu, D., Kakade, S.M., Telgarsky, M.: Tensor decompositions for learning latent variable models. J. Mach. Learn. Res. 15, 2773–2832 (2014)

    MathSciNet  MATH  Google Scholar 

  3. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka–Łojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)

    Article  MathSciNet  Google Scholar 

  4. Beck, A.: First-order methods in optimization. In: MOSSIAM Series on Optimization, 25, xii+475 pp. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA; Mathematical Optimization Society, Philadelphia (2017)

  5. Bolte, J., Daniilidis, A., Ley, O., Mazet, L.: Characterizations of Łojasiewicz inequalities and applications: subgradient flows, talweg, convexity. Trans. Am. Math. Soc. 362, 3319–3363 (2010)

    Article  Google Scholar 

  6. Boothby, W.M.: An Introduction to Differentiable Manifolds and Riemannian Geometry. Academic Press, New York (1975)

    MATH  Google Scholar 

  7. Bott, R., Tu, L.W.: Differential Forms in Algebraic Topology. Springer, New York (1982)

    Book  Google Scholar 

  8. Bro, R.: PARAFAC, tutorial and applications. Chemom. Intell. Lab. Syst. 38, 149–171 (1997)

    Article  Google Scholar 

  9. Cardoso, J.-F., Comon, P.: Independent component analysis, a survey of some algebraic methods. In: The IEEE International Symposium on Circuits and Systems, vol. 2, pp. 93–96. IEEE, New York (1996)

  10. Cardoso, J.-F., Souloumiac, A.: Jacobi angles for simultaneous diagonalization. SIAM J. Matrix Anal. Appl. 17, 161–164 (1996)

    Article  MathSciNet  Google Scholar 

  11. Carroll, J.D., Pruzansky, S.: The CANDECOMP-CANDELINC family of models and methods for multidimensional data analysis. In: Law, H.G., Snyder, C.W., Hattie, J.A., McDonald, R.P. (eds.) Research Methods for Multimode Data Analysis, pp. 372–402. Praeger, New York (1984)

    Google Scholar 

  12. Comon, P.: MA identification using fourth order cumulants. Signal Process. 26, 381–388 (1992)

    Article  Google Scholar 

  13. Comon, P.: Independent component analysis, a new concept? Signal Process. 36, 287–314 (1994)

    Article  Google Scholar 

  14. Comon, P., Mourrain, B.: Decomposition of quantics in sums of powers of linear forms. Signal Process. 53, 93–107 (1996)

    Article  Google Scholar 

  15. D’ Acunto, D., Kurdyka, K.: Explicit bounds for the Łojasiewicz exponent in the gradient inequality for polynomials. Ann. Polon. Math 87, 51–61 (2005)

    Article  MathSciNet  Google Scholar 

  16. De Lathauwer, L.: Signal processing based on multilinear algebra. Ph.D. thesis, Katholieke Universiteit Leuven, Leuven, Belgium (1997)

  17. De Lathauwer, L., De Moor, B., Vandewalle, J.: A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl. 21, 1253–1278 (2000)

    Article  MathSciNet  Google Scholar 

  18. De Lathauwer, L., De Moor, B., Vandewalle, J.: On the best rank-1 and rank-\((R_1, R_2,\dots, R_N)\) approximation of higher-order tensors. SIAM J. Matrix Anal. Appl. 21, 1324–1342 (2000)

    Article  MathSciNet  Google Scholar 

  19. De Lathauwer, L., Comon, P., De Moor, B., Vandewalle, J.: Higher-order power method: application in independent component analysis. In: Proceedings of the International Symposium on Nonlinear Theory and its Applications (NOLTA’95), pp. 91–96. Las Vegas, NV (1995)

  20. de Silva, V., Lim, L.-H.: Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM J. Matrix Anal. Appl. 30, 1084–1127 (2008)

    Article  MathSciNet  Google Scholar 

  21. do Carmo, M.P.: Riemannian Geometry. Springer, Berlin (1992)

    Book  Google Scholar 

  22. Edelman, A., Arias, T., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20, 303–353 (1998)

    Article  MathSciNet  Google Scholar 

  23. Espig, M., Hackbusch, W., Khachatryan, A.: On the convergence of alternating least squares optimization in tensor format representations (2015). arXiv:1506.00062v1

  24. Espig, M., Khachatryan, A.: Convergence of alternating least squares optimization for rank-one approximation to higher order tensors (2015). arXiv:1503.05431

  25. Frankel, P., Garrigos, G., Peypouquet, J.: Splitting methods with variable metric for Kurdyka–Łojasiewicz functions and general convergence rates. J. Optim. Theory Appl. 165, 874–900 (2015)

    Article  MathSciNet  Google Scholar 

  26. Friedland, S., Ottaviani, G.: The number of singular vector tuples and uniqueness of best rank-one approximation of tensors. Found. Comput. Math. 14, 1209–1242 (2014)

    Article  MathSciNet  Google Scholar 

  27. Gelfand, I.M., Kapranov, M.M., Zelevinsky, A.V.: Discriminants, Resultants and Multidimensional Determinants. Birkhäuser, Boston (1994)

    Book  Google Scholar 

  28. Golub, G.H., Van Loan, C.F.: Matrix Computations, 4th edn. Johns Hopkins University Press, Baltimore (2013)

    MATH  Google Scholar 

  29. Grasedyck, L., Kressner, D., Tobler, C.: A literature survey of low-rank tensor approximation techniques. GAMM-Mitteilungen 36, 53–78 (2013)

    Article  MathSciNet  Google Scholar 

  30. Hackbusch, W.: Tensor Spaces and Numerical Tensor Calculus. Springer, Berlin (2012)

    Book  Google Scholar 

  31. Hillar, C.J., Lim, L.-H.: Most tensor problems are NP-hard. J. ACM 60(45), 1–39 (2013)

    Article  MathSciNet  Google Scholar 

  32. Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press, New York (1985)

    Book  Google Scholar 

  33. Kofidis, E., Regalia, P.: On the best rank-1 approximation of higher-order supersymmetric tensors. SIAM J. Matrix Anal. Appl. 23, 863–884 (2002)

    Article  MathSciNet  Google Scholar 

  34. Kolda, T.G.: Orthogonal tensor decompositions. SIAM J. Matrix Anal. Appl. 23, 243–255 (2001)

    Article  MathSciNet  Google Scholar 

  35. Kolda, T.G., Bader, B.W.: Tensor decompositions and applications. SIAM Rev. 51, 455–500 (2009)

    Article  MathSciNet  Google Scholar 

  36. Kroonenberg, P.M.: Three-Mode Principal Component Analysis. DSWO Press, Leiden (1983)

    MATH  Google Scholar 

  37. Landsberg, J.M.: Tensors: Geometry and Applications. AMS, Providence (2012)

    MATH  Google Scholar 

  38. Lang, S.: Algebra, 3rd edn. Springer, Berlin (2004)

    MATH  Google Scholar 

  39. Li, G., Mordukhovich, B.S., Pham, T.S.: New fractional error bounds for polynomial systems with applications to Hölderian stability in optimization and spectral theory of tensors. Math. Program. 153, 333–362 (2015)

    Article  MathSciNet  Google Scholar 

  40. Li, G., Mordukhovich, B.S., Nghia, T.T.A., Pham, T.S.: Error bounds for parametric polynomial systems with applications to higher-order stability analysis and convergence rates. Math. Program. 168, 313–346 (2018)

    Article  MathSciNet  Google Scholar 

  41. Li, G., Pong, T.K.: Calculus of the exponent of Kurdyka–Łojasiewicz inequality and its applications to linear convergence of first-order methods. Found. Comput. Math. (2017). https://doi.org/10.1007/s10208-017-9366-8

    Article  Google Scholar 

  42. Li, G., Qi, L.Q., Yu, G.: Semismoothness of the maximum eigenvalue function of a symmetric tensor and its application. Linear Algebra Appl. 438, 813–833 (2013)

    Article  MathSciNet  Google Scholar 

  43. Lim, L.-H.: Singular values and eigenvalues of tensors: a variational approach. In Proceedings of the 1st IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing, pp. 129–132 (2005)

  44. Łojasiewicz, S.: Une propriété topologique des sous-ensembles analytiques réels, Les Équations aux Dérivées Partielles. Éditions du centre National de la Recherche Scientifique, Paris, pp. 87–89 (1963)

  45. Mohlenkamp, M.J.: Musings on multilinear fitting. Linear Algebra Appl. 438, 834–852 (2013)

    Article  MathSciNet  Google Scholar 

  46. Morse, M.: The Calculus of Variations in the Large. American Mathematical Society, Providence (1934)

    Book  Google Scholar 

  47. Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, Dordrecht (2004)

    Book  Google Scholar 

  48. Nie, J., Wang, L.: Semidefinite relaxations for best rank-1 tensor approximations. SIAM J. Matrix Anal. Appl. 35, 1155–1179 (2014)

    Article  MathSciNet  Google Scholar 

  49. Ortega, J.M., Rheinboldt, W.C.: Iterative Solution of Nonlinear Equations in Several Variables. Springer, Berlin (1970)

    MATH  Google Scholar 

  50. Qi, L.: Eigenvalues of a real supersymmetric tensor. J. Symb. Comput. 40, 1302–1324 (2005)

    Article  MathSciNet  Google Scholar 

  51. Qi, L.: The best rank-one approximation ratio of a tensor space. SIAM J. Matrix Anal. Appl. 32, 430–442 (2011)

    Article  MathSciNet  Google Scholar 

  52. Robeva, E.: Orthogonal decomposition of symmetric tensors. SIAM J. Matrix Anal. Appl. 37, 86–102 (2016)

    Article  MathSciNet  Google Scholar 

  53. Robeva, E., Seigal, A.: Singular vectors of orthogonally decomposable tensors. Linear and Multilinear Algebra (2017). https://doi.org/10.1080/03081087.2016.1277508

    Article  MathSciNet  MATH  Google Scholar 

  54. Rockafellar, R.T., Wets, R.: Variational Analysis. Grundlehren der Mathematischen Wissenschaften, vol. 317. Springer, Berlin (1998)

    Google Scholar 

  55. Schneider, R., Uschmajew, A.: Convergence results for projected line-search methods on varieties of low-rank matrices via Łojasiewicz inequality. SIAM J. Optim. 25, 622–646 (2015)

    Article  MathSciNet  Google Scholar 

  56. Shafarevich, I.R.: Basic Algebraic Geometry. Springer, Berlin (1977)

    MATH  Google Scholar 

  57. Sommese, A.J., Wampler II, C.W.: The Numerical Solution of Systems of Polynomials Arising in Engineering and Science. World Scientific, Hackensack (2005)

    Book  Google Scholar 

  58. Uschmajew, A.: Local convergence of the alternating least squares algorithm for canonical tensor approximation. SIAM J. Matrix Anal. Appl. 33, 639–652 (2012)

    Article  MathSciNet  Google Scholar 

  59. Uschmajew, A.: A new convergence proof for the high-order power method and generalizations. Pacific J. Optim. 11, 309–321 (2015)

    MathSciNet  MATH  Google Scholar 

  60. Wang, L., Chu, M.: On the global convergence of the alternating least squares method for rank-one approximation to generic tensors. SIAM J. Matrix Anal. Appl. 23, 1058–1072 (2014)

    Article  MathSciNet  Google Scholar 

  61. Yang, Y., Feng, Y., Huang, X., Suykens, J.A.K.: Rank-1 tensor properties with applications to a class of tensor optimization problems. SIAM J. Optim. 26, 171–196 (2016)

    Article  MathSciNet  Google Scholar 

  62. Yang, Y., Feng, Y., Suykens, J.A.K.: A rank-one tensor updating algorithm for tensor completion. IEEE Signal Process. Lett. 22, 1633–1637 (2015)

    Article  Google Scholar 

  63. Zhang, T., Golub, G.H.: Rank-one approximation to high order tensors. SIAM J. Matrix Anal. Appl. 23, 534–550 (2001)

    Article  MathSciNet  Google Scholar 

  64. Zhang, X., Ling, C., Qi, L.: The best rank-1 approximation of a symmetric tensor and related spherical optimization problems. SIAM J. Matrix Anal. Appl. 33, 806–821 (2012)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors are grateful to the referees and the editor for their constructive comments and helpful suggestions which have contributed to the final presentation of the paper. The authors would also like to thank Dr. Yang Qi (University of Chicago) for pointing out the reference [7]. The first author’s work is partially supported by National Science Foundation of China (Grant No. 11771328), Young Elite Scientists Sponsorship Program by Tianjin, and Innovation Research Foundation of Tianjin University (Grant Nos. 2017XZC-0084 and 2017XRG-0015). The second author is partially supported by a Future fellowship from Australian Research Council (FT130100038) and a discovery project from Australian Research Council (DP180100745).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Guoyin Li.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, S., Li, G. Convergence rate analysis for the higher order power method in best rank one approximations of tensors. Numer. Math. 140, 993–1031 (2018). https://doi.org/10.1007/s00211-018-0981-3

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00211-018-0981-3

Mathematics Subject Classification

Navigation