Skip to main content

Advertisement

Springer Nature Link
Log in

Analysis of Optimization Algorithms via Sum-of-Squares

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

We introduce a new framework for unifying and systematizing the performance analysis of first-order black-box optimization algorithms for unconstrained convex minimization. The low-cost iteration complexity enjoyed by first-order algorithms renders them particularly relevant for applications in machine learning and large-scale data analysis. Relying on sum-of-squares (SOS) optimization, we introduce a hierarchy of semidefinite programs that give increasingly better convergence bounds for higher levels of the hierarchy. Alluding to the power of the SOS hierarchy, we show that the (dual of the) first level corresponds to the performance estimation problem (PEP) introduced by Drori and Teboulle (Math Program 145(1):451–482, 2014), a powerful framework for determining convergence rates of first-order optimization algorithms. Consequently, many results obtained within the PEP framework can be reinterpreted as degree-1 SOS proofs, and thus, the SOS framework provides a promising new approach for certifying improved rates of convergence by means of higher-order SOS certificates. To determine analytical rate bounds, in this work, we use the first level of the SOS hierarchy and derive new results for noisy gradient descent with inexact line search methods (Armijo, Wolfe, and Goldstein).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Ahmadi, A.A.: Sum of squares (SOS) techniques: an introduction. http://www.princeton.edu/~amirali/Public/Teaching/ORF523/S16/ORF523_S16_Lec15.pdf. Accessed 2020

  2. Bertsekas, D.P.: Nonlinear Programming, 2nd edn. Athena Scientific, Nashua (1999)

    MATH  Google Scholar 

  3. Blekherman, G., Parrilo, P.A., Thomas, R.R.: Semidefinite Optimization and Convex Algebraic Geometry, vol. 13. MOS-SIAM Series on Optimization (2012)

  4. Boyd, S.P., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2009)

    MATH  Google Scholar 

  5. Cohen, A.I.: Rate of convergence of several conjugate gradient algorithms. SIAM J. Numer. Anal. 9(2), 248–259 (1972)

    Article  MathSciNet  Google Scholar 

  6. Dai, Y.-H.: Nonlinear conjugate gradient methods. Wiley Encyclopedia of Operations Research and Management Science (2011)

  7. de Klerk, E., Glineur, F., Taylor, A.B.: On the worst-case complexity of the gradient method with exact line search for smooth strongly convex functions. Optim. Lett. 11(7), 1185–1199 (2017)

    Article  MathSciNet  Google Scholar 

  8. de Klerk, E., Glineur, F., Taylor, A.B.: Worst-case convergence analysis of gradient and Newton methods through semidefinite programming performance estimation. Technical Report, https://arxiv.org/abs/1709.05191 (2017). Accessed 2017

  9. Drori, Y., Taylor, A.B.: Efficient first-order methods for convex minimization: a constructive approach. Math. Program. 184, 183–220 (2019)

    Article  MathSciNet  Google Scholar 

  10. Drori, Y., Teboulle, M.: Performance of first-order methods for smooth convex minimization: a novel approach. Math. Program. 145(1), 451–482 (2014)

    Article  MathSciNet  Google Scholar 

  11. Drori, Y., Teboulle, M.: An optimal variant of Kelley’s cutting-plane method. Math. Program. 160(1), 321–351 (2016)

    Article  MathSciNet  Google Scholar 

  12. Fazlyab, M., Morari, M., Preciado, V.M.: Design of first-order optimization algorithms via sum-of-squares programming. In: The 18th IEEE Conference on Decision and Control, p. 4445–4452 (2018)

  13. Fazlyab, M., Ribeiro, A., Morari, M., Preciado, V.M.: Analysis of optimization algorithms via integral quadratic constraints: nonstrongly convex problems. SIAM J. Optim. 28(3), 2654–2689 (2018)

    Article  MathSciNet  Google Scholar 

  14. Grant, M., Boydm, S.: Graph implementations for nonsmooth convex programs. In: Blondel, V., Boyd, S., Kimura, H. (eds.) Recent Advances in Learning and Control, Lecture Notes in Control and Information Sciences, pp. 95–110. Springer, Cham (2008)

    Google Scholar 

  15. Grant, M., Boyd, S.: CVX: Matlab software for disciplined convex programming, version 2.1. http://cvxr.com/cvx (2014). Accessed 2020

  16. Gu, G., Yang, J.: Optimal nonergodic sublinear convergence rate of proximal point algorithm for maximal monotone inclusion problems. Technical Report, https://arxiv.org/abs/1904.05495 (2019)

  17. Hu, B., Seiler, P., Lessard, L.: Analysis of approximate stochastic gradient using quadratic constraints and sequential semidefinite programs. Technical Report, https://arxiv.org/pdf/1711.00987.pdf (2017). Accessed 2020

  18. Kim, D.: Accelerated proximal point method and forward method for monotone inclusions. Technical Report, https://arxiv.org/abs/1905.05149 (2019)

  19. Kim, D., Fessler, J.A.: Optimized first-order methods for smooth convex minimization. Math. Program. 159(1), 81–107 (2016)

    Article  MathSciNet  Google Scholar 

  20. Kim, D., Fessler, J.A.: Optimizing the efficiency of first-order methods for decreasing the gradient of smooth convex functions. Technical Report, https://arxiv.org/abs/1803.06600 (2018)

  21. Lasserre, J.: A sum of squares approximation of nonnegative polynomials. SIAM Rev. 49(4), 651–669 (2007)

    Article  MathSciNet  Google Scholar 

  22. Laurent, M.: Sums of Squares, Moment Matrices and Optimization Over Polynomials, pp. 157–270. Springer, New York (2009)

    MATH  Google Scholar 

  23. Lessard, L., Recht, B., Packard, A.: Analysis and design of optimization algorithms via integral quadratic constraints. SIAM J. Optim. 26(1), 57–95 (2016)

    Article  MathSciNet  Google Scholar 

  24. Lieder, F.: On the convergence rate of the Halpern-iteration. Technical Report, http://www.optimization-online.org/DB_FILE/2017/11/6336.pdf (2019). Accessed 2020

  25. Luenberger, D.G., Ye, Y.: Linear and Nonlinear Programming, 4th edn. Springer International Publishing, Cham (2016)

    Book  Google Scholar 

  26. Nemirovski, A.: Optimization II: Numerical methods for nonlinear continuous optimization. https://www2.isye.gatech.edu/~nemirovs/Lect_OptII.pdf. Accessed 2020

  27. Nishihara, R., Lessard, L., Recht, B., Packard, A., Jordan, M.I.: A general analysis of the convergence of ADMM. In: Proceedings of the 32nd International Conference on Machine Learning, vol. 37 (2015)

  28. Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, New York (2006)

    MATH  Google Scholar 

  29. Parrilo, P.A.: Semidefinite programming relaxations for semialgebraic problems. Math. Program. 96(2), 293–320 (2003)

    Article  MathSciNet  Google Scholar 

  30. Parrilo, P.A.: Polynomial Optimization, Sums of Squares, and Applications. Society for Industrial and Applied Mathematics, Philadelphia (2013)

    MATH  Google Scholar 

  31. Putinar, M.: Positive polynomials on compact semi-algebraic sets. Indiana Univ. Math. J. 42(3), 969–984 (1993)

    Article  MathSciNet  Google Scholar 

  32. Ryu, E.K., Taylor, A.B., Bergeling, C., Giselsson, P.: Operator splitting performance estimation: tight contraction factors and optimal parameter selection. Technical Report, https://arxiv.org/abs/1812.00146 (2018)

  33. Tan, S.S.Y.: Performance analysis of optimization algorithms using semidefinite programming. Master’s thesis, Department of Electrical and Computer Engineering, National University of Singapore, https://scholarbank.nus.edu.sg/handle/10635/170801 (2020). Accessed 2020

  34. Tan, S.S.Y., Varvitsiotis, A., Tan, V.Y.F.: A unified framework for the convergence analysis of optimization algorithms via sums-of-squares. In: The Signal Processing with Adaptive Sparse Structured Representations (SPARS) Workshop (2019)

  35. Taylor, A., Scoy, B.V., Lessard, L.: Lyapunov functions for first-order methods: tight automated convergence guarantees. In: 35th International Conference on Machine Learning (2018)

  36. Taylor, A.B., Bach, F.: Stochastic first-order methods: non-asymptotic and computer-aided analyses via potential functions. Proc. Mach. Learn. Res. 99, 1–58 (2019)

    Google Scholar 

  37. Taylor, A.B., Hendrickx, J., Glineur, F.: Exact worst-case performance of first-order methods for composite convex optimization. SIAM J. Optim. 27(3), 1283–1313 (2017)

    Article  MathSciNet  Google Scholar 

  38. Taylor, A.B., Hendrickx, J.M., Glineur, F.: Smooth strongly convex interpolation and exact worst-case performance of first-order methods. Math. Program. 161(1), 307–345 (2017)

    Article  MathSciNet  Google Scholar 

  39. Taylor, A.B., Hendrickx, J.M., Glineur, F.: Exact worst-case convergence rates of the proximal gradient method for composite convex minimization. J. Optim. Theory Appl. 178(2), 455–476 (2018)

    Article  MathSciNet  Google Scholar 

  40. Toh, K.C., Todd, M.J., Tütüncü, R.H.: SDPT3—a Matlab software package for semidefinite programming, Version 1.3. Optim. Methods Softw. 11(1–4), 545–581 (1999)

    Article  MathSciNet  Google Scholar 

  41. Tütüncü, R.H., Toh, K.C., Todd, M.J.: Solving semidefinite-quadratic-linear programs using SDPT3. Math. Program. 95(2), 189–217 (2003)

    Article  MathSciNet  Google Scholar 

  42. Wolfram Research, Inc. Mathematica, Version 12.0. Champaign, IL (2019)

Download references

Acknowledgements

The authors are supported by a Singapore National Research Foundation (NRF) Fellowship (R-263-000-D02-281).

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore

    Sandra S. Y. Tan

  2. Department of Electrical and Computer Engineering and Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore, Singapore

    Antonios Varvitsiotis

  3. Department of Electrical and Computer Engineering and Department of Mathematics, National University of Singapore, Singapore, Singapore

    Vincent Y. F. Tan

Authors
  1. Sandra S. Y. Tan
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Antonios Varvitsiotis
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Vincent Y. F. Tan
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Antonios Varvitsiotis.

Additional information

Communicated by Marc Teboulle.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tan, S.S.Y., Varvitsiotis, A. & Tan, V.Y.F. Analysis of Optimization Algorithms via Sum-of-Squares. J Optim Theory Appl 190, 56–81 (2021). https://doi.org/10.1007/s10957-021-01869-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-021-01869-0

Keywords