Skip to main content
SpringerLink
Log in

On Convergence of the Arrow–Hurwicz Method for Saddle Point Problems

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

The Arrow–Hurwicz method is an inexact version of the Uzawa method; it has been widely applied to solve various saddle point problems in different areas including many fundamental image processing problems. It is also the basis of a number of important algorithms such as the extragradient method and the primal–dual hybrid gradient method. Convergence of the classic Arrow–Hurwicz method, however, is known only when some more restrictive conditions are additionally assumed, such as strong convexity of the functions or some demanding requirements on the step sizes. In this short note, we show by very simple counterexamples that the classic Arrow–Hurwicz method with any constant step size is not necessarily convergent for solving generic convex saddle point problems, including some fundamental cases such as the canonical linear programming model and the bilinear saddle point problem. This result plainly fathoms the convergence understanding of the Arrow–Hurwicz method and retrospectively validates the rationale of studying its convergence under various additional conditions in image processing literature.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Arrow, K.J., Hurwicz, L., Uzawa, H.: Studies in Linear and Non-linear Programming. Stanford University Press, Stanford (1958)

    MATH  Google Scholar 

  2. Bacuta, C.: A unified approach for Uzawa algorithms. SIAM J. Numer. Anal. 44, 2633–2649 (2006)

    MathSciNet  MATH  Google Scholar 

  3. Benzi, M., Golub, G.H., Liesen, J.: Numerical solution of saddle point problems. Acta Numer. 14, 1–137 (2005)

    MathSciNet  MATH  Google Scholar 

  4. Bonettini, S., Ruggiero, V.: On the convergence of primal-dual hybrid gradient algorithms for total variation image restoration. J. Math. Imaging Vis. 44, 236–253 (2012)

    MathSciNet  MATH  Google Scholar 

  5. Bramble, J.H., Pasciak, J.E., Vassilev, A.T.: Analysis of the inexact Uzawa algorithm for saddle point problems. SIAM J. Numer. Anal. 34, 1072–1092 (1997)

    MathSciNet  MATH  Google Scholar 

  6. Brezzi, F., Fortin, M.: Mixed and Hybrid Finite Element Methods. Springer Series in Computational Mathematics, vol. 15. Springer, New York (2012)

    Google Scholar 

  7. Chambolle, A., Caselles, V., Cremers, D., Novaga, M., Pock, T.: An introduction to total variation for image analysis. In: Theoretical Foundations and Numerical Methods for Sparse Recovery, Volume 9 of Radon Series on Computational and Applied Mathematics. Walter de Gruyter, Berlin, pp. 263–340 (2010)

  8. Chambolle, A., Pock, T.: A first-order primal–dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40, 120–145 (2011)

    MathSciNet  MATH  Google Scholar 

  9. Chambolle, A., Pock, T.: An introduction to continuous optimization for imaging. Acta Numer. 25, 161–319 (2016)

    MathSciNet  MATH  Google Scholar 

  10. Chambolle, A., Pock, T.: On the ergodic convergence rates of a first-order primal–dual algorithm. Math. Program. 159, 253–287 (2016)

    MathSciNet  MATH  Google Scholar 

  11. Ciarlet, P.G., Miara, B., Thomas, J.-M.: Introduction to Numerical Linear Algebra and Optimization. Cambridge University Press, Cambridge (1989)

    Google Scholar 

  12. Elman, H.C., Golub, G.H.: Inexact and preconditioned Uzawa algorithms for saddle point problems. SIAM J. Numer. Anal. 31, 1645–1661 (1994)

    MathSciNet  MATH  Google Scholar 

  13. Esser, E., Zhang, X., Chan, T.F.: A general framework for a class of first order primal–dual algorithms for convex optimization in imaging science. SIAM J. Imaging Sci. 3, 1015–1046 (2010)

    MathSciNet  MATH  Google Scholar 

  14. Goldstein, T., Li, M., Yuan, X.M.: Adaptive primal-dual splitting methods for statistical learning and image processing. In: Advances in Neural Information Processing Systems, pp. 2089–2097 (2015)

  15. He, B.S., Ma, F., Yuan, X.M.: An algorithmic framework of generalized primal–dual hybrid gradient methods for saddle point problems. J. Math. Imaging Vis. 58, 279–293 (2017)

    MathSciNet  MATH  Google Scholar 

  16. He, B.S., You, Y.F., Yuan, X.M.: On the convergence of primal–dual hybrid gradient algorithm. SIAM J. Imaging Sci. 7, 2526–2537 (2014)

    MathSciNet  MATH  Google Scholar 

  17. He, B.S., Yuan, X.M.: Convergence analysis of primal–dual algorithms for a saddle-point problem: from contraction perspective. SIAM J. Imaging Sci. 5, 119–149 (2012)

    MathSciNet  MATH  Google Scholar 

  18. Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)

    MathSciNet  MATH  Google Scholar 

  19. Hu, Q., Zou, J.: Nonlinear inexact Uzawa algorithms for linear and nonlinear saddle-point problems. SIAM J. Optim. 16, 798–825 (2006)

    MathSciNet  MATH  Google Scholar 

  20. Kincaid, D., Cheney, W.: Numerical Analysis: Mathematics of Scientific Computing, 3rd edn. American Mathematical Society, Providence (2002)

    MATH  Google Scholar 

  21. Korpelevich, G.M.: The extragradient method for finding saddle points and other problems. Ekon. Mat. Metody. 12(4), 747–756 (1976)

    MathSciNet  MATH  Google Scholar 

  22. Luenberger, D.G., Ye, Y.: Linear and Nonlinear Programming. Springer, New York (2008)

    MATH  Google Scholar 

  23. Malitsky, Y., Pock, T.: A first-order primal–dual algorithm with line search. SIAM J. Optim. 28, 411–432 (2018)

    MathSciNet  MATH  Google Scholar 

  24. Möllenhoff, T., Strekalovskiy, E., Moeller, M., Cremers, D.: The primal–dual hybrid gradient method for semiconvex splittings. SIAM J. Imaging Sci. 8(2), 827–857 (2015)

    MathSciNet  MATH  Google Scholar 

  25. Pestana, J., Wathen, A.J.: Natural preconditioning and iterative methods for saddle point systems. SIAM Rev. 57(1), 71–91 (2015)

    MathSciNet  MATH  Google Scholar 

  26. Pock, T., Chambolle, A.: Diagonal preconditioning for first order primal–dual algorithms in convex optimization. In: IEEE International Conference on Computer Vision, pp. 1762–1769 (2011)

  27. Popov, L.D.: A modification of the Arrow–Hurwitz method of search for saddle points. Mat. Zametki. 28(5), 777–784 (1980)

    MathSciNet  MATH  Google Scholar 

  28. Powell, M.J.: A method for non-linear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization. Academic Press, New York (1969)

    Google Scholar 

  29. Quarteroni, A., Valli, A.: Numerical Approximation of Partial Differential Equations. Springer Series in Computational Mathematics, vol. 23. Springer, Berlin (1994)

    MATH  Google Scholar 

  30. Queck, W.: The convergence factor of preconditioned algorithms of the Arrow–Hurwicz type. SIAM J. Numer. Anal. 26, 1016–1030 (1989)

    MathSciNet  MATH  Google Scholar 

  31. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60(1–4), 259–268 (1992)

    MathSciNet  MATH  Google Scholar 

  32. Song, Y., Yuan, X., Yue, H.: An inexact Uzawa algorithmic framework for nonlinear saddle point problems with applications to elliptic optimal control problem. SIAM J. Numer. Anal. 57, 2656–2684 (2019)

    MathSciNet  MATH  Google Scholar 

  33. Stoer, J., Bulirsch, R.: Introduction to Numerical Analysis. Texts in Applied Mathematics, vol. 12. Springer, New York (2013)

    Google Scholar 

  34. Wesseling, P.: Principles of Computational Fluid Dynamics. Springer Series in Computational Mathematics, vol. 29. Springer, Berlin (2001)

    Google Scholar 

  35. Zhang, X., Burger, M., Osher, S.: A unified primal–dual algorithm framework based on Bregman iteration. J. Sci. Comput. 46, 20–46 (2011)

    MathSciNet  MATH  Google Scholar 

  36. Zhu, M., Chan, T.: An efficient primal–dual hybrid gradient algorithm for total variation image restoration. CAM report 08–34, UCLA, Los Angeles, CA (2008)

Download references

Funding

This study was supported by National Natural Science Foundation of China (Grant No. 11871029) and Research Grants Council, University Grants Committee (Grant No. 12302318).

Author information

Author notes

  1. Shengjie Xu and Xiaoming Yuan have contributed equally to this work.

Authors and Affiliations

  1. Department of Mathematics, Nanjing University, Nanjing, China

    Bingsheng He

  2. Department of Mathematics, Harbin Institute of Technology, Harbin, China

    Shengjie Xu

  3. Department of Mathematics, Southern University of Science and Technology, Shenzhen, China

    Shengjie Xu

  4. Department of Mathematics, The University of Hong Kong, Pok Fu Lam, Hong Kong

    Xiaoming Yuan

Authors
  1. Bingsheng He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shengjie Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoming Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoming Yuan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, B., Xu, S. & Yuan, X. On Convergence of the Arrow–Hurwicz Method for Saddle Point Problems. J Math Imaging Vis 64, 662–671 (2022). https://doi.org/10.1007/s10851-022-01089-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-022-01089-9

Keywords

Search

Navigation