Skip to main content
SpringerLink
Log in

Unified linear convergence of first-order primal-dual algorithms for saddle point problems

  • Original Paper
  • Published:
Optimization Letters Aims and scope Submit manuscript

Abstract

In this paper, we study the linear convergence of several well-known first-order primal-dual methods for solving a class of convex-concave saddle point problems. We first unify the convergence analysis of these methods and prove the O(1/N) convergence rates of the primal-dual gap generated by these methods in the ergodic sense, where N counts the number of iterations. Under a mild calmness condition, we further establish the global Q-linear convergence rate of the distances between the iterates generated by these methods and the solution set, and show the R-linear rate of the iterates in the nonergodic sense. Moreover, we demonstrate that the matrix games, fused lasso and constrained TV-\(\ell _2\) image restoration models as application examples satisfy this calmness condition. Numerical experiments on fused lasso demonstrate the linear rates for these methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Similar content being viewed by others

References

  1. Cai, X., Han, D., Xu, L.: An improved first-order primal-dual algorithm with a new correction step. J. Global Optim. 57, 1419–1428 (2013)

    Article  MathSciNet  Google Scholar 

  2. Chambolle, A., Ehrhardt, M.J., Richtárik, P., Schonlieb, C.-B.: Stochastic primal-dual hybrid gradient algorithm with arbitrary sampling and imaging applications. SIAM J. Optim. 28, 2783–2808 (2018)

    Article  MathSciNet  Google Scholar 

  3. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40, 120–145 (2011)

    Article  MathSciNet  Google Scholar 

  4. Chambolle, A., Pock, T.: On the ergodic convergence rates of a first-order primal-dual algorithm. Math. Program. 159, 253–287 (2016)

    Article  MathSciNet  Google Scholar 

  5. Chen, P., Huang, J., Zhang, X.: A primal-dual fixed point algorithm for convex separable minimization with applications to image restoration. Inverse Prob. 29, 025011 (2013)

    Article  MathSciNet  Google Scholar 

  6. Chen, P., Huang, J., Zhang, X.: A primal-dual fixed point algorithm for minimization of the sum of three convex separable functions. Fixed Point Theory Appl. 2016, 1–18 (2016)

    Article  MathSciNet  Google Scholar 

  7. Condat, L.: A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms. J. Optim. Theory Appl. 158, 460–479 (2013)

    Article  MathSciNet  Google Scholar 

  8. Dontchev, A.L., Rockafellar, R.T.: Implicit Functions and Solution Mappings. Springer Monographs in Mathematics, vol. 208. Springer, Berlin (2009)

    Book  Google Scholar 

  9. Drori, Y., Sabach, S., Teboulle, M.: A simple algorithm for a class of nonsmooth convex-concave saddle-point problems. Oper. Res. Lett. 43, 209–214 (2015)

    Article  MathSciNet  Google Scholar 

  10. Esser, E., Zhang, X., Chan, T.F.: A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM J. Imag. Sci. 3, 1015–1046 (2010)

    Article  MathSciNet  Google Scholar 

  11. Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Numerical Solution of Boundary-Value Problems, pp. 299–331. North-Holland, Amsterdam (1983)

  12. Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2, 17–40 (1976)

    Article  Google Scholar 

  13. Glowinski, R., Marroco, A.: Sur l’approximation, par éléments finis d’ordre un, et la résolution, par pénalisation-dualité d’une classe de problèmes de dirichlet non linéaires, ESAIM: Mathematical Modelling and Numerical Analysis-Modélisation Mathématique et Analyse Numérique, 9 (1975), pp. 41–76

  14. Han, D., He, H., Yang, H., Yuan, X.: A customized Douglas–Rachford splitting algorithm for separable convex minimization with linear constraints. Numer. Math. 127, 167–200 (2014)

    Article  MathSciNet  Google Scholar 

  15. Han, D., Sun, D., Zhang, L.: Linear rate convergence of the alternating direction method of multipliers for convex composite programming. Math. Oper. Res. 43, 622–637 (2017)

    Article  MathSciNet  Google Scholar 

  16. He, B., Yuan, X.: Convergence analysis of primal-dual algorithms for a saddle-point problem: from contraction perspective. SIAM J. Imag. Sci. 5, 119–149 (2012)

    Article  MathSciNet  Google Scholar 

  17. He, H., Desai, J., Wang, K.: A primal-dual prediction-correction algorithm for saddle point optimization. J. Global Optim. 66, 573–583 (2016)

    Article  MathSciNet  Google Scholar 

  18. Jiang, F., Cai, X., Wu, Z., Han, D.: Approximate first-order primal-dual algorithms for saddle point problems. Math. Comput. 90, 1227–1262 (2021)

    Article  MathSciNet  Google Scholar 

  19. Jiang, F., Wu, Z., Cai, X., Zhang, H.: A first-order inexact primal-dual algorithm for a class of convex-concave saddle point problems. Numer. Algorithms 88, 1109–1136 (2021)

  20. Latafat, P., Patrinos, P.: Asymmetric forward-backward-adjoint splitting for solving monotone inclusions involving three operators. Comput. Optim. Appl. 68, 57–93 (2017)

    Article  MathSciNet  Google Scholar 

  21. Liu, J., Yuan, L., Ye, J.: An efficient algorithm for a class of fused lasso problems. In: Proceedings of the 16th ACM SIGKDD International Donference on Knowledge Discovery and Data Mining, ACM, pp. 323–332 (2010)

  22. Loris, I., Verhoeven, C.: On a generalization of the iterative soft-thresholding algorithm for the case of non-separable penalty. Inverse Prob. 27, 125007 (2011)

    Article  MathSciNet  Google Scholar 

  23. Malitsky, Y., Pock, T.: A first-order primal-dual algorithm with linesearch. SIAM J. Optim. 28, 411–432 (2018)

    Article  MathSciNet  Google Scholar 

  24. Mokhtari, A., Ozdaglar, A., Pattathil, S.: A unified analysis of extra-gradient and optimistic gradient methods for saddle point problems: Proximal point approach. In: International Conference on Artificial Intelligence and Statistics, PMLR, pp. 1497–1507 (2020)

  25. Mokhtari, A., Ozdaglar, A.E., Pattathil, S.: Convergence rate of O(1/k) for optimistic gradient and extragradient methods in smooth convex-concave saddle point problems. SIAM J. Optim. 30, 3230–3251 (2020)

    Article  MathSciNet  Google Scholar 

  26. Morini, B., Porcelli, M., Chan, R.H.: A reduced Newton method for constrained linear least-squares problems. J. Comput. Appl. Math. 233, 2200–2212 (2010)

    Article  MathSciNet  Google Scholar 

  27. Nesterov, Y.: Lectures on Convex Optimization, vol. 137. Springer, Berlin (2018)

    Book  Google Scholar 

  28. O’Connor, D., Vandenberghe, L.: On the equivalence of the primal-dual hybrid gradient method and Douglas–Rachford splitting. Math. Program. 179, 85–108 (2020)

  29. Parikh, N., Boyd, S., et al.: Proximal algorithms, Foundations and Trends®. Optimization 1, 127–239 (2014)

    Google Scholar 

  30. Rasch, J., Chambolle, A.: Inexact first-order primal-dual algorithms. Comput. Optim. Appl. 2, 381–430 (2020)

    Article  MathSciNet  Google Scholar 

  31. Robinson, S.M.: An implicit-function theorem for generalized variational inequalities. tech. rep., Wisconsin Univ Madison Mathmatics Research Center, (1976)

  32. Robinson, S.M.: Some continuity properties of polyhedral multifunctions. In: König, H., Korte, B., Ritter, K. (eds.) Mathematical Programming at Oberwolfach, pp. 206–214. Springer, Berlin (1981)

  33. Rockafellar, R.T., Wets, R.J.-B.: Variational Analysis, vol. 317. Springer Science & Business Media, Berlin (2009)

    MATH  Google Scholar 

  34. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60, 259–268 (1992)

    Article  MathSciNet  Google Scholar 

  35. Shi, F., Cheng, J., Wang, L., Yap, P.-T., Shen, D.: Low-rank total variation for image super-resolution. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp. 155–162 (2013)

  36. Shi, F., Cheng, J., Wang, L., Yap, P.-T., Shen, D.: LRTV: MR image super-resolution with low-rank and total variation regularizations. IEEE Trans. Med. Imaging 34, 2459–2466 (2015)

    Article  Google Scholar 

  37. Tibshirani, R., Saunders, M., Rosset, S., Zhu, J., Knight, K.: Sparsity and smoothness via the fused lasso. J. Royal Stat. Soc. Ser. B (Stat. Methodol.) 67, 91–108 (2005)

    Article  MathSciNet  Google Scholar 

  38. Vũ, B.C.: A splitting algorithm for dual monotone inclusions involving cocoercive operators. Adv. Comput. Math. 38, 667–681 (2013)

    Article  MathSciNet  Google Scholar 

  39. Yan, M.: A new primal-dual algorithm for minimizing the sum of three functions with a linear operator. J. Sci. Comput. 76, 1698–1717 (2018)

    Article  MathSciNet  Google Scholar 

  40. Yuan, M., Lin, Y.: Model selection and estimation in regression with grouped variables. J. Royal Stat. Soc. Ser. B (Statistical Methodology) 68, 49–67 (2006)

    Article  MathSciNet  Google Scholar 

  41. Zhu, M., Chan, T.: An efficient primal-dual hybrid gradient algorithm for total variation image restoration, UCLA CAM Report, 34 (2008)

Download references

Author information

Authors and Affiliations

  1. Department of Information and Computing Science, School of Mathematics and Statistics, Nanjing University of Information Science and Technology, Nanjing, 210044, China

    Fan Jiang

  2. School of Management Science and Engineering, Nanjing University of Information Science and Technology, Nanjing, 210044, China

    Zhongming Wu

  3. School of Mathematical Sciences, Jiangsu Key Laboratory for NSLSCS, Nanjing Normal University, Nanjing, 210023, China

    Xingju Cai

  4. Department of Mathematics, Louisiana State University, Baton Rouge, LA, 70803-4918, USA

    Hongchao Zhang

Authors
  1. Fan Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhongming Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xingju Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hongchao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fan Jiang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This research was partially supported by the National Natural Science Foundation of China under grants 11571178, 11871279, 11971238 and 12001286, and by the USA National Science Foundation under grants 1819161 and 2110722.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, F., Wu, Z., Cai, X. et al. Unified linear convergence of first-order primal-dual algorithms for saddle point problems. Optim Lett 16, 1675–1700 (2022). https://doi.org/10.1007/s11590-021-01832-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11590-021-01832-y

Keywords

Search

Navigation