Skip to main content
SpringerLink
Log in

An alternative extrapolation scheme of PDHGM for saddle point problem with nonlinear function

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

Primal-dual hybrid gradient (PDHG) method is a canonical and popular prototype for solving saddle point problem (SPP). However, the nonlinear coupling term in SPP excludes the application of PDHG on far-reaching real-world problems. In this paper, following the seminal work by Valkonen (Inverse Problems 30, 2014), we devise a variant iterative scheme for solving SPP with nonlinear function by exerting an alternative extrapolation procedure. The novel iterative scheme falls exactly into the proximal point algorithmic framework without any residuals, which indicates that the associated inclusion problem is “nearer” to the KKT mapping induced by SPP. Under the metrically regular assumption on KKT mapping, we simplify the local convergence of the proposed method on contractive perspective. Numerical simulations on a PDE-constrained nonlinear inverse problem demonstrate the compelling performance of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability statement

All data generated or analyzed during this study are available from the corresponding author.

Notes

  1. By the acronym in [18, 43], we shall term the recursion (2) with \(\theta =0\) (resp. \(\theta \ne 0\)) as primal-dual hybrid gradient method (resp. modified primal-dual hybrid gradient).

  2. The critical point of SPP varies in different literature. We herein follow the definition in [13].

  3. Herein, for ease of description, we present the nonlinear inverse problem in continuum context as in [13, 15].

  4. Codes of accelerated-NL-PDHGM are available at https://github.com/clason/nlpdegm.

References

  1. Artacho, F.J.A., Dontchev, A.L., Gaydu, M., Geoffroy, M.H., Veliov, V.M.: Metric regularity of Newton’s iteration. SIAM J. Control Optim. 49, 339–362 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  2. Başar, T., Olsder, G.J.: Dynamic Noncooperative Game Theory. SIAM (1998)

  3. Ben-Tal, A., El Ghaoui, L., Nemirovski, A.: Robust Optimization. Princeton University Press, Princeton (2009)

    Book  MATH  Google Scholar 

  4. Benzi, M., Golub, G.H., Liesen, J.: Numerical solution of saddle point problems. Acta Numer. 14, 1–137 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  5. Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146, 459–494 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  6. Cai, X., Han, D., Xu, L.: An improved first-order primal-dual algorithm with a new correction step. J. Global Optim. 57, 1419–1428 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  7. Chambolle, A., Ehrhardt, M.J., Richtárik, P., Schönlieb, C.B.: Stochastic primal-dual hybrid gradient algorithm with arbitrary sampling and imaging applications. SIAM J. Optim. 28, 2783–2808 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  8. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vision 40, 120–145 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  9. Chambolle, A., Pock, T.: On the ergodic convergence rates of a first-order primal-dual algorithm. Math. Program. 159, 253–287 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  10. Chen, Y., Lan, G., Ouyang, Y.: Optimal primal-dual methods for a class of saddle point problems. SIAM J. Optim. 24, 1779–1814 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  11. Chung, J., Nagy, J.G.: An efficient iterative approach for large-scale separable nonlinear inverse problems. SIAM J. Sci. Comput. 31, 4654–4674 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  12. Clason, C., Jin, B.: A semismooth Newton method for nonlinear parameter identification problems with impulsive noise. SIAM J. Imaging Sci. 5, 505–536 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  13. Clason, C., Mazurenko, S., Valkonen, T.: Acceleration and global convergence of a first-order primal-dual method for nonconvex problems. SIAM J. Optim. 29, 933–963 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  14. Clason, C., Mazurenko, S., Valkonen, T.: Primal-dual proximal splitting and generalized conjugation in non-smooth non-convex optimization. Appl. Math. Optim. 84, 1239–1284 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  15. Clason, C., Valkonen, T.: Primal-dual extragradient methods for nonlinear nonsmooth PDE-constrained optimization. SIAM J. Optim. 27, 1314–1339 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  16. Clason, C., Valkonen, T.: Stability of saddle points via explicit coderivatives of pointwise subdifferentials. Set-Valued Var. Anal. 25, 69–112 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  17. Condat, L.: A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms. J. Optim. Theory Appl. 158, 460–479 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  18. Esser, E., Zhang, X., Chan, T.F.: A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM J. Imaging Sci. 3, 1015–1046 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  19. Fercoq, O., Bianchi, P.: A coordinate-descent primal-dual algorithm with large step size and possibly nonseparable functions. SIAM J. Optim. 29, 100–134 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  20. Fusek, P.: On metric regularity for weakly almost piecewise smooth functions and some applications in nonlinear semidefinite programming. SIAM J. Optim. 23, 1041–1061 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  21. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Proc. Adv. Neural Inf. Process. Syst. pp 2672–2680 (2014)

  22. Gu, G., He, B., Yuan, X.: Customized proximal point algorithms for linearly constrained convex minimization and saddle-point problems: a unified approach. Comput. Optim. Appl. 59, 135–161 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  23. Hart, P.E., Stork, D.G., Duda, R.O.: Pattern Classification. Wiley Hoboken, Hoboken (2000)

    MATH  Google Scholar 

  24. He, B., Ma, F., Yuan, X.: An algorithmic framework of generalized primal-dual hybrid gradient methods for saddle point problems. J. Math. Imaging Vision 58, 279–293 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  25. He, B., You, Y., Yuan, X.: On the convergence of primal-dual hybrid gradient algorithm. SIAM J. Imaging Sci. 7, 2526–2537 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  26. He, B., Yuan, X.: Convergence analysis of primal-dual algorithms for a saddle-point problem: from contraction perspective. SIAM J. Imaging Sci. 5, 119–149 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  27. He, B., Yuan, X.: Balanced augmented Lagrangian method for convex programming. arXiv:2108.08554 (2021)

  28. He, B., Yuan, X., Zhang, W.: A customized proximal point algorithm for convex minimization with linear constraints. Comput. Optim. Appl. 56, 559–572 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  29. Hiriart-Urruty, J.B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms. Springer, Berlin (1993)

    Book  MATH  Google Scholar 

  30. Jiang, F., Cai, X., Wu, Z., Han, D.: Approximate first-order primal-dual algorithms for saddle point problems. Math. Comput. 90, 1227–1262 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  31. Keung, Y.L., Zou, J.: An efficient linear solver for nonlinear parameter identification problems. SIAM J. Sci. Comput. 22, 1511–1526 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  32. Li, Z., Yan, M.: New convergence analysis of a primal-dual algorithm with large stepsizes. Adv. Comput. Math. 47, 1–20 (2021)

    Article  MathSciNet  Google Scholar 

  33. Liu, Y., Xu, Y., Yin, W.: Acceleration of primal-dual methods by preconditioning and simple subproblem procedures. J. Sci. Comput. 86, 1–34 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  34. Malitsky, Y., Pock, T.: A first-order primal-dual algorithm with linesearch. SIAM J. Optim. 28, 411–432 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  35. Mazurenko, S., Jauhiainen, J., Valkonen, T.: Primal-dual block-proximal splitting for a class of non-convex problems. Electron. Trans. Numer. Anal. 52, 509–552 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  36. Rasch, J., Chambolle, A.: Inexact first-order primal-dual algorithms. Comput. Optim. Appl. 76, 381–430 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  37. Robbins, H., Siegmund, D.: A convergence theorem for nonnegative almost supermartingales and some applications. Optimizing Methods in Statistics. In: Proc. Sympos., Ohio State Univ., Ohio, pp. 233–257 (1971)

  38. Rockafellar, R.T., Wets, R.J.: Variational Analysis. Springer, Berlin (1998)

    Book  MATH  Google Scholar 

  39. Uzawa, H.: Iterative methods for concave programming. Stud. Linear Nonlinear Program. (Arrow et al, eds) 6, 154–165 (1958)

    Google Scholar 

  40. Valkonen, T.: A primal-dual hybrid gradient method for nonlinear operators with applications to MRI. Inverse Problems 30, 45 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  41. Valkonen, T.: Block-proximal methods with spatially adapted acceleration. Electron. Trans. Numer. Anal. 51, 15–49 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  42. Valkonen, T.: First-order primal–dual methods for nonsmooth non-convex optimisation. In: Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging: Mathematical Imaging and Vision (Chen et al, eds) pp. 1–42 (2021)

  43. Valkonen, T., Pock, T.: Acceleration of the PDHGM on partially strongly convex functions. J. Math. Imaging Vision 59, 394–414 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  44. Vũ, B.C.: A splitting algorithm for dual monotone inclusions involving cocoercive operators. Adv. Comput. Math. 38, 667–681 (2013)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the anonymous referees for their valuable comments, which helped improving the manuscript substantially. This work is support by NSFC grant (11971003), and partially by Science and Technology Department of Sichuan Province (2021YFG0125) and ZYGX2019J090.

Author information

Authors and Affiliations

  1. School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu, China

    Ying Gao & Wenxing Zhang

Authors
  1. Ying Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenxing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wenxing Zhang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (zip 700 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, Y., Zhang, W. An alternative extrapolation scheme of PDHGM for saddle point problem with nonlinear function. Comput Optim Appl 85, 263–291 (2023). https://doi.org/10.1007/s10589-023-00453-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-023-00453-8

Keywords

Search

Navigation