Skip to main content
SpringerLink
Log in

A Peaceman–Rachford Splitting Method with Monotone Plus Skew-Symmetric Splitting for Nonlinear Saddle Point Problems

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

This paper is devoted to solving the linearly constrained convex optimization problems by Peaceman–Rachford splitting method with monotone plus skew-symmetric splitting on KKT operators. This approach generalizes the Hermitian and skew-Hermitian splitting method, an unconditionally convergent algorithm for non-Hermitian positive definite linear systems, to the nonlinear scenario. The convergence of the proposed algorithm is guaranteed under some mild assumptions, e.g., the strict convexity on objective functions and the consistency on constraints, even though the Lions–Mercier property is not fulfilled. In addition, we explore an inexact version of the proposed algorithm, which allows solving the subproblems approximately with some inexactness criteria. Numerical simulations on an image restoration problem demonstrate the compelling performance of the proposed algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. The PRSM in [21, 37] extended the range of applicability of classical PRSM for positive definite linear system to maximal monotone nonlinear system. In this sense, the recursion (1.5) with \(H\) and \(S\) being positive semi-definite or skew-symmetric matrices can also be deemed as PRSM because those matrices are essentially monotone operators. Henceforth, the terminologies “PRSM” and “DRSM” are named after the literature [21, 37].

  2. A function \(\theta :\mathbb {R}^n\rightarrow (-\infty ,+\infty ]\) is proper if its domain \(\hbox {dom}(\theta ):=\{x\in \mathbb {R}^n\mid \theta (x)<+\infty \}\) is nonempty, and it is said to be closed if its epigraph \(\hbox {epi}(\theta ):=\{(x,y)\in \mathbb {R}^n\times \mathbb {R}\mid \theta (x)\le y\}\) is closed.

  3. The \(\mu \)-strongly monotonicity and L-Lipschitz continuity on \(\nabla \theta \) indicate that the function \(\theta \) satisfies \(\frac{\mu }{2}\Vert x-y\Vert _2^2\le \theta (y)-\theta (x)-\nabla \theta (x)^\top (y-x)\le \frac{L}{2}\Vert x-y\Vert _2^2\) for all \(x\in \mathbb {R}^n\), \(y\in \mathbb {R}^n\).

  4. The strictly monotonicity and L-Lipschitz continuity on \(\nabla \theta \) indicate that the function \(\theta \) satisfies \(0<\theta (y)-\theta (x)-(y-x)^\top \nabla \theta \le \frac{L}{2}\Vert x-y\Vert _2^2\) for all \(x\in \mathbb {R}^n\), \(y\in \mathbb {R}^n\) and \(x\ne y\).

References

  1. Bai, Z.-Z., Golub, G.H., Ng, M.K.: Hermitian and skew-Hermitian splitting methods for non-Hermitian positive definite linear systems. SIAM J. Matrix Anal. Appl. 24, 603–626 (2003)

    MathSciNet  MATH  Google Scholar 

  2. Bai, Z.-Z., Golub, G.H., Ng, M.K.: On successive-overrelaxation acceleration of the hermitian and skew-hermitian splitting iterations. Numer. Linear Algebra Appl. 19, 319–335 (2012)

    MathSciNet  MATH  Google Scholar 

  3. Bai, Z.-Z., Golub, G.H., Pan, J.-Y.: Preconditioned Hermitian and skew-Hermitian splitting methods for non-Hermitian positive semidefinite linear systems. Numer. Math. 98, 1–32 (2004)

    MathSciNet  MATH  Google Scholar 

  4. Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, Berlin (2017)

    MATH  Google Scholar 

  5. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2, 183–202 (2009)

    MathSciNet  MATH  Google Scholar 

  6. Benzi, M.: A generalization of the Hermitian and skew-Hermitian splitting iteration. SIAM J. Matrix Anal. Appl. 31, 360–374 (2009)

    MathSciNet  MATH  Google Scholar 

  7. Benzi, M., Golub, G.H.: A preconditioner for generalized saddle point problems. SIAM J. Matrix Anal. Appl. 26, 20–41 (2004)

    MathSciNet  MATH  Google Scholar 

  8. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends. Mach. Learn. 3, 1–122 (2011)

    MATH  Google Scholar 

  9. Arias, L.M.B., Combettes, P.L.: A monotone + skew splitting model for composite monotone inclusions in duality. SIAM J. Optim. 21, 1230–1250 (2011)

    MathSciNet  MATH  Google Scholar 

  10. Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51, 34–81 (2009)

    MathSciNet  MATH  Google Scholar 

  11. Cai, X., Han, D., Yuan, X.: On the convergence of the direct extension of admm for three-block separable convex minimization models with one strongly convex function. Comput. Optim. Appl. 66, 1–35 (2017)

    MathSciNet  MATH  Google Scholar 

  12. Candès, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math. 9, 717–772 (2009)

    MathSciNet  MATH  Google Scholar 

  13. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40, 120–145 (2011)

    MathSciNet  MATH  Google Scholar 

  14. Chan, T.F., Shen, J.: Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods. SIAM, Philadelphia, PA (2005)

    MATH  Google Scholar 

  15. Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Math. Program. 155, 57–79 (2016)

    MathSciNet  MATH  Google Scholar 

  16. Chen, C., He, B., Yuan, X.: Matrix completion via an alternating direction method. IMA J. Numer. Anal. 32, 227–245 (2012)

    MathSciNet  MATH  Google Scholar 

  17. Combettes, P.L., Vũ, B.C.: Variable metric forward–backward splitting with applications to monotone inclusions in duality. Optimization 63, 1289–1318 (2014)

    MathSciNet  MATH  Google Scholar 

  18. Condat, L.: A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms. J. Optim. Theory Appl. 158, 460–479 (2013)

    MathSciNet  MATH  Google Scholar 

  19. Dai, Y.-H., Han, D., Yuan, X., Zhang, W.: A sequential updating scheme of the lagrange multiplier for separable convex programming. Math. Comput. 86, 315–343 (2017)

    MathSciNet  MATH  Google Scholar 

  20. Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57, 1413–1457 (2004)

    MathSciNet  MATH  Google Scholar 

  21. Eckstein, J., Bertsekas, D.P.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)

    MathSciNet  MATH  Google Scholar 

  22. Fazel, M., Pong, T.K., Sun, D., Tseng, P.: Hankel matrix rank minimization with applications to system identification and realization. SIAM J. Matrix Anal. Appl. 34, 946–977 (2013)

    MathSciNet  MATH  Google Scholar 

  23. Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2, 17–40 (1976)

    MATH  Google Scholar 

  24. Giselsson, P., Boyd, S.: Linear convergence and metric selection for Douglas-Rachford splitting and ADMM. IEEE Trans. Automat. Control 62, 532–544 (2017)

    MathSciNet  MATH  Google Scholar 

  25. Glowinski, R.: On alternating direction methods of multipliers: a historical perspective, in modeling, simulation and optimization for science and technology. Comput. Methods Appl. Sci. 34, 59–82 (2014)

    MATH  Google Scholar 

  26. Glowinski, R., Marrocco, A.: Sur l’approximation paréléments finis d’ordre un et la résolution parpénalisation-dualité d’une classe de problèmes de dirichlet non linéaires. Revue Fr. Autom. Inform. Rech. Oper., Anal. Numer. 2, 41–76 (1975)

    Google Scholar 

  27. Guo, K., Han, D., Yuan, X.: Convergence analysis of Douglas-Rachford splitting method for “strongly + weakly” convex programming. SIAM J. Numer. Anal. 55, 1549–1577 (2017)

    MathSciNet  MATH  Google Scholar 

  28. Han, D., Yuan, X.: Convergence analysis of the Peaceman-Rachford splitting method for nonsmooth convex optimization. Optim. Online 1–19 (2012)

  29. Hansen, P.C., Nagy, J.G., O’Leary, D.P.: Deblurring Images: Matrices, Spectra, and Filtering. SIAM, Philadelphia, PA (2006)

    MATH  Google Scholar 

  30. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)

    MATH  Google Scholar 

  31. He, B., Liao, L.: Improvements of some projection methods for monotone nonlinear variational inequalities. J. Optim. Theory Appl. 112, 111–128 (2002)

    MathSciNet  MATH  Google Scholar 

  32. He, B., You, Y., Yuan, X.: On the convergence of primal-dual hybrid gradient algorithm. SIAM J. Imaging Sci. 7, 2526–2537 (2014)

    MathSciNet  MATH  Google Scholar 

  33. He, B., Yuan, X.: Convergence analysis of primal-dual algorithms for a saddle-point problem: from contraction perspective. SIAM J. Imaging Sci. 5, 119–149 (2012)

    MathSciNet  MATH  Google Scholar 

  34. Huang, Y., Ng, M.K., Wen, Y.: Fast image restoration methods for impulse and Gaussian noises removal. IEEE Signal Process. Lett. 16, 457–460 (2009)

    Google Scholar 

  35. Liang, J., Fadili, J., Peyré, G.: Local linear convergence analysis of primal-dual splitting methods. Optimization 67, 821–853 (2018)

    MathSciNet  MATH  Google Scholar 

  36. Liao, H., Li, F., Ng, M.K.: Selection of regularization parameter in total variation image restoration. J. Opt. Soc. Am. A 26, 2311 (2009)

    MathSciNet  Google Scholar 

  37. Lions, P.-L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979)

    MathSciNet  MATH  Google Scholar 

  38. Liu, L., Han, Z.: Multi-block ADMM for big data optimization in smart grid. In: International Conference on Computing, IEEE, Networking and Communications. pp. 556–561 (2015)

  39. Markowitz, H.: Portfolio selection. J. Finance 7, 77–91 (1952)

    Google Scholar 

  40. Ng, M.K., Yuan, X., Zhang, W.: Coupled variational image decomposition and restoration model for blurred cartoon-plus-texture images with missing pixels. IEEE Trans. Image Process. 22, 2233–2246 (2013)

    MathSciNet  MATH  Google Scholar 

  41. Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (2006)

    MATH  Google Scholar 

  42. O’Connor, D., Vandenberghe, L.: Primal-dual decomposition by operator splitting and applications to image deblurring. SIAM J. Imaging Sci. 7, 1724–1754 (2014)

    MathSciNet  MATH  Google Scholar 

  43. Patriksson, M.: A survey on the continuous nonlinear resource allocation problem. Eur. J. Oper. Res. 185, 1–46 (2008)

    MathSciNet  MATH  Google Scholar 

  44. Robbins, H., Siegmund, D.: A convergence theorem for nonnegative almost supermartingales and some applications. In: Optimizing Methods in Statistics (Proceedings of Symposium, Ohio State Univ., Ohio), pp. 233–257 (1971)

    Google Scholar 

  45. Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton, NJ (1970)

    MATH  Google Scholar 

  46. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60, 259–268 (1992)

    MathSciNet  MATH  Google Scholar 

  47. Ruszczyński, A.: Parallel decomposition of multistage stochastic programming problems. Math. Program. 58, 201–228 (1993)

    MathSciNet  MATH  Google Scholar 

  48. Ryu, E.K., Boyd, S.: A primer on monotone operator methods (survey). Appl. Comput. Math. 15, 3–43 (2016)

    MathSciNet  MATH  Google Scholar 

  49. Ryu, E.K., Taylor, A.B., Bergeling, C., Giselsson, P.: Operator splitting performance estimation: tight contraction factors and optimal parameter selection. arXiv:1812.00146 (2018)

  50. Simoncini, V., Benzi, M.: Spectral properties of the Hermitian and skew-Hermitian splitting preconditioner for saddle point problems. SIAM J. Matrix Anal. Appl. 26, 377–389 (2004)

    MathSciNet  MATH  Google Scholar 

  51. Tao, M., Yuan, X.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)

    MathSciNet  MATH  Google Scholar 

  52. Varga, R.S.: Matrix Iterative Analysis. Springer, Berlin (2000)

    MATH  Google Scholar 

  53. Wen, Y.-W., Chan, R.H.: Parameter selection for total-variation-based image restoration using discrepancy principle. IEEE Trans. Image Process. 21, 1770–1781 (2012)

    MathSciNet  MATH  Google Scholar 

  54. Xu, H., Feng, C., Li, B.: Temperature aware workload managementin geo-distributed data centers. IEEE Trans. Parallel Distrib. Syst. 26, 1743–1753 (2015)

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the anonymous referees for their valuable comments. W.Y. Ding is partially supported by NSFC Grants 11801479, HKRGC GRF 12301619 and HKBU RC-NACAD-DW. M.K. Ng is partially supported by HKRGC GRF 12306616, 12200317, 12300218 and 12300519. W.X. Zhang is partially supported by NSFC Grant 11571074.

Author information

Authors and Affiliations

  1. Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong, China

    Weiyang Ding

  2. Department of Mathematics, The University of Hong Kong, Pokfulam, Hong Kong, China

    Michael K. Ng

  3. School of Mathematical Sciences, University of Electronic Science and Technology of China, Chengdu, China

    Wenxing Zhang

Authors
  1. Weiyang Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael K. Ng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenxing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wenxing Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ding, W., Ng, M.K. & Zhang, W. A Peaceman–Rachford Splitting Method with Monotone Plus Skew-Symmetric Splitting for Nonlinear Saddle Point Problems. J Sci Comput 81, 763–788 (2019). https://doi.org/10.1007/s10915-019-01034-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10915-019-01034-w

Keywords

Search

Navigation