Skip to main content
SpringerLink
Log in

A customized Douglas–Rachford splitting algorithm for separable convex minimization with linear constraints

  • Published:
Numerische Mathematik Aims and scope Submit manuscript

Abstract

We consider applying the Douglas–Rachford splitting method (DRSM) to the convex minimization problem with linear constraints and a separable objective function. The dual application of DRSM has been well studied in the literature, resulting in the well known alternating direction method of multipliers (ADMM). In this paper, we show that the primal application of DRSM in combination with an appropriate decomposition can yield an efficient structure-exploiting algorithm for the model under consideration, whose subproblems could be easier than those of ADMM. Both the exact and inexact versions of this customized DRSM are studied; and their numerical efficiency is demonstrated by some preliminary numerical results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)

    Article  MATH  MathSciNet  Google Scholar 

  2. Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, New York (2011)

    Book  MATH  Google Scholar 

  3. Bertsekas, D.P., Tsitsiklis, J.N.: Parallel and Distributed Computation. Numerical Methods. Prentice-Hall, Englewood Cliffs (1989)

    MATH  Google Scholar 

  4. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1–122 (2010)

    Article  MATH  Google Scholar 

  5. Cai, J.F., Osher, S., Shen, Z.W.: Linearized Bregman iterations for compressed sensing. Math. Comput. 78, 1515–1536 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  6. Cai, J.F., Osher, S., Shen, Z.W.: Linearized Bregman iterations for frame-based image deblurring. SIAM J. Imaging Sci. 2, 226–252 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  7. Candès, E.J., Li, X.D., Ma, Y., Wright, J.: Robust principal component analysis? J. Assoc. Comput. Mach. 58, 1–37 (2011)

    Article  Google Scholar 

  8. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40, 120–145 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  9. Chan, R.H., Yang, J.F., Yuan, X.M.: Alternating direction method for image inpainting in wavelet domain. SIAM J. Imaging Sci. 4, 807–826 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  10. Chan, T.F., Glowinski, R.: Finite element approximation and iterative solution of a class of mildly non-linear elliptic equations. STAN-CS Report 78–674, Computer Science Department, Stanford University (1978)

  11. Chandrasekaran, V., Sanghavi, S., Parrilo, P.A., Willskyc, A.S.: Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim. 21, 572–596 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  12. Chen, C.H., He, B.S., Yuan, X.M.: Matrix completion via alternating direction method. IMA J. Numer. Anal. 32, 227–245 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  13. Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Math. Program. 64, 81–101 (1994)

    Article  MATH  MathSciNet  Google Scholar 

  14. Dai, Y.H., Fletcher, R.: Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming. Numerische Mathematik 100, 21–47 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  15. Donoho, D.: Compressed sensing. IEEE Trans. Inf. Theory 52, 1289–1306 (2006)

    Article  MathSciNet  Google Scholar 

  16. Douglas, J., Rachford, H.H.: On the numerical solution of the heat conduction problem in 2 and 3 space variables. Trans. Am. Math. Soc. 82, 421–439 (1956)

    Article  MATH  MathSciNet  Google Scholar 

  17. Eaves, B.C.: On the basic theorem of complementarity. Math. Program. 1, 68–75 (1971)

    Article  MATH  MathSciNet  Google Scholar 

  18. Eckstein, J., Bertsekas, D.P.: On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)

    Article  MATH  MathSciNet  Google Scholar 

  19. Esser, E.: Applications of Lagrangian-based alternating direction methods and connections to split Bregman. CAM Report 09–31, UCLA (2009)

  20. Facchinei, F., Pang, J.S.: Finite-Dimensional Variational Inequalities and Complementarity Problems. Springer, New York (2003)

    Google Scholar 

  21. Fortin, M., Glowinski, R.: Augmented Lagrangian Methods: Applications to the Numerical Solutions of Boundary Value Problems. Studies in Mathematics and Its Applications. Northolland, Amsterdam (1983)

    Google Scholar 

  22. Fukushima, M.: Application of the alternating direction method of multipliers to separable convex programming problems. Comput. Optim. Appl. 2, 93–111 (1992)

    Article  MathSciNet  Google Scholar 

  23. Fukushima, M.: The primal Douglas–Rachford splitting algorithm for a class of monotone mappings with application to the traffic equilibrium problem. Math. Program. 72, 1–15 (1996)

    MATH  MathSciNet  Google Scholar 

  24. Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Numerical Solution of Boundary-Value Problems, pp. 299–331. North-Holland, Amsterdam (1983)

  25. Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximations. Comput. Math. Appl. 2, 16–40 (1976)

    Article  Google Scholar 

  26. Glowinski, R.: Numerical Methods for Nonlinear Variational Problems. Springer, New York (1984)

    Book  MATH  Google Scholar 

  27. Glowinski, R., Kärkkäinen, T., Majava, K.: On the convergence of operator-splitting methods. In: Kuznetsov, Y., Neittanmaki, P., Pironneau, O. (eds.) Numerical Methods for Scientific Computing. Variational Problems and Applications, Barcelona (2003)

  28. Glowinski, R., Le Tallec, P.: Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics. SIAM Studies in Applied Mathematics. SIAM, Philadelphia (1989)

    Book  Google Scholar 

  29. Glowinski, R., Marrocco, A.: Approximation par \(\acute{e}\)l\(\acute{e}\)ments finis d’ordre un et r\(\acute{e}\)solution par p\(\acute{e}\)nalisation-dualit\(\acute{e}\) d’une classe de probl\(\grave{e}\)mes non lin\(\acute{e}\)aires. R.A.I.R.O., R2:41–76 (1975)

  30. Goldstein, T., Osher, S.: The split Bregman method for \(\ell _1\) regularized problems. SIAM J. Imaging Sci. 2, 323–343 (2008)

    Article  MathSciNet  Google Scholar 

  31. Golub, G.H., von Matt, U.: Quadratically constrained least squares and quadratic problems. Numerische Mathematik 59, 561–580 (1990)

    Article  Google Scholar 

  32. Han, D.R., Xu, W., Yang, H.: An operator splitting method for variational inequalities with partially unknown mappings. Numerische Mathematik 111, 207–237 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  33. Hansen, P.C., Nagy, J.G., O’Leary, D.P.: Deblurring Images: Matrices, Spectra, and Filtering. SIAM, Philadelphia (2006)

    Book  Google Scholar 

  34. He, B.S., Liao, L.Z.: Improvements of some projection methods for monotone nonlinear variational inequalities. J. Optim. Theory Appl. 112, 111–128 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  35. He, B.S., Liao, L.Z., Han, D.R., Yang, H.: A new inexact alternating direction method for monotone variational inequalities. Math. Program. 92, 103–118 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  36. He, B.S., Liao, L.Z., Wang, S.L.: Self-adaptive operator splitting methods for monotone variational inequalities. Numerische Mathematik 94, 715–737 (2003)

    MATH  MathSciNet  Google Scholar 

  37. He, B.S., Xu, M.H., Yuan, X.M.: Solving large-scale least squares covariance matrix problems by alternating direction methods. SIAM J. Matrix Anal. Appl. 32, 136–152 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  38. He, B.S., Yang, H., Wang, S.L.: Alternating direction method with self-adaptive penalty parameters for monotone variational inequalities. J. Optim. Theory Appl. 106, 337–356 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  39. He, B.S., Yuan, X.M.: On the \(O(1/n)\) convergence rate of Douglas–Rachford alternating direction method. SIAM J. Numer. Anal. 50, 700–709 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  40. Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)

    Article  MATH  MathSciNet  Google Scholar 

  41. Kontogiorgis, S., Meyer, R.R.: A variable-penalty alternating directions method for convex optimization. Math. Program. 83, 29–53 (1998)

    MATH  MathSciNet  Google Scholar 

  42. Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979)

    Article  MATH  MathSciNet  Google Scholar 

  43. Morini, S., Porcelli, M., Chan, R.H.: A reduced Newton method for constrained linear least squares problems. J. Comput. Appl. Math. 233, 2200–2212 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  44. Ng, M., Wang, F., Yuan, X.M.: Inexact alternating direction methods for image recovery. SIAM J. Sci. Comput. 33, 1643–1668 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  45. Ng, M., Weiss, P.A., Yuan, X.M.: Solving constrained total-variation problems via alternating direction methods. SIAM J. Sci. Comput. 32, 2710–2736 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  46. Peng, Y.G., Ganesh, A., Wright, J., Xu, W.L., Ma, Y.: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Trans. Pattern Anal. Mach. Intell. 34, 2233–2246 (2012)

    Article  Google Scholar 

  47. Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, New York (1969)

  48. Pratt, W.K.: Digital Image Processing: PIKS Inside, 3rd edn. Wiley, New York (2001)

  49. Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60, 227–238 (1992)

    Article  Google Scholar 

  50. Sun, J., Zhang, S.: A modified alternating direction method for convex quadratically constrained quadratic semidefinite programs. Eur. J. Oper. Res. 207, 1210–1220 (2010)

    Article  MATH  Google Scholar 

  51. Tao, M., Yuan, X.M.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  52. Tikhonov, A.N., Arsenin, V.Y.: Solutions of Ill-Posed Problems. Winston, New York (1977)

    MATH  Google Scholar 

  53. Tseng, P.: Applications of a splitting algorithm to decomposition in convex programming and variational inequalities. SIAM J. Control Optim. 29, 119–138 (1991)

    Article  MATH  MathSciNet  Google Scholar 

  54. Varga, R.S.: Matrix Iterative Analysis. Prentice-Hall, Englewood Cliffs (1966)

  55. Wen, Z.W., Goldfarb, D., Yin, W.T.: Alternating direction augmented Lagrangian methods for semidefinite programming. Math. Program. Comput. 2, 203–230 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  56. Yang, J.F., Zhang, Y.: Alternating direction algorithms for \(\ell _1\)-problems in compressive sensing. SIAM J. Sci. Comput. 332, 250–278 (2011)

    Article  Google Scholar 

  57. Yuan, X.M.: Alternating direction methods for covariance selection models. J. Sci. Comput. 51, 261–273 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  58. Yuan, X.M., Yang, J.F.: Sparse and low rank sparse matrix decomposition via alternating directions method. Pac. J. Optim. 9(1), 167–180 (2013)

    MATH  Google Scholar 

  59. Zhang, S., Ang, J., Sun, J.: An alternating direction method for solving convex nonlinear semidefinite programming problem. Optimization 62(4), 527–543 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  60. Zhang, X.Q., Burger, M., Bresson, X., Osher, S.: Bregmanized nonlocal regularization for deconvolution and sparse reconstruction. SIAM J. Imaging Sci. 3, 253–276 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  61. Zhang, X.Q., Burger, M., Osher, S.: A unified primal-dual algorithm framework based on Bregman iteration. J. Sci. Comput. 46, 20–46 (2011)

    Article  MATH  MathSciNet  Google Scholar 

Download references

Acknowledgments

The authors are grateful to two anonymous referees for their valuable comments on earlier versions of this paper; especially for one referee bringing our attention to the relevant references [13, 42]. This work was also partially supported by Institute of Computational and Theoretical Studies of Hong Kong Baptist University while the first author was a visiting research fellow of this institute.

Author information

Authors and Affiliations

  1. School of Mathematical Sciences, Jiangsu Key Labratory for NSLSCS, Nanjing Normal University, Nanjing, 210023, China

    Deren Han

  2. School of Science, Hangzhou Dianzi University, Hangzhou, 310018, China

    Hongjin He

  3. Department of Civil Engineering, The Hong Kong University of Science and Technology, Hong Kong, China

    Hai Yang

  4. Department of Mathematics, Institute of Computational and Theoretical Studies, Hong Kong Baptist University, Hong Kong, China

    Xiaoming Yuan

Authors
  1. Deren Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongjin He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hai Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoming Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoming Yuan.

Additional information

D. Han was supported by the National Natural Science Foundation of China No. 11071122. H. He was supported by the Research Foundation of Hangzhou Dianzi University at Grant No. KYS075612037. X. Yuan was supported by the General Research Fund from Hong Kong Research Grants Council: 203712.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Han, D., He, H., Yang, H. et al. A customized Douglas–Rachford splitting algorithm for separable convex minimization with linear constraints. Numer. Math. 127, 167–200 (2014). https://doi.org/10.1007/s00211-013-0580-2

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00211-013-0580-2

Mathematics Subject Classification

Search

Navigation