Skip to main content
SpringerLink
Log in

Inexact alternating direction methods of multipliers for separable convex optimization

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

Inexact alternating direction multiplier methods (ADMMs) are developed for solving general separable convex optimization problems with a linear constraint and with an objective that is the sum of smooth and nonsmooth terms. The approach involves linearized subproblems, a back substitution step, and either gradient or accelerated gradient techniques. Global convergence is established. The methods are particularly useful when the ADMM subproblems do not have closed form solution or when the solution of the subproblems is expensive. Numerical experiments based on image reconstruction problems show the effectiveness of the proposed methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 2

Similar content being viewed by others

References

  1. Afonso, M., Bioucas-Dias, J., Figueiredo, M.: Fast image recovery using variable splitting and constrained optimization. IEEE Trans. Image Process. 19, 2345–2356 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  2. Barzilai, J., Borwein, J.M.: Two point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  3. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Mach. Learn. 3, 1–122 (2010)

    Article  MATH  Google Scholar 

  4. Cai, X., Han, D., Yuan, X.: On the convergence of the direct extension of ADMM for three-block separable convex minimization models with one strongly convex function. Comput. Optim. Appl. 66, 39–73 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  5. Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Math. Program. 155, 57–79 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  6. Chen, C., Li, M., Liu, X., Ye, Y.: On the convergence of multi-block alternating direction method of multipliers and block coordinate descent method (2015). arXiv:1508.00193

  7. Chen, C., Shen, Y., You, Y.: On the convergence analysis of the alternating direction method of multipliers with three blocks. Abstr. Appl. Anal. (2013). https://doi.org/10.1155/2013/183961

    MathSciNet  MATH  Google Scholar 

  8. Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Math. Program. 64, 81–101 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  9. Chen, L., Sun, D., Toh, K.: An efficient inexact symmetric Gauss–Seidel based majorized ADMM for high-dimensional convex composite conic programming. Math. Program. 161, 237–270 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  10. Chen, Y., Hager, W.W., Yashtini, M., Ye, X., Zhang, H.: Bregman operator splitting with variable stepsize for total variation image reconstruction. Comput. Optim. Appl. 54, 317–342 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  11. Davis, D., Yin, W.: A three-operator splitting scheme and its optimization applications. Set Valued Var. Anal. 25, 829–858 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  12. Eckstein, J., Bertsekas, D.: On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Program. 55, 293–318 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  13. Eckstein, J., Silva, P.J.S.: A practical relative error criterion for augmented Lagrangians. Math. Program. 141, 319–348 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  14. Eckstein, J., Yao, W.: Approximate ADMM algorithms derived from Lagrangian splitting. Comput. Optim. Appl. 68, 363–405 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  15. Eckstein, J., Yao, W.: Relative-error approximate versions of Douglas–Rachford splitting and special cases of the ADMM. Math. Program. 170, 417–444 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  16. Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite-element approximations. Comput. Math. Appl. 2, 17–40 (1976)

    Article  MATH  Google Scholar 

  17. Glowinski, R.: Numerical Methods for Nonlinear Variational Problems. Springer, New York (1984)

    Book  MATH  Google Scholar 

  18. Goldfarb, D., Ma, S.: Fast multiple-splitting algorithms for convex optimization. SIAM J. Optim. 22, 533–556 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  19. Golshtein, E., Tretyakov, N.: Modified Lagrangians in convex programming and their generalizations, in point-to-set maps and mathematical programming. In: Huard, P. (ed.) Mathematical Programming Studies, vol. 10, pp. 86–97. Springer, Berlin (1979)

    Google Scholar 

  20. Hager, W.W., Yashtini, M., Zhang, H.: An O(1/k) convergence rate for the variable stepsize Bregman operator splitting algorithm. SIAM J. Numer. Anal. 54, 1535–1556 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  21. Hager, W.W., Zhang, H.: Projection onto a polyhedron that exploits sparsity. SIAM J. Optim. 29, 1773–1798 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  22. Han, D., Yuan, X.: A note on the alternating direction method of multipliers. J. Optim. Theory Appl. 155, 227–238 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  23. He, B., Liao, L., Han, D., Yan, H.: A new inexact alternating directions method for monotone variational inequalities. Math. Program. 92, 103–118 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  24. He, B., Tao, M., Xu, M., Yuan, X.: An alternating direction-based contraction method for linearly constrained separable convex programming problems. Optimization 62, 573–596 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  25. He, B., Tao, M., Yuan, X.: Alternating direction method with Gaussian back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  26. He, B., Yuan, X.: On the \({\cal{O}}(1/n)\) convergence rate of the Douglas–Rachford alternating direction method. SIAM J. Numer. Anal. 50, 700–709 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  27. Hong, M., Luo, Z.-Q.: On the linear convergence of the alternating direction method of multipliers. Math. Program. 162, 165–199 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  28. Li, J., Chen, G., Dong, Z., Wu, Z.: A fast dual proximal-gradient method for separable convex optimization with linear coupled constraints. Comput. Optim. Appl. 64, 671–697 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  29. Li, M., Liao, L., Yuan, X.: Inexact alternating direction methods of multipliers with logarithmic–quadratic proximal regularization. J. Optim. Theory Appl. 159, 412–436 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  30. Li, M., Sun, D., Toh, K.C.: A convergent 3-block semi-proximal ADMM for for convex minimization problems with one strongly convex block. Asia Pac. J. Oper. Res. 32, 1–19 (2015)

    MathSciNet  MATH  Google Scholar 

  31. Lin, T., Ma, S., Zhang, S.: On the global linear convergence of the ADMM with multiblock variables. SIAM J. Optim. 25, 1478–1497 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  32. Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)

    Book  MATH  Google Scholar 

  33. Solodov, M.V., Svaiter, B.F.: An inexact hybrid generalized proximal point algorithm and some new results on the theory of Bregman functions. Math. Oper. Res. 25, 214–230 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  34. Tao, M., Yuan, X.M.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  35. Wen, Z., Goldfarb, D., Yin, W.: Alternating direction augmented Lagrangian methods for semidefinite programming. Math. Program. Comput. 2, 203–230 (2010)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, University of Florida, PO Box 118105, Gainesville, FL, 32611-8105, USA

    William W. Hager

  2. Department of Mathematics, Louisiana State University, Baton Rouge, LA, 70803-4918, USA

    Hongchao Zhang

Authors
  1. William W. Hager
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongchao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to William W. Hager.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

December 19, 2017. The authors gratefully acknowledge support by the National Science Foundation under Grants 1522629, 1522654, 1819002, and 1819161, and by the Office of Naval Research under Grants N00014-15-1-2048 and N00014-18-1-2100.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hager, W.W., Zhang, H. Inexact alternating direction methods of multipliers for separable convex optimization. Comput Optim Appl 73, 201–235 (2019). https://doi.org/10.1007/s10589-019-00072-2

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-019-00072-2

Keywords

Mathematics Subject Classification

Search

Navigation