Skip to main content
SpringerLink
Log in

A partially isochronous splitting algorithm for three-block separable convex minimization problems

  • Published:
Advances in Computational Mathematics Aims and scope Submit manuscript

A Correction to this article was published on 20 February 2018

This article has been updated

Abstract

During the last decade, the state-of-the-art alternating direction method of multipliers (ADMM) has successfully been used to solve many two-block separable convex minimization problems arising from several applied areas such as signal/image processing and statistical and machine learning. It however remains an interesting problem of how to implement ADMM to three-block separable convex minimization problems as required by the situation where many objective functions in the above-mentioned areas are actually more conveniently decomposed to the sum of three convex functions, due also to the observation that the straightforward extension of ADMM from the two-block case to the three-block case is apparently not convergent. In this paper, we shall introduce a new algorithm that is called a partially isochronous splitting algorithm (PISA) in order to implement ADMM for the three-block separable model. The main idea of our algorithm is to incorporate only one proximal term into the last subproblem of the extended ADMM so that the resulting algorithm maximally inherits the promising properties of ADMM. A remarkable superiority over the extended ADMM is that we can simultaneously solve two of the subproblems, thereby taking advantages of the separable structure and parallel architectures. Theoretically, we will establish the global convergence of our algorithm under standard conditions, and also the O(1/t) rate of convergence in both ergodic and nonergodic senses, where t is the iteration counter. The computational competitiveness of our algorithm is shown by numerical experiments on an application to the well-tested robust principal component analysis model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Change history

  • 20 February 2018

    The original publication of this article has an error.

References

  1. Bauschke, H., Combettes, P.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, New York (2011)

    Book  MATH  Google Scholar 

  2. Bertsekas, D., Tsitsiklis, J.: Parallel and Distributed Computation, Numerical Methods. Prentice-Hall, Englewood Cliffs (1989)

    MATH  Google Scholar 

  3. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1–122 (2010)

    Article  MATH  Google Scholar 

  4. Cai, J., Candés, E., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20, 1956–1982 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  5. Cai, J., Chan, R., Nikolova, M.: Two-phase approach for deblurring images corrupted by impulse plus gaussian noise. Inverse Prob. Imag. 2, 187–204 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  6. Cai, X., Han, D., Yuan, X.: On the convergence of the direct extension of ADMM for three-block separable convex minimization models with one strongly convex function. Comput. Optim. Appl. 66, 39–73 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  7. Candės, E., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM 58, 1–37 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  8. Chandrasekaran, V., Parrilo, P., Willskyc, A.: Latent variable graphical model selection via convex optimization. Ann. Statist. 40, 1935–1967 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  9. Chandrasekaran, V., Sanghavi, S., Parrilo, P., Willskyc, A.: Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim. 21, 572–596 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  10. Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Math. Program. Ser. A 155, 57–79 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  11. Chen, L., Sun, D., Toh, K.C.: A note on the convergence of admm for linearly constrained convex optimization problems. Comput. Optim. Appl. 66(2), 327–343 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  12. Combettes, P., Wajs, V.: Signal recovery by proximal forward-backward splitting. Multiscale Model Simul. 4, 1168–1200 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  13. Eckstein, J.: Splitting methods for monotone operators with applications to parallel optimization. Ph.D. thesis Massachusetts Institute of Technology (1989)

  14. Eckstein, J.: Augmented Lagrangian and alternating direction methods for convex optimization: A tutorial and some illustrative computational results. Tech. Rep. 32–2012 Rutgers University (2012)

  15. Facchinei, F., Pang, J.: Finite-Dimensional Variational Inequalities and Complementarity Problems. Springer, New York (2003)

    MATH  Google Scholar 

  16. Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximations. Comput. Math. Appl. 2, 16–40 (1976)

    Article  MATH  Google Scholar 

  17. Glowinski, R.: On alternating direction methods of multipliers: a historical perspective. In: Fitzgibbon, W., Kuznetsov, Y.A., Neittaanmäki, P., Pironneau, O. (eds.) Modeling, Simulation and Optimization for Science and Technology, Computational Methods in Applied Sciences, chap. 4, vol. 34, pp. 59–82. Springer, New York (2014)

  18. Glowinski, R., Marrocco, A.: Approximation par éléments finis d’ordre un et résolution par pénalisation-dualité d’une classe de problèmes non linéaires. R.A.I.R.O. R2, 41–76 (1975)

    MATH  Google Scholar 

  19. Han, D., He, H., Yang, H., Yuan, X.: A customized Douglas-Rachford splitting algorithm for separable convex minimization with linear constraints. Numer. Math. 127, 167–200 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  20. Han, D., Yuan, X.: A note on the alternating direction method of multipliers. J. Optim. Theory Appl. 155(1), 227–238 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  21. Han, D., Yuan, X., Zhang, W.: An augmented-Lagrangian-based parallel splitting method for separable convex minimization with applications to image processing. Math. Comput. 83, 2263–2291 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  22. Han, D., Yuan, X., Zhang, W., Cai, X.: An ADM-based splitting method for separable convex programming. Comput. Optim. Appl. 54, 343–369 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  23. He, B.: Parallel splitting augmented Lagrangian methods for monotone structured variational inequalities. Comput. Optim. Appl. 42, 195–212 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  24. He, B., Hou, L., Yuan, X.: On full Jacobian decomposition of the augmented lagrangian method for separable convex programming. SIAM J. Optim. 25(4), 2274–2312 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  25. He, B., Tao, M., Xu, M., Yuan, X.: Alternating directions based contraction method for generally separable linearly constrained convex programming problems. Optimizaition 62, 573–596 (2013)

    Article  MATH  Google Scholar 

  26. He, B., Tao, M., Yuan, X.: Alternating direction method with Gaussian-back substitution for separable convex programming. SIAM J. Optim. 22, 313–340 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  27. He, B., Tao, M., Yuan, X.: A splitting method for separate convex programming. IMA J. Numer. Anal. 35, 394–426 (2014)

    Article  MATH  Google Scholar 

  28. He, B., Xu, H., Yuan, X.: On the proximal Jacobian decomposition of ALM for multiple-block separable convex minimization problems and its relationship to ADMM. J. Sci. Comput. 66, 1204–1217 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  29. He, B., Yuan, X.: On the O(1/n) convergence rate of Douglas-Rachford alternating direction method. SIAM J. Numer. Anal. 50, 700–709 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  30. He, B., Yuan, X.: On the direct extension of ADMM for multi-block separable convex programming and beyond: From variational inequality perspective. http://www.optimization-online.org/DB_FILE/2014/03/4293.pdf (2014)

  31. He, H., Han, D.: A distributed Douglas-Rachford splitting method for multi-block convex minimization problems. Adv. Comput. Math. 42, 27–53 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  32. Hestenes, M.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)

    Article  MathSciNet  MATH  Google Scholar 

  33. Hong, M., Luo, Z.: On the linear convergence of the alternating direction method of multipliers. Math. Program. Ser. A 162, 165–199 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  34. Honorio, J., Samaras, D.: Multi-task learning of Gaussian graphical models. In: Proceedings of the 27th International Conference on Machine Learning, pp. 447–454. Haifa, Israel (2010)

  35. Larsen, R.: PROPACK-software for large and sparse SVD calculations. http://sun.stanford.edu/srmunk/PROPACK/

  36. Li, M., Sun, D., Toh, K.C.: A convergent 3-block semi-proximal ADMM for convex minimization problems with one strongly convex block. Asia-Pacific J. Oper. Res. 32(3), 1550024 (2015). (19 pages)

    Article  MathSciNet  MATH  Google Scholar 

  37. McLachlan, G.: Discriminant Analysis and Statistical Pattern Recoginition. Wiley, Hoboken (2004)

    MATH  Google Scholar 

  38. Moreau, J.: Proximitė et dualitė dans un espace Hilbertien. Bulletin de la Societe Mathematique de France 95, 153–171 (1965)

    MATH  Google Scholar 

  39. Nesterov, Y.: Introductory Lectures on Convex Optimization: Basic Course. Kluwer, Boston (2003)

    MATH  Google Scholar 

  40. Nesterov, Y., Nemirovskii, A.: Interior-Point Polynomial Algorithms in Convex Programming Studies in Applied and Numerical Mathematics, 2nd edn., vol. 13. SIAM, Philadelphia (1994)

    Book  MATH  Google Scholar 

  41. Nocedal, J., Wright, S.: Numerical Optimization, 2nd edn. Springer Series in Operations Research and Financial Engineering Springer-Verlag (2006)

  42. Powell, M.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp 283–298, London (1969)

  43. Rockafellar, R.: On the maximal monotonicity of subdifferential mappings. Pac. J. Math 33(1), 209–216 (1970)

    Article  MathSciNet  MATH  Google Scholar 

  44. Schaeffer, H., Osher, S.: A low patch-rank interpretation of texture. SIAM J. Imaging Sci. 6(1), 226–262 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  45. Tao, M., Yuan, X.: Recovering low-rank and sparse components of matrices from incomplete and noisy observations. SIAM J. Optim. 21, 57–81 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  46. Wang, X., Hong, M., Ma, S., Luo, Z.: Solving multiple-block separable convex minimization problems using two-block alternating direction method of multipliers. Pac. J. Optim. 11, 645–667 (2015)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the two anonymous referees for their careful reading of and critical comments on this manuscript, which helped us dramatically improve the presentation of the manuscript. The first author was partially supported by the National Natural Science Foundation of China (NSFC) (Nos. 11771113 and 11571087) and Natural Science Foundation of Zhejiang Province (No. LY17A010028). The second author was partially supported by NSFC Grant 11471156. The third author was supported in part by NSFC Grant 11571087.

Author information

Authors and Affiliations

  1. Department of Mathematics, School of Science, Hangzhou Dianzi University, Hangzhou, 310018, China

    Hongjin He & Hong-Kun Xu

  2. School of Information Engineering, Nanjing Xiaozhuang University, Nanjing, China

    Liusheng Hou

  3. Department of Mathematics, King Abdulaziz University, P. O. Box 80203, Jeddah, 21589, Saudi Arabia

    Hong-Kun Xu

Authors
  1. Hongjin He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liusheng Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hong-Kun Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hong-Kun Xu.

Additional information

Communicated by: Stefan Volkwein

The original version of this article was revised: In page 8 all instances of variable xi (\(\protect \mathbf {x}_{i}^{*}\)) were corrected and replaced by ξ (ξ∗).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, H., Hou, L. & Xu, HK. A partially isochronous splitting algorithm for three-block separable convex minimization problems. Adv Comput Math 44, 1091–1115 (2018). https://doi.org/10.1007/s10444-017-9574-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10444-017-9574-4

Keywords

Mathematics Subject Classification (2010)

Search

Navigation