Abstract
We present an alternating forward–backward splitting method for solving linearly constrained structured optimization problems. The algorithm takes advantage of the separable structure and possibly asymmetric regularity properties of the objective functions involved. We also describe some applications to the study of non-Newtonian fluids and image reconstruction problems. We conclude with a numerical example, and its comparison with Condat’s algorithm. An acceleration heuristic is also briefly outlined.
Similar content being viewed by others
References
Acker, F., Prestel, M.A.: Convergence d’un schéma de minimisation alternée. Ann. Faculté des Sci. Toulouse V Sér. Math. 2, 1–9 (1980)
Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Alternating proximal algorithms for weakly coupled convex minimization problems. Applications to dynamical games and PDE’s. J. Convex Anal. 15(3), 485–506 (2008)
Attouch, H., Brézis, H.: Duality for the sum of convex functions in general Banach spaces. Asp. Math. Appl. J. Barroso Ed. 34, 125–133 (1986)
Attouch, H., Cabot, A., Frankel, P., Peypouquet, J.: Alternating proximal algorithms for constrained variational inequalities. Application to domain decomposition for PDE’s. Nonlinear Anal. 74(18), 7455–7473 (2011)
Attouch, H., Czarnecki, M.-O., Peypouquet, J.: Prox-penalization and splitting methods for constrained variational problems. SIAM J. Optim. 21, 149–173 (2011)
Attouch, H., Czarnecki, M.-O., Peypouquet, J.: Coupling forward–backward with penalty schemes and parallel splitting for constrained variational inequalities. SIAM J. Optim. 21(4), 1251–1274 (2011)
Attouch, H., Peypouquet, J.: The rate of convergence of Nesterov’s accelerated forward–backward method is actually faster than 1/k2. SIAM J. Optim. 26(3), 1824–1834 (2016)
Attouch, H., Redont, P., Soubeyran, A.: A new class of alternating proximal minimization algorithms with costs-to-move. SIAM J. Optim. 18, 1061–1081 (2007)
Auslender, A., Crouzeix, J.P., Fedit, P.: Penalty-proximal methods in convex programming. J. Optim. Theory Appl. 55, 1–21 (1987)
Bauschke, H.H., Combettes, P.L., Reich, S.: The asymptotic behavior of the composition of two resolvents. Nonlinear Anal. 60, 283–301 (2005)
Beck, A., Teboulle, M.: Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Process. 18(11), 2419–2434 (2009)
Bildhauer, M., Fuchs, M.: A variational approach to the denoising of images based on different variants of the TV-regularization. Appl. Math. Optim. 66, 331–361 (2012)
Bertels, S.: Total variation minimization with finite elements: convergence and iterative solution. SIAM J. Numer. Anal. 50(3), 1162–1180 (2012)
Briceño-Arias, L.: Forward-Douglas–Rachford splitting and forward-partial inverse method for solving monotone inclusions. Optimization 64(5), 1239–1261 (2015)
Briceño-Arias, L., López, S.: A projected primal-dual splitting for solving constrained monotone inclusions. arXiv:1805.11687v1 [math.OC]
Cabot, A.: Proximal point algorithm controlled by a slowly vanishing term: applications to hierarchical minimization. SIAM J. Optim. 15(2), 555–572 (2005)
Candès, E.J., Eldar, Y.C., Needell, D., Randall, P.: Compressed sensing with coherent and redundant dictionaries. Appl. Comput. Harmon. Anal. 31(1), 59–73 (2011)
Carter, J.L.: Dual methods for total variation-based image restoration. Ph.D. Thesis, UCLA (2001)
Cauchy, A.L.: Méthode générale pour la résolution des systèmes d’équations simultanées. C. R. Acad. Sci. Paris 25, 536–538 (1847)
Chambolle, A., Dossal, C.: On the convergence of the iterates of the “fast iterative shrinkage/thresholding algorithm”. J. Optim. Theory Appl. 166(3), 968–982 (2015)
Chambolle, A., Lions, P.L.: Image recovery via total variation minimization and related problems. Numer. Math. 76, 167–188 (1997)
Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)
Chan, T., Esedoglu, S., Park, F., Yip, A.: Total variation image restoration: overview and recent developments. In: Paragios, N., Chen, Y., Faugeras, O.D. (eds.) Handbook of mathematical models in computer vision, pp. 17–31. Springer, New York (2006)
Chan, T.F., Golub, G.H., Mulet, P.: A nonlinear primal-dual method for total variation-based image restoration. SIAM J. Sci. Comput. 20(6), 1964–1977 (1999)
Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Math. Program. 64(1–3), 81–101 (1994)
Combettes, P.L.: Iterative construction of the resolvent of a sum of maximal monotone operators. J. Convex Anal. 16(3–4), 727–748 (2009)
Combettes, P.L., Pesquet, J.C.: A proximal decomposition method for solving convex variational inverse problems. Inverse Probl. 24(26) (2008)
Combettes, P.L., Pesquet, J.C.: Proximal splitting methods in signal processing. Fixed-point algorithms for inverse problems in science and engineering. In: Bauschke, H.H., Burachik, R.S.S., Combettes, P.L., Elser, V., Luke, D.R., Wolkowicz, H. (eds.) Springer Optimization and Its Applications, vol. 49, pp. 185–212. Springer, New York (2011)
Combettes, P.L., Pesquet, J.C.: Primal-dual splitting algorithm for solving inclusions with mixtures of composite, Lipschitzian, and parallel-sum type monotone operators. Set Valued Var. Anal. 20(2), 307–330 (2012)
Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward–backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005)
Condat, L.: A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms. J. Optim. Theory Appl. 158(2), 460–479 (2013)
Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)
De Oliveira, W., Sagastizábal, C., Lemaréchal, C.: Convex proximal bundle methods in depth: a unified analysis for inexact oracles. Math. Program. Ser. B 148(1–2), 241–277 (2014)
Dobson, D.C., Vogel, C.R.: Convergence of an iterative method for total variation denoising. SIAM J. Numer. Anal. 34(5), 1779–1791 (1997)
Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two and three space variables. Trans. Am. Math. Soc. 82(2), 421–439 (1956)
Eckstein, J., Silva, P.J.S.: Proximal methods for nonlinear programming: double regularization and inexact subproblems. Comput. Optim. Appl. 46(2), 279–304 (2010)
Garrigos, G., Rosasco, L., Villa, S.: Iterative regularization via dual diagonal descent. J. Math. Imaging Vis. 60, 189–215 (2017). https://doi.org/10.1007/s10851-017-0754-0
Goldstein, A.A.: Convex programming in Hilbert space. Bull. Am. Math. Soc. 70(5), 709–710 (1964)
Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009)
González-Andrade, S.: A preconditioned descent algorithm for variational inequalities of the second kind involving the \(p\)-Laplacian operator. Comput. Optim. Appl. 66(1), 123–162 (2017)
Hare, W., Planiden, C.: Computing proximal points of convex functions with inexact subgradients. Set Valued Var. Anal. 26(3), 469–492 (2018)
Hare, W., Sagastizábal, C., Solodov, M.: A proximal bundle method for nonsmooth nonconvex functions with inexact information. Comput. Optim. Appl. 63(1), 1–28 (2016)
Lai, M.J., Messi, L.M.: Piecewise linear approximation of the continuous Rudin–Osher–Fatemi model for image denoising. SIAM J. Numer. Anal. 50(5), 2446–2466 (2012)
Levitin, E.S., Polyak, B.T.: Constrained minimization problems. USSR Comput. Math. Math. Phys. 6, 1–50 (1966) (English translation of original paper in Zh. Vychisl. Mat. Fiz. 6, 787–823, 1965)
Martinet, B.: Régularisation d’inéquations variationnelles par approximations successives. Rev. Française Informat. Recherche Opérationnelle 4, 154–158 (1970)
Molinari, C., Peypouquet, J.: Lagrangian penalization algorithm with parallel forward–backward splitting. J. Optim. Theory Appl. 177(2), 413–447 (2018)
Moreau, J.J.: Fonctionnelles Convexes. Cours Collège de France, Paris (1967)
Nesterov, Y.: A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). Sov. Math. Dokl. 27, 372–376 (1983)
Opial, Z.: Weak convergence of the sequence of successive approximations for nonexpansive mappings. Bull. Am. Math. Soc. 73, 591–597 (1967)
Parikh, N., Boyd, S.: Proximal algorithms. Found. Trends Optim. 1(3), 123–231 (2014)
Peypouquet, J.: Convex Optimization in Normed Spaces. Theory, Methods and Examples. With a Foreword by Hedy Attouch. Springer, Berlin (2015)
Reem, D., Reich, S.: Solutions to inexact resolvent inclusion problems with applications to nonlinear analysis and optimization. Rend. Circ. Mat. Palermo (2) 67(2), 337–371 (2018)
Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)
Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)
Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992)
Shen, J., Xia, Z.Q., Pang, L.P.: A proximal bundle method with inexact data for convex nondifferentiable minimization. Nonlinear Anal. 66(9), 2016–2027 (2007)
Su, W., Boyd, S., Candès, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. NIPS 27, 2510–2518 (2014)
Tai, X.C., Wu, C.: Augmented Lagrangian method, dual methods and split Bregman iteration for ROF model. In: Tai, X.C., Moerken, K., Lysaker, M., Lie, K.A. (eds.) Scale space and variational methods in computer vision, pp. 502–513 (2009)
Vogel, C.R., Oman, M.E.: Iterative methods for total variation denoising. SIAM J. Sci. Comput. 17(1), 227–238 (1996)
Vu, B.C.: A splitting algorithm for coupled system of primal-dual monotone inclusions. J. Optim. Theory Appl. 164(3), 993–1025 (2015)
Wu, C., Zhang, J., Tai, X.C.: Augmented Lagrangian method for total variation restoration with non-quadratic fidelity. Inverse Probl. Imaging 5(1), 237–261 (2011)
Zhu, M., Wright, S.J., Chan, T.F.: Duality-based algorithms for total-variation-regularized image restoration. Comput. Optim. Appl. 47, 377–400 (2010)
Acknowledgements
Supported by Fondecyt Grant 1181179 and Basal Project CMM Universidad de Chile. The first author was also supported by CONICYT-PCHA/Doctorado Nacional/2016 scholarship.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Molinari, C., Peypouquet, J. & Roldan, F. Alternating forward–backward splitting for linearly constrained optimization problems. Optim Lett 14, 1071–1088 (2020). https://doi.org/10.1007/s11590-019-01388-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11590-019-01388-y