Abstract
In this paper, we study a special class of first-order methods, namely bundle-level (BL) type methods, which can utilize historical first-order information through cutting plane models to accelerate the solutions in practice. Recently, it has been shown in Lan (149(1–2):1–45, 2015) that an accelerated prox-level (APL) method and its variant, the uniform smoothing level (USL) method, have optimal iteration complexity for solving black-box and structured convex programming (CP) problems without requiring input of any smoothness information. However, these algorithms require the assumption on the boundedness of the feasible set and their efficiency relies on the solutions of two involved subproblems. Some other variants of BL methods which could handle unbounded feasible set have no iteration complexity provided. In this work we develop the fast APL (FAPL) method and fast USL (FUSL) method that can significantly improve the practical performance of the APL and USL methods in terms of both computational time and solution quality. Both FAPL and FUSL enjoy the same optimal iteration complexity as APL and USL, while the number of subproblems in each iteration is reduced from two to one, and an exact method is presented to solve the only subproblem in these algorithms. Furthermore, we introduce a generic algorithmic framework to solve unconstrained CP problems through solutions to a series of ball-constrained CP problems that also exhibits optimal iteration complexity. Our numerical results on solving some large-scale least squares problems and total variation based image reconstructions have shown advantages of these new BL type methods over APL, USL, and some other first-order methods.
Similar content being viewed by others
References
Lan, G.: Bundle-level type methods uniformly optimal for smooth and nonsmooth convex optimization. Math. Program. 149(1–2), 1–45 (2015)
Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D: Nonlinear Phenom. 60(2), 259–268 (1992)
Osher, S., Burger, M., Goldfarb, D., Jinjun, X., Yin, W.: An iterative regularization method for total variation-based image restoration. Multiscale Modeling Simul. 4(2), 460–489 (2005)
Nemirovski, A.S., Yudin, D.: Problem complexity and method efficiency in optimization. Wiley-Interscience Series in Discrete Mathematics. John Wiley, XV (1983)
Nesterov, Y.E.: Smooth minimization of nonsmooth functions. Math. Program. 103, 127–152 (2005)
Nesterov, Y.E.: A method for unconstrained convex minimization problem with the rate of convergence $O(1/k^2)$. Dokl. AN USSR. 269, 543–547 (1983)
Nesterov, Y.E.: Introductory Lectures on Convex Optimization: a Basic Course. Kluwer Academic Publishers, Massachusetts (2004)
Chen, Y., Lan, G., Ouyang, Y.: Optimal primal-dual methods for a class of saddle point problems. SIAM J. Optim. 24(4), 1779–1814 (2014)
He, Y., Monteiro, R.D.C.: An accelerated hpe-type algorithm for a class of composite convex-concave saddle-point problems. SIAM J. Optim. 26(1), 29–56 (2016)
Hong, M., Luo, Z.Q.: On the linear convergence of the alternating direction method of multipliers. Math. Program. 162, 1–35 (2012)
Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. J. Sci. Comput. 66(3), 889–916 (2016)
Goldfarb, D., Ma, S., Scheinberg, K.: Fast alternating linearization methods for minimizing the sum of two convex functions. Math. Program. 141(1–2), 349–382 (2013)
Monteiro, R.D.C., Svaiter, B.F.: Iteration-complexity of block-decomposition algorithms and the alternating direction method of multipliers. SIAM J. Optim. 23(1), 475–507 (2013)
Goldstein, T., O’Donoghue, B., Setzer, S., Baraniuk, R.: Fast alternating direction optimization methods. SIAM J. Imaging Sci. 7(3), 1588–1623 (2014)
Ouyang, Y., Chen, Y., Lan, G., Pasiliao Jr., E.: An accelerated linearized alternating direction method of multipliers. SIAM J. Imaging Sci. 8(1), 644–681 (2015)
Kelley, J.E.: The cutting plane method for solving convex programs. J. SIAM. 8, 703–712 (1960)
Kiwiel, K.C.: An aggregate subgradient method for nonsmooth convex minimization. Math. Program. 27, 320–341 (1983)
Lemaréchal, C.: An extension of davidon methods to non-differentiable problems. Math. Program. Study. 3, 95–109 (1975)
Kiwiel, K.C.: Proximal level bundle method for convex nondifferentable optimization, saddle point problems and variational inequalities. Math. Program. Ser. B. 69, 89–109 (1995)
Lemaréchal, C., Nemirovski, A.S., Nesterov, Y.E.: New variants of bundle methods. Math. Program. 69, 111–148 (1995)
Kiwiel, K.C.: Proximity control in bundle methods for convex nondifferentiable minimization. Math. Program. 46, 105–122 (1990)
Ben-Tal, A., Nemirovski, A.S.: Non-Euclidean restricted memory level method for large-scale convex optimization. Math. Program. 102, 407–456 (2005)
van Ackooij, W., Sagastizábal, C.: Constrained bundle methods for upper inexact oracles with application to joint chance constrained energy problems. SIAM J. Optim. 24(2), 733–765 (2014)
de Oliveira, W., Sagastizábal, C., Scheimberg, S.: Inexact bundle methods for two-stage stochastic programming. SIAM J. Optim. 21(2), 517–544 (2011)
de Oliveira, W., Sagastizábal, C., Lemaréchal, C.: Convex proximal bundle methods in depth: a unified analysis for inexact oracles. Math. Program. 148(1–2), 241–277 (2014)
Richtárik, P.: Approximate level method for nonsmooth convex minimization. J. Optim. Theory Appl. 152(2), 334–350 (2012)
Kiwiel, K.C.: A proximal bundle method with approximate subgradient linearizations. SIAM J. optim. 16(4), 1007–1023 (2006)
Kiwiel, Krzysztof C: Bundle methods for convex minimization with partially inexact oracles. Comput. Optim. Appl., available from the web site SemanticScholar
de Oliveira, W, Sagastizábal, C: Level bundle methods for oracles with on-demand accuracy. Optim. Methods Softw. (ahead-of-print): 29,1–30 (2014)
Kiwiel, K.C., Lemaréchal, C.: An inexact bundle variant suited to column generation. Math. program. 118(1), 177–206 (2009)
Brännlund, U., Kiwiel, K.C., Lindberg, P.O.: A descent proximal level bundle method for convex nondifferentiable optimization. Op. Res. Lett. 17(3), 121–126 (1995)
Cruz, J.B., de Oliveira, W.: Level bundle-like algorithms for convex optimization. J. Glob. Optim. 59, 1–23 (2013)
de Oliveira, W., Sagastizábal, C.: Bundle methods in the xxist century: a bird’s-eye view. Pesqui. Op. 34(3), 647–670 (2014)
Becker, S., Bobin, J., Candès, E.J.: Nesta: a fast and accurate first-order method for sparse recovery. SIAM J. Imaging Sci. 4(1), 1–39 (2011)
Astorino, A., Frangioni, A., Gaudioso, M., Gorgone, E.: Piecewise-quadratic approximations in convex numerical optimization. SIAM J. Optim. 21(4), 1418–1438 (2011)
Ouorou, A.: A proximal cutting plane method using chebychev center for nonsmooth convex optimization. Math. Program. 119(2), 239–271 (2009)
Mosek. The mosek optimization toolbox for matlab manual. version 6.0 (revision 93). http://www.mosek.com
Chen, Y., Hager, W., Huang, F., Phan, D., Ye, X., Yin, W.: Fast algorithms for image reconstruction with application to partially parallel mr imaging. SIAM J. Imaging Sci. 5(1), 90–118 (2012)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
December, 2014. This research was partially supported by NSF Grants CMMI-1254446, DMS-1319050, DMS-1719932, and ONR Grant N00014-13-1-0036.
Appendix A. Solving the subproblems of FAPL and FUSL
Appendix A. Solving the subproblems of FAPL and FUSL
In this section, we introduce an efficient method to solve the subproblems (2.6) in the FAPL and FUSL methods, which are given in the form of
Here, Q is a closed polyhedral set described by m linear inequalities, i.e.,
where \(A_i\in \mathbb {R}^n\) and \(b_i\in \mathbb {R}\) for \(1\le i\le m\).
Now let us examine the Lagrange dual of (A.1) given by
It can be checked from the theorem of alternatives that problem (A.2) is solvable if and only if \(Q \ne \emptyset \). Indeed, if \(Q \ne \emptyset \), it is obvious that the optimal value of (A.2) is finite. On the other hand, if \(Q = \emptyset \), then there exists \(\bar{\lambda } \ge 0\) such that \(\bar{\lambda }^T A = 0\) and \(\bar{\lambda }^T b < 0\), which implies that the optimal value of (A.2) goes to infinity. Moreover, if (A.2) is solvable and \(\lambda ^*\) is one of its optimal dual solutions, then
It can also be easily seen that (A.2) is equivalent to
where \( M_{ij}:=\left\langle A_i,A_j\right\rangle ,\ C_i:=\left\langle A_i,p\right\rangle -b_i, \ \forall i,j=1,2,\ldots ,m. \) Hence, we can determine the feasibility of (A.1) or compute its optimal solution by solving the relatively simple problem (A.4).
Many algorithms are capable of solving the above nonnegative quadratic programming in (A.4) efficiently. Due to its low dimension (usually less than 10 in our practice), we propose a brute-force method to compute the exact solution of this problem. Consider the Lagrange dual associated with (A.4):
where the dual variable is \(\mu :=(\mu _1,\mu _2,\ldots ,\mu _m)\). Applying the KKT condition, we can see that \(\lambda ^{*}\ge 0\) is a solution to problem (A.4) if and only if there exists \(\mu ^*\ge 0\) such that
Note that the first identity in (A.5) is equivalent to a linear system:
where I is the \(m\times m\) identity matrix. The above linear system has 2m variables and m equations. But for any \(i=1,\ldots ,m\), we have either \(\lambda _i=0\) or \(\mu _i=0\), and hence we only need to consider \(2^m\) possible cases on the non-negativity of these variables. Since m is rather small in practice, it is possible to exhaust all these \(2^m\) cases to find the exact solution to (A.5). For each case, we first remove the m columns in the matrix \((M \ -I)\) which correspond to the m variables assumed to be 0, and then solve the remaining determined linear system. If all variables of the computed solution are non-negative, then solution \((\lambda ^{*},\mu ^{*})\) to (A.5) is found, and the exact solution \(x_c^*\) to (A.1) is computed by (A.3), otherwise, we continue to examine the next case. It is interesting to observe that these different cases can also be considered in parallel to take the advantages of high performance computing techniques.
Rights and permissions
About this article
Cite this article
Chen, Y., Lan, G., Ouyang, Y. et al. Fast bundle-level methods for unconstrained and ball-constrained convex optimization. Comput Optim Appl 73, 159–199 (2019). https://doi.org/10.1007/s10589-019-00071-3
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10589-019-00071-3