Abstract
In this paper, we propose an active-set proximal-Newton algorithm for solving \(\ell _1\) regularized convex/nonconvex optimization problems subject to box constraints. Our algorithm first relies on the KKT error to estimate the active and free variables, and then smoothly combines the proximal gradient iteration and the Newton iteration to efficiently pursue the convergence of the active and free variables, respectively. We show the global convergence without the convexity of the objective function. For some structured convex problems, we further design a safe screening procedure that is able to identify/remove active variables, and can be integrated into the basic active-set proximal-Newton algorithm to accelerate the convergence. The algorithm is evaluated on various synthetic and real data, and the efficiency is demonstrated particularly on \(\ell _1\) regularized convex/nonconvex quadratic programs and logistic regression problems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We may construct \(x^k\) as \(x_i^k=\left\{ \begin{array}{l}x_i^*,i\in \mathcal{I}_0^0(x^*)\cup \mathcal{I}_l^+(x^*)\cup \mathcal{I}_u^+(x^*),\\ w_i/k,i\in \mathcal{I}_0^+(x^*)\cup \mathcal{I}_0^-(x^*) \end{array}\right. \) which satisfies all required conditions.
We point out that \(\bar{\mathcal{I}}_u^k\), \(\bar{\mathcal{I}}_l^k\) and \(\bar{\mathcal{I}}_0^k\) are the estimated sets of active constraints corresponding to the lower boundary l, the upper boundary u, and 0, respectively, whereas \(\bar{\mathcal{N}}^k\) is the estimated set of non-active constraints.
The LIBSVM [9] data sets are available at https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/.
References
Bagirov, A.M., Karmitsa, N., Mäkelä, M.M.: Introduction to Nonsmooth Optimization: Theory Practice and Software. Springer, Berlin (2014)
Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)
Becker, S., Bobin, J., Candès, E.J.: NESTA: a fast and accurate first-order method for sparse recovery. SIAM J. Imaging Sci. 4, 1–39 (2011)
Bioucas-Dias, J., Figueiredo, M.: A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. 6(12), 2992–3004 (2007)
Birgin, E.G., Martinez, J.M., Raydan, M.: Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Optim. 10(4), 1196–1211 (2000)
Byrd, R.H., Chin, G.M., Nocedal, J., Oztoprak, F.: A family of second-order methods for convex \(l_1\)-regularized optimization. Math. Program. 159, 435–467 (2016)
Byrd, R.H., Nocedal, J., Oztoprak, F.: An inexact successive quadratic approximation method for \(\ell _1\) regularized optimization. Math. Program. 157, 375–396 (2016)
Candès, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math. 9, 717–772 (2009)
Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 27:1–27:27 (2011)
Dai, Y.H.: A nonmonotone conjugate gradient algorithm for unconstrained optimization. J. Syst. Sci. Complex. 15(2), 139–145 (2002)
Dolan, E.D., Moré, J.: Benchmarking optimization software with performance profiles. Math. Program. 91, 201–213 (2002)
Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)
El Ghaoui, L., Viallon, V., Rabbani, T.: Safe feature elimination in sparse supervised learning. J. Pacific Optim. 8(4), 667–698 (2012)
Fercoq, O., Gramfort, A., Salmon, J.: Mind the duality gap: safer rules for the lasso. In: ICML, pp. 333–342 (2015)
Figueiredo, M.A.T., Nowak, R.D., Wright, S.J.: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Topics Signal Process. 1(4), 586–597 (2007)
Fountoulakis, K., Gondzio, J., Zhlobich, P.: Matrix-free interior point method for compressed sensing problems. Math. Program. Comput. 6, 1–31 (2014)
Friedman, J., Hastie, T., Tibshirani, R.: Sparse inverse covariance estimation with the graphical lasso. Biostatistics 9, 432–441 (2008)
Gafni, E.M., Bertsekas, D.P.: Two-metric projection methods for constrained optimization. SIAM J. Control Optim. 22(6), 936–964 (1984)
Grippo, L., Lampariello, F., Lucidi, S.: A nonmonotone line search technique for Newton’s method. SIAM J. Numer. Anal. 23(4), 707–716 (1986)
Grippo, L., Lampariello, F., Lucidi, S.: A truncated Newton method with nonmonotone line search for unconstrained optimization. J. Optim. Theory Appl. 60(3), 401–419 (1989)
Hale, E.T., Yin, W., Zhang, Y.: Fixed-point continuation for \(l_1\)-minimization: methodology and convergence. SIAM J. Optim. 19, 1107–1130 (2008)
Hiriart-Urruty, J.-B., Lemaréchal, C.: Convex Analysis and Minimization Algorithms II, vol. 306. Springer-Verlag, Berlin (1993)
Keskar, N., Nocedal, J., Oztoprak, F., Wachter, A.: A second-order method for convex \(l_1\)-regularized optimization with active-set prediction. Optim. Methods Softw. 31, 605–621 (2016)
Li, X., Sun, D., Toh, K.: A highly efficient semismooth Newton augmented Lagrangian method for solving Lasso problems. SIAM J. Optim. 28(1), 433–458 (2018)
Lopes, R., Santos, S.A., Silva, P.J.S.: Accelerating block coordinate descent methods with identification strategies. Comput. Optim. Appl. 72(3), 609–640 (2019)
Lucidi, S., Rochetich, F., Roma, M.: Curvilinear stabilization techniques for truncated Newton methods in large-scale unconstrained optimization. SIAM J. Optim. 8(4), 916–939 (1998)
Maros, I., C. Mészáros C, : A repository of convex quadratic programming problems. Optim. Methods Softw. 11&12, 671–681 (1999)
Milzarek, A., Ulbrich, M.: A semismooth Newton method with multidimensional filter globalization for \(\ell _1\)-optimization. SIAM J. Optim. 24(1), 298–333 (2014)
Mohri, M., Rostamizadeh, A., Talwalkar, A.: Foundations of Machine Learning. The MIT Press, Cambridge (2012)
Ndiaye, E., Fercoq, O., Gramfort, A., Salmon, J.: Gap safe screening rules for sparse multi-task and multi-class models. In: NIPS, pp. 811–819 (2015)
Ndiaye, E., Fercoq, O., Gramfort, A., Salmon, J.: GAP safe screening rules for sparse group lasso. In: NIPS, pp 388–396 (2016)
Nesterov, Y.: A method of solving a convex programming problem with convergence rate \({\cal{O}}(1/k^2)\). Soviet Math. Dokl. 27, 372–376 (1983)
Nocedal, J., Wright, S.: Numerical Optimization, 2nd edn. Springer, New York (2006)
Parikh, N., Boyd, S.: Proxmal algorithms. Found. Trends Optim. 1(3), 123–231 (2013)
Platt, J.C.: Fast training of support vector machines using sequential minimal optimization. In: Schölkopf, Bernhard, Burges, Christopher J.C., Smola, Alexander J. (eds.) Advances in Kernel Methods—Support Vector Learning. MIT Press, Cambridge, MA (1998)
Prokhorov, D.: IJCNN 2001 neural network competition. Slide presentation in IJCNN1, Ford Research Laboratory (2001)
Richtárik, P., Takác̀ M., : Efficient Serial and Parallel Coordinate Descent Methods For Huge-Scale Truss Topology Design. Springer, Berlin (2012)
Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B Stat. Methodol. 58, 267–288 (1996)
UCI Machine Learning Repository, http://archive.ics.uci.edu/ml/machine-learningdatabases/statlog/german/
Uzilov, A.V., Keegan, J.M., Mathews, D.H.: Detection of non-coding RNAs on the basis of predicted secondary structure formation free energy change. BMC Bioinformatics 7, 173 (2006)
van den Berg, E., Friedlander, M.P.: Probing the Pareto frontier for basis pursuit solutions. SIAM J. Sci. Comput. 31, 890–912 (2008)
Wen, Z., Yin, W., Goldfarb, D., Zhang, Y.: A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation. SIAM J. Sci Comput. 32(4), 1832–1857 (2010)
Wright, S.J., Nowak, R.D., Figueiredo, M.A.T.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57(7), 2479–2493 (2009)
Xiao, X., Li, Y., Wen, Z., Zhang, L.: A regularized semi-smooth Newton method with projection steps for composite convex programs. J. Sci. Comput. 76(1), 364–389 (2018)
Yan, Y.M.: Maros and Mészáros Convex QP test problems in MAT format, https://github.com/YimingYAN/QP-Test-Problems
Yuan, M., Lin, Y.: Model selection and estimation in regression with grouped variables. J. R. Stat. Soc. Ser. B (Stat. Methodol.) 68(1), 49–67 (2006)
Zhang, H., Cheng, L.: Projected shrinkage algorithm for box-constrained \(l_1\)-minimization. Optim. Lett. 11, 55–70 (2017)
Zhang, H., Hager, W.W.: A nonmonotone line search technique and its application to unconstrained optimization. SIAM J. Optim. 14(4), 1043–1056 (2004)
Acknowledgements
The authors would like to thank the Associate Editor and anonymous referees for their helpful suggestions.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
The work of Wenjuan Xue was supported by National Natural Science Foundation of China (No. 11601318). The work of Lei-Hong Zhang was supported in part by the National Natural Science Foundation of China NSFC-11671246 and NSFC-12071332.
Appendix
Rights and permissions
About this article
Cite this article
Shen, C., Xue, W., Zhang, LH. et al. An Active-Set Proximal-Newton Algorithm for \(\ell _1\) Regularized Optimization Problems with Box Constraints. J Sci Comput 85, 57 (2020). https://doi.org/10.1007/s10915-020-01364-0
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10915-020-01364-0