Abstract
In this paper, we propose an accelerated iterative hard thresholding algorithm for solving the \(l_0\) regularized box constrained regression problem. We substantiate that there exists a threshold, if the extrapolation coefficients are chosen below this threshold, the proposed algorithm is equivalent to the accelerated proximal gradient algorithm for solving a corresponding constrained convex problem after finite iterations. Under some proper conditions, we get that the sequence generated by the proposed algorithm is convergent to a local minimizer of the \(l_0\) regularized problem, which satisfies a desired lower bound. Moreover, when the data fitting function satisfies the error bound condition, we prove that both the iterate sequence and the corresponding sequence of objective function values are R-linearly convergent. Finally, we use several numerical experiments to verify our theoretical results.
Similar content being viewed by others
References
Aubin, J.P., Cellina, A.: Differential Inclusions: Set-Valued Maps and Viability Theory. Springer, Berlin (1984)
Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)
Bian, W., Chen, X.J.: Optimality and complexity for constrained optimization problems with nonconvex regularization. Math. Oper. Res. 42(4), 1063–1084 (2017)
Blumensath, T., Davies, M.E.: Iterative thresholding for sparse approximations. J. Fourier Anal. Appl. 14(5–6), 629–654 (2008)
Blumensath, T., Davies, M.E.: Iterative hard thresholding for compressed sensing. Appl. Comput. Harmon. Anal. 27(3), 265–274 (2009)
Blumensath, T., Davies, M.E.: Normalized iterative hard thresholding: guaranteed stability and performance. IEEE J. Sel. Top. Signal Process. 4(2), 298–309 (2010)
Candes, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)
Candes, E.J., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 51(12), 4203–4215 (2005)
Candes, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(l_1\) minimization. J. Fourier Anal. Appl. 14(5–6), 877–905 (2008)
Chambolle, A.: An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 20(1–2), 89–97 (2004)
Chambolle, A., DeVore, R.A., Lee, N.Y., Lucier, B.J.: Nonlinear wavelet image processing: variational problems, compression, and noise removal through wavelet shrinkage. IEEE Trans. Image Process. 7(3), 319–335 (1998)
Chartrand, R.: Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Process. Lett. 14(10), 707–710 (2007)
Chartrand, R., Staneva, V.: Restricted isometry properties and nonconvex compressive sensing. Inverse Probl. 24(3), 1–14 (2008)
Chen, S., Cowan, C.F.N., Grant, P.M.: Orthogonal least-squares learning algorithm for radial basis function networks. IEEE Trans. Neural Netw. 2(2), 302–309 (1991)
Chen, X.J., Ng, M.K., Zhang, C.: Non-Lipschitz \(l_p\)-regularization and box constrained model for image restoration. IEEE Trans. Image Process. 21(12), 4709–4721 (2012)
Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)
Fan, J.Q., Li, R.Z.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96(456), 1348–1360 (2001)
Figueiredo, M.A.T., Nowak, R.D.: An EM algorithm for wavelet-based image restoration. IEEE Trans. Image Process. 12(8), 906–916 (2003)
Foucart, S., Lai, M.J.: Sparsest solutions of underdetermined linear systems via \(l_q\)-minimization for \(0<q \le 1\). Appl. Comput. Harmon. Anal. 26(3), 395–407 (2009)
Hale, E.T., Yin, W.T., Zhang, Y.: Fixed-point continuation for \(l_1\)-minimization: methodology and convergence. SIAM J. Optim. 19(3), 1107–1130 (2008)
Hale, E.T., Yin, W.T., Zhang, Y.: Fixed-point continuation applied to compressed sensing: implementation and numerical experiments. J. Comput. Math. 28(2), 170–194 (2010)
Huang, J., Horowitz, J.L., Ma, S.G.: Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Stat. 36(2), 587–613 (2008)
Johnstone, P.R., Moulin, P.: Local and global convergence of a general inertial proximal splitting scheme for minimizing composite functions. Comput. Optim. Appl 67(2), 259–292 (2017)
Lan, G.H., Monteiro, R.D.C.: Iteration-complexity of first-order penalty methods for convex programming. Math. Program. 138(1–2), 115–139 (2013)
Le Thi, H.A., Dinh, T.P., Le, H.M., Vo, X.T.: DC approximation approaches for sparse optimization. Eur. J. Oper. Res. 244(1), 26–46 (2015)
Liao, J.G., Chin, K.V.: Logistic regression for disease classification using microarray data: model selection in a large \(p\) and small \(n\) case. Bioinformatics 23(15), 1945–1951 (2007)
Lu, Z.S.: Iterative hard thresholding methods for \(l_0\) regularized convex cone programming. Math. Program. 147(1–2), 125–154 (2014)
Lu, Z.S., Zhang, Y.: Sparse approximation via penalty decomposition methods. SIAM J. Optim. 23(4), 2448–2478 (2013)
Luo, Z.Q., Tseng, P.: Error bounds and convergence analysis of feasible descent methods: a general approach. Ann. Oper. Res. 46(1), 157–178 (1993)
Luo, Z.Q., Tseng, P.: On the convergence rate of dual ascent methods for linearly constrained convex minimization. Math. Oper. Res. 18(4), 846–867 (1993)
Mallat, S.G., Zhang, Z.F.: Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Process. 41(12), 3397–3415 (1993)
Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140(1), 125–161 (2013)
Nikolova, M.: Local strong homogeneity of a regularized estimator. SIAM J. Appl. Math. 61(2), 633–658 (2000)
Nikolova, M.: Description of the minimizers of least squares regularized with \(l_0\) norm. Uniqueness of the global minimizer. SIAM J. Imaging Sci. 6(2), 904–937 (2013)
Pati, Y.C., Rezaiifar, R., Krishnaprasad, P.S.: Orthogonal matching pursuit-recursive function approximation with applications to wavelet decomposition. In: Conference Record of the 27th Asilomar Conference on Signal, Systems and Computers vol. 1–2, pp. 40–44 (1993)
Peleg, D., Meir, R.: A bilinear formulation for vector sparsity optimization. Signal Process. 88(2), 375–389 (2008)
Philiastides, M.G., Sajda, P.: Temporal characterization of the neural correlates of perceptual decision making in the human brain. Cereb. Cortex 16(4), 509–518 (2006)
Pilanci, M., Wainwright, M.J., El Ghaoui, L.: Sparse learning via boolean relaxations. Math. Program. 151(1), 63–87 (2015)
Rockafellar, R.T., Wets, R.J.B.: Variational Analysis, vol. 317. Springer, Berlin (1998)
Soubies, E., Blanc-Feraud, L., Aubert, G.: A continuous exact \(l_0\) penalty (CEL0) for least squares regularized problem. SIAM J. Imaging Sci. 9(1), 490–494 (2016)
Tropp, J.A.: Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inf. Theory 50(10), 2231–2242 (2004)
Tropp, J.A.: Just relax: convex programming methods for identifying sparse signals in noise. IEEE Trans. Inf. Theory 52(3), 1030–1051 (2006)
Tseng, P., Yun, S.W.: A coordinate gradient descent method for linearly constrained smooth optimization and support vector machines training. Comput. Optim. Appl. 47(2), 179–206 (2010)
Tsuruoka, Y., McNaught, J., Tsujii, J., Ananiadou, S.: Learning string similarity measures for gene/protein name dictionary look-up using logistic regression. Bioinformatics 23(20), 2768–2774 (2007)
Wen, B., Chen, X.J., Pong, T.K.: Linear convergence of proximal gradient algorithm with extrapolation for a class of nonconvex nonsmooth minimization problems. SIAM J. Optim. 27(1), 124–145 (2017)
Wright, S.J., Nocedal, J.: Numerical Optimization. Springer, New York (2006)
Xu, Z.B., Chang, X.Y., Xu, F.M., Zhang, H.: \({L}_{1/2}\) regularization: a thresholding representation theory and a fast solver. IEEE Trans. Neural Netw. Learn. Syst. 23(7), 1013–1027 (2012)
Yang, F., Shen, Y., Liu, Z.S.: The proximal alternating iterative hard thresholding method for \(l_0\) minimization, with complexity \({O}(1/\sqrt{k})\). J. Comput. Appl. Math. 311, 115–129 (2017)
Yap, P.T., Zhang, Y., Shen, D.G.: Multi-tissue decomposition of diffusion MRI signals via \(l_0\) sparse-group estimation. IEEE Trans. Image Process. 25(9), 4340–4353 (2016)
Zhang, H., Yin, W.T., Cheng, L.Z.: Necessary and sufficient conditions of solution uniqueness in \(1\)-norm minimization. J. Optim. Theory Appl. 164(1), 109–122 (2015)
Zheng, Z.M., Fan, Y.Y., Lv, J.C.: High dimensional thresholded regression and shrinkage effect. J. R. Stat. Soc. Ser. B Stat. Methodol. 76(3), 627–649 (2014)
Acknowledgements
This work was funded in part by the NSF foundation (11871178, 61773136) of China.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Wu, F., Bian, W. Accelerated iterative hard thresholding algorithm for \(l_0\) regularized regression problem. J Glob Optim 76, 819–840 (2020). https://doi.org/10.1007/s10898-019-00826-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10898-019-00826-6