Skip to main content
Log in

Accelerated iterative hard thresholding algorithm for \(l_0\) regularized regression problem

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

In this paper, we propose an accelerated iterative hard thresholding algorithm for solving the \(l_0\) regularized box constrained regression problem. We substantiate that there exists a threshold, if the extrapolation coefficients are chosen below this threshold, the proposed algorithm is equivalent to the accelerated proximal gradient algorithm for solving a corresponding constrained convex problem after finite iterations. Under some proper conditions, we get that the sequence generated by the proposed algorithm is convergent to a local minimizer of the \(l_0\) regularized problem, which satisfies a desired lower bound. Moreover, when the data fitting function satisfies the error bound condition, we prove that both the iterate sequence and the corresponding sequence of objective function values are R-linearly convergent. Finally, we use several numerical experiments to verify our theoretical results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Aubin, J.P., Cellina, A.: Differential Inclusions: Set-Valued Maps and Viability Theory. Springer, Berlin (1984)

    MATH  Google Scholar 

  2. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)

    MathSciNet  MATH  Google Scholar 

  3. Bian, W., Chen, X.J.: Optimality and complexity for constrained optimization problems with nonconvex regularization. Math. Oper. Res. 42(4), 1063–1084 (2017)

    MathSciNet  MATH  Google Scholar 

  4. Blumensath, T., Davies, M.E.: Iterative thresholding for sparse approximations. J. Fourier Anal. Appl. 14(5–6), 629–654 (2008)

    MathSciNet  MATH  Google Scholar 

  5. Blumensath, T., Davies, M.E.: Iterative hard thresholding for compressed sensing. Appl. Comput. Harmon. Anal. 27(3), 265–274 (2009)

    MathSciNet  MATH  Google Scholar 

  6. Blumensath, T., Davies, M.E.: Normalized iterative hard thresholding: guaranteed stability and performance. IEEE J. Sel. Top. Signal Process. 4(2), 298–309 (2010)

    Google Scholar 

  7. Candes, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)

    MathSciNet  MATH  Google Scholar 

  8. Candes, E.J., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 51(12), 4203–4215 (2005)

    MathSciNet  MATH  Google Scholar 

  9. Candes, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(l_1\) minimization. J. Fourier Anal. Appl. 14(5–6), 877–905 (2008)

    MathSciNet  MATH  Google Scholar 

  10. Chambolle, A.: An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 20(1–2), 89–97 (2004)

    MathSciNet  MATH  Google Scholar 

  11. Chambolle, A., DeVore, R.A., Lee, N.Y., Lucier, B.J.: Nonlinear wavelet image processing: variational problems, compression, and noise removal through wavelet shrinkage. IEEE Trans. Image Process. 7(3), 319–335 (1998)

    MathSciNet  MATH  Google Scholar 

  12. Chartrand, R.: Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Process. Lett. 14(10), 707–710 (2007)

    Google Scholar 

  13. Chartrand, R., Staneva, V.: Restricted isometry properties and nonconvex compressive sensing. Inverse Probl. 24(3), 1–14 (2008)

    MathSciNet  MATH  Google Scholar 

  14. Chen, S., Cowan, C.F.N., Grant, P.M.: Orthogonal least-squares learning algorithm for radial basis function networks. IEEE Trans. Neural Netw. 2(2), 302–309 (1991)

    Google Scholar 

  15. Chen, X.J., Ng, M.K., Zhang, C.: Non-Lipschitz \(l_p\)-regularization and box constrained model for image restoration. IEEE Trans. Image Process. 21(12), 4709–4721 (2012)

    MathSciNet  MATH  Google Scholar 

  16. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)

    MathSciNet  MATH  Google Scholar 

  17. Fan, J.Q., Li, R.Z.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96(456), 1348–1360 (2001)

    MathSciNet  MATH  Google Scholar 

  18. Figueiredo, M.A.T., Nowak, R.D.: An EM algorithm for wavelet-based image restoration. IEEE Trans. Image Process. 12(8), 906–916 (2003)

    MathSciNet  MATH  Google Scholar 

  19. Foucart, S., Lai, M.J.: Sparsest solutions of underdetermined linear systems via \(l_q\)-minimization for \(0<q \le 1\). Appl. Comput. Harmon. Anal. 26(3), 395–407 (2009)

    MathSciNet  MATH  Google Scholar 

  20. Hale, E.T., Yin, W.T., Zhang, Y.: Fixed-point continuation for \(l_1\)-minimization: methodology and convergence. SIAM J. Optim. 19(3), 1107–1130 (2008)

    MathSciNet  MATH  Google Scholar 

  21. Hale, E.T., Yin, W.T., Zhang, Y.: Fixed-point continuation applied to compressed sensing: implementation and numerical experiments. J. Comput. Math. 28(2), 170–194 (2010)

    MathSciNet  MATH  Google Scholar 

  22. Huang, J., Horowitz, J.L., Ma, S.G.: Asymptotic properties of bridge estimators in sparse high-dimensional regression models. Ann. Stat. 36(2), 587–613 (2008)

    MathSciNet  MATH  Google Scholar 

  23. Johnstone, P.R., Moulin, P.: Local and global convergence of a general inertial proximal splitting scheme for minimizing composite functions. Comput. Optim. Appl 67(2), 259–292 (2017)

    MathSciNet  MATH  Google Scholar 

  24. Lan, G.H., Monteiro, R.D.C.: Iteration-complexity of first-order penalty methods for convex programming. Math. Program. 138(1–2), 115–139 (2013)

    MathSciNet  MATH  Google Scholar 

  25. Le Thi, H.A., Dinh, T.P., Le, H.M., Vo, X.T.: DC approximation approaches for sparse optimization. Eur. J. Oper. Res. 244(1), 26–46 (2015)

    MathSciNet  MATH  Google Scholar 

  26. Liao, J.G., Chin, K.V.: Logistic regression for disease classification using microarray data: model selection in a large \(p\) and small \(n\) case. Bioinformatics 23(15), 1945–1951 (2007)

    Google Scholar 

  27. Lu, Z.S.: Iterative hard thresholding methods for \(l_0\) regularized convex cone programming. Math. Program. 147(1–2), 125–154 (2014)

    MathSciNet  MATH  Google Scholar 

  28. Lu, Z.S., Zhang, Y.: Sparse approximation via penalty decomposition methods. SIAM J. Optim. 23(4), 2448–2478 (2013)

    MathSciNet  MATH  Google Scholar 

  29. Luo, Z.Q., Tseng, P.: Error bounds and convergence analysis of feasible descent methods: a general approach. Ann. Oper. Res. 46(1), 157–178 (1993)

    MathSciNet  MATH  Google Scholar 

  30. Luo, Z.Q., Tseng, P.: On the convergence rate of dual ascent methods for linearly constrained convex minimization. Math. Oper. Res. 18(4), 846–867 (1993)

    MathSciNet  MATH  Google Scholar 

  31. Mallat, S.G., Zhang, Z.F.: Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Process. 41(12), 3397–3415 (1993)

    MATH  Google Scholar 

  32. Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140(1), 125–161 (2013)

    MathSciNet  MATH  Google Scholar 

  33. Nikolova, M.: Local strong homogeneity of a regularized estimator. SIAM J. Appl. Math. 61(2), 633–658 (2000)

    MathSciNet  MATH  Google Scholar 

  34. Nikolova, M.: Description of the minimizers of least squares regularized with \(l_0\) norm. Uniqueness of the global minimizer. SIAM J. Imaging Sci. 6(2), 904–937 (2013)

    MathSciNet  MATH  Google Scholar 

  35. Pati, Y.C., Rezaiifar, R., Krishnaprasad, P.S.: Orthogonal matching pursuit-recursive function approximation with applications to wavelet decomposition. In: Conference Record of the 27th Asilomar Conference on Signal, Systems and Computers vol. 1–2, pp. 40–44 (1993)

  36. Peleg, D., Meir, R.: A bilinear formulation for vector sparsity optimization. Signal Process. 88(2), 375–389 (2008)

    MATH  Google Scholar 

  37. Philiastides, M.G., Sajda, P.: Temporal characterization of the neural correlates of perceptual decision making in the human brain. Cereb. Cortex 16(4), 509–518 (2006)

    Google Scholar 

  38. Pilanci, M., Wainwright, M.J., El Ghaoui, L.: Sparse learning via boolean relaxations. Math. Program. 151(1), 63–87 (2015)

    MathSciNet  MATH  Google Scholar 

  39. Rockafellar, R.T., Wets, R.J.B.: Variational Analysis, vol. 317. Springer, Berlin (1998)

    MATH  Google Scholar 

  40. Soubies, E., Blanc-Feraud, L., Aubert, G.: A continuous exact \(l_0\) penalty (CEL0) for least squares regularized problem. SIAM J. Imaging Sci. 9(1), 490–494 (2016)

    MathSciNet  MATH  Google Scholar 

  41. Tropp, J.A.: Greed is good: algorithmic results for sparse approximation. IEEE Trans. Inf. Theory 50(10), 2231–2242 (2004)

    MathSciNet  MATH  Google Scholar 

  42. Tropp, J.A.: Just relax: convex programming methods for identifying sparse signals in noise. IEEE Trans. Inf. Theory 52(3), 1030–1051 (2006)

    MathSciNet  MATH  Google Scholar 

  43. Tseng, P., Yun, S.W.: A coordinate gradient descent method for linearly constrained smooth optimization and support vector machines training. Comput. Optim. Appl. 47(2), 179–206 (2010)

    MathSciNet  MATH  Google Scholar 

  44. Tsuruoka, Y., McNaught, J., Tsujii, J., Ananiadou, S.: Learning string similarity measures for gene/protein name dictionary look-up using logistic regression. Bioinformatics 23(20), 2768–2774 (2007)

    Google Scholar 

  45. Wen, B., Chen, X.J., Pong, T.K.: Linear convergence of proximal gradient algorithm with extrapolation for a class of nonconvex nonsmooth minimization problems. SIAM J. Optim. 27(1), 124–145 (2017)

    MathSciNet  MATH  Google Scholar 

  46. Wright, S.J., Nocedal, J.: Numerical Optimization. Springer, New York (2006)

    MATH  Google Scholar 

  47. Xu, Z.B., Chang, X.Y., Xu, F.M., Zhang, H.: \({L}_{1/2}\) regularization: a thresholding representation theory and a fast solver. IEEE Trans. Neural Netw. Learn. Syst. 23(7), 1013–1027 (2012)

    Google Scholar 

  48. Yang, F., Shen, Y., Liu, Z.S.: The proximal alternating iterative hard thresholding method for \(l_0\) minimization, with complexity \({O}(1/\sqrt{k})\). J. Comput. Appl. Math. 311, 115–129 (2017)

    MathSciNet  MATH  Google Scholar 

  49. Yap, P.T., Zhang, Y., Shen, D.G.: Multi-tissue decomposition of diffusion MRI signals via \(l_0\) sparse-group estimation. IEEE Trans. Image Process. 25(9), 4340–4353 (2016)

    MathSciNet  MATH  Google Scholar 

  50. Zhang, H., Yin, W.T., Cheng, L.Z.: Necessary and sufficient conditions of solution uniqueness in \(1\)-norm minimization. J. Optim. Theory Appl. 164(1), 109–122 (2015)

    MathSciNet  MATH  Google Scholar 

  51. Zheng, Z.M., Fan, Y.Y., Lv, J.C.: High dimensional thresholded regression and shrinkage effect. J. R. Stat. Soc. Ser. B Stat. Methodol. 76(3), 627–649 (2014)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was funded in part by the NSF foundation (11871178, 61773136) of China.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Wei Bian.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, F., Bian, W. Accelerated iterative hard thresholding algorithm for \(l_0\) regularized regression problem. J Glob Optim 76, 819–840 (2020). https://doi.org/10.1007/s10898-019-00826-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-019-00826-6

Keywords

Navigation