Skip to main content
SpringerLink
Log in

A non-convex piecewise quadratic approximation of \(\ell _{0}\) regularization: theory and accelerated algorithm

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

Non-convex regularization has been recognized as an especially important approach in recent studies to promote sparsity. In this paper, we study the non-convex piecewise quadratic approximation (PQA) regularization for sparse solutions of the linear inverse problem. It is shown that exact recovery of sparse signals and stable recovery of compressible signals are possible through local optimum of this regularization. After developing a thresholding representation theory for PQA regularization, we propose an iterative PQA thresholding algorithm (PQA algorithm) to solve this problem. The PQA algorithm converges to a local minimizer of the regularization, with an eventually linear convergence rate. Furthermore, we adopt the idea of accelerated gradient method to design the accelerated iterative PQA thresholding algorithm (APQA algorithm), which is also linearly convergent, but with a faster convergence rate. Finally, we carry out a series of numerical experiments to assess the performance of both algorithms for PQA regularization. The results show that PQA regularization outperforms \(\ell _1\) and \(\ell _{1/2}\) regularizations in terms of accuracy and sparsity, while the APQA algorithm is demonstrated to be significantly better than the PQA algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2, 183–202 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  2. Blumensath, T., Davies, M.E.: Iterative hard thresholding for compressed sensing. Appl. Comput. Harmon. Anal. 27(3), 265–274 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  3. Blumensath, T., Davies, M.E.: Iterative thresholding for sparse approximations. J. Fourier Anal. Appl 14, 629–654 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51(1), 34–81 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  5. Candes, E.J., Recht, B.: Exact matrix completion via convex optimization. Found. Comput. Math. 9(6), 717–772 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  6. Candes, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  7. Candes, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(l_{1}\) minimization. J. Fourier Anal. Appl. 14, 877–905 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  8. Cao, W.F., Sun, J., Xu, Z.B.: Fast image deconvolution using closed-form thresholding formulas of lq (q=1/2,2/3) regularization. J. Vis. Commun. Image Represent 24(1), 1529–1542 (2013)

    Article  Google Scholar 

  9. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  10. Chen, X.M.: Stability of compressed sensing for dictionaries and almost sure convergence rate for the kaczmarz algorithm. Ph.D. thesis, Vanderbilt University (2012)

  11. Cohen, A., Dahmen, W., DeVore, R.: Compressed sensing and best k-term approximation. J Am Math Soc 22(1), 211–231 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  12. Daubechies, I., Defrise, M., Christine, D.M.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math 57(1), 1413–1457 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  13. Daubechies, I., Devore, R., Fornasier, M., Gunturk, C.S.: Iteratively reweighted least squares minimization for sparse recovery. Comm. Pure Appl. Math. 63, 1–38 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  14. Devore, R., Jawerth, B.: Image compression through wavelet transform coding. IEEE Trans. Inf. Theory 38(2), 719–746 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  15. Donoho, D.L.: De-noising by soft-thresholding. IEEE Trans. Inf. Theory 41(3), 613–627 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  16. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  17. Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Amer. Stat. Assoc. 96, 1348–1360 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  18. Ghadimi, S., Lan, G.: Accelerated gradient methods for non-convex nonlinear and stochastic programming. Math. Program 156, 59–99 (2013)

    Article  MATH  Google Scholar 

  19. Gong, P.H., Zhang, C.S., Lu, Z.S., Huang, J.H., Ye, J.P.: A general iteartive shrinkage and thresholding algorithm for non-convex regularized optimization problems. Presented at the 30th Int. Conf. Mach. Learn. (ICML), Atlanta, GA, USA (2013)

  20. Jing, W., Meng, D., Qiao, C., Peng, Z.: Eliminating vertical stripe defects on silicon steel surface by regularization. Math. Probl. Eng. 2011, 1–13 (2011)

    Article  MathSciNet  Google Scholar 

  21. Lai, M.J., Xu, Y.Y., Yin, W.T.: Improved iteratively reweighted least squares for unconstrained smoothed \(l_{q}\) minimization. SIAM J. Numer. Anal. 51, 927–957 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  22. Li, H., Lin, Z.: Accelerated proximal gradient methods for nonconvex programming. Adv. Neural Inf. Process. Syst. 28 (2015)

  23. Li, Q., Bai, Y.Q., Yu, C.J., Yuan, Y.X.: A new piecewise quadratic approximation approach for \(l_0\) norm minimization problem. Sci. China Math. 62(1), 185–204 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  24. Mallat, S., Zhang, Z.: Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Process. 41, 3397–3415 (1993)

    Article  MATH  Google Scholar 

  25. Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24(2), 227–234 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  26. Needell, D., Vershynin, R.: Signal recovery from incomplete and inaccurate measurements via regularized orthogonal matching pursuit. IEEE J. Sel. Topics Signal Process. 4, 310–316 (2010)

    Article  Google Scholar 

  27. Nesterov, Y.: A method for unconstrained convex minimization problem with the rate of convergence \(\cal{O} (1/k^2)\). Doklady AN SSSR 269, 545–547 (1983)

    Google Scholar 

  28. Nesterov, Y.: Smoothing minimization of non-smooth functions. Math. Program 103, 127–152 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  29. Rakotomamonjy, A., Flamary, R., Gasso, G., Canu, S.: lp-lq penalty for sparse linear and sparse multiple kernel multitask learning. IEEE Trans. Neural Netw. 22, 1307–1320 (2011)

    Article  Google Scholar 

  30. Tibshiranit, R.: Regression shrinkage and selection via the lasso. J. Royal Stat. Soc. Ser. B 58(1), 267–288 (1996)

    MathSciNet  Google Scholar 

  31. Tran, H., Webster, C.: A class of null space conditions for sparse recovery via nonconvex, non-separable minimizations. Results Appl Math 3, 100011 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  32. Tropp, J.A., Gilbert, A.C.: Signal recovery from partial information via orthogonal matching pursuit. IEEE Trans. Inf. Theor. 53, 4655–4666 (2007)

    Article  MATH  Google Scholar 

  33. Tseng, P.: On accelerated proximal gradient methods for convex-concave optimization. SIAM J Optim (2008)

  34. Wen, B., Chen, X., Pong, T.K.: Linear convergence of proximal gradient algorithm with extrapolation for a class of nonconvex nonsmooth minimization problems. Siam J Optim 27(1), 124–145 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  35. Xu, Z.B.: Data modeling: visual psychology approach and \(l_{1/2}\) regularization theory. Proc. Int. Congr. Math 4, 3151–3184 (2010)

    MATH  Google Scholar 

  36. Xu, Z.B., Chang, X.Y., Xu, F.M., Zhang, H.: \(l_{1/2}\) regularization: a thresholding representation theory and a fast solver. IEEE Trans. Neural Netw. Learn. Syst. 23(7), 1013–1027 (2012)

    Article  Google Scholar 

  37. Xu, Z.B., Guo, H.L., Wang, Y., Zhang, H.: The representation of \(l_{1/2}\) regularizer among \(l_{q}, (0<q<1)\) regularizer: An experimental study based on phase diagram. Acta Autom. Sin. 38, 1225–1228 (2012)

    Article  Google Scholar 

  38. Yang, A.Y., Ganesh, A., Ma, Y.: Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31(2), 210–227 (2009)

    Article  Google Scholar 

  39. Yang, A.Y., Ganesh, A., Zhou, Z.H., Sastry, S.S., Ma, Y.: Fast \(l_{1}\)-minimization algorithms for robust face recognition. IEEE Trans. Image Process. 22, 3234–3246 (2013)

    Article  Google Scholar 

  40. Yin, P.H., Lou, Y.F., He, Q., Xin, J.: Minimization of \(l_{1-2}\) for compressed sensing. SIAM J. Sci. Comput. 37, 536–563 (2015)

    Article  MathSciNet  Google Scholar 

  41. Zeng, J., Xu, Z., Zhang, B., Hong, W., Wu, Y.: Accelerated \(l_{1/2}\) regularization based sar imaging via bcr and reduced newton skills. Signal Process. 93(7), 1831–1844 (2013)

    Article  Google Scholar 

  42. Zeng, J.S., Jian, F., Xu, Z.B.: Sparse sar imaging based on \(l_{1/2}\) regularization. Sci. China Inf. Sci. 55(8), 1755–1775 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  43. Zeng, J.S., Lin, S.B., Wang, Y., Xu, Z.B.: \(l_{1/2}\) regularization: convergence of iterative half thresholding algorithm. IEEE Signal Process. 62(9), 2317–2329 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  44. Zeng, J.S., Lin, S.B., Xu, Z.B.: Sparse regularization: convergence of iterative jumping thresholding algorithm. IEEE Signal Process. 64, 5106–5118 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  45. Zhang, C.H.: Nearly unbiaised variable selection under minimax concave penalty. Ann. Stat. 38, 894–942 (2010)

    Article  MATH  Google Scholar 

Download references

Acknowledgements

We thank the editors and the anonymous reviewers for their valuable comments and suggestions. This research was supported by the National Natural Science Foundations of China (Grants No. 11901382, 12101247, 11771275, 11971302) and the Shanghai Chenguang Project (Grants No. 19CG67).

Author information

Authors and Affiliations

  1. School of Mathematics, Physics and Statistics, Shanghai University of Engineering Science, Shanghai, 201620, China

    Qian Li & Guoqiang Wang

  2. School of Mathematics, South China University of Technology, Guangzhou, 510640, China

    Wei Zhang

  3. Department of Mathematics, Shanghai University, Shanghai, 200444, China

    Yanqin Bai

Authors
  1. Qian Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yanqin Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guoqiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Q., Zhang, W., Bai, Y. et al. A non-convex piecewise quadratic approximation of \(\ell _{0}\) regularization: theory and accelerated algorithm. J Glob Optim 86, 323–353 (2023). https://doi.org/10.1007/s10898-022-01257-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-022-01257-6

Keywords

Mathematics Subject Classification

Search

Navigation