Skip to main content
Springer Nature Link
Log in

A variable fixing version of the two-block nonlinear constrained Gauss–Seidel algorithm for \(\ell _1\)-regularized least-squares

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

The problem of finding sparse solutions to underdetermined systems of linear equations is very common in many fields as e.g. signal/image processing and statistics. A standard tool for dealing with sparse recovery is the \(\ell _1\)-regularized least-squares approach that has recently attracted the attention of many researchers. In this paper, we describe a new version of the two-block nonlinear constrained Gauss–Seidel algorithm for solving \(\ell _1\)-regularized least-squares that at each step of the iteration process fixes some variables to zero according to a simple active-set strategy. We prove the global convergence of the new algorithm and we show its efficiency reporting the results of some preliminary numerical experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Afonso, M.V., Bioucas-Dias, J.M., Figueiredo, M.A.T.: Fast image recovery using variable splitting and constrained optimization. IEEE Trans. Image Process. 19(9), 2–45 (2010)

    Article  MathSciNet  Google Scholar 

  2. Beck, A., Teboulle, M.: A fast iterative shrinkage–threshold algorithm for linear inverse problems. SIAM J. Imaging Sci. 2, 183–202 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  3. Bertsekas, D.P.: Nonlinear Programming, 2nd edn. Athena Scientific, Belmont (1999)

    MATH  Google Scholar 

  4. Bioucas-Dias, J., Figueiredo, M.: A new twist: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. 16, 2992–3004 (2007)

    Article  MathSciNet  Google Scholar 

  5. Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51(1), 34–81 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  6. Candès, E., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)

    Article  MATH  Google Scholar 

  7. Candès, E., Romberg, J.: Quantitative robust uncertainty principles and optimally sparse decompositions. Found. Comput. Math. 6(2), 227–254 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  8. Candès, E., Romberg, J., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math. 59(8), 1207–1223 (2006)

    Article  MATH  Google Scholar 

  9. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition basis pursuit. SIAM Rev. 43, 129–159 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  10. Daubechies, I., Defrise, M., Mol, C.D.: An iterative thresholding algorithm for linear inverse problems with sparsity constraints. Commun. Pure Appl. Math. 57, 1413–1457 (2004)

    Article  MATH  Google Scholar 

  11. Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program 91, 201221 (2002)

    Article  Google Scholar 

  12. Donoho, D.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  13. Facchinei, F., Fischer, A., Kanzow, C.: On the accurate identification of active constraints. SIAM J. Optim. 9, 14–32 (1999)

    Article  MathSciNet  Google Scholar 

  14. Figueiredo, M., Nowak, R., Wright, S.: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1, 586–598 (2007)

    Article  Google Scholar 

  15. Fletcher, R.: A class of methods for nonlinear programming with termination and convergence properties. In: Abadie, J. (ed.) Integer and Nonlinear Programming, pp. 157–175. North-Holland, The Netherlands (1970)

    Google Scholar 

  16. Fountoulakis, K., Gondzio, J., Zhlobich, P.: Matrix-free interior point method for compressed sensing problems. Math. Program. Comput. 6(1), 1–31 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  17. Grippo, L., Sciandrone, M.: Globally convergent block-coordinate techniques for unconstrained optimization. Optim. Methods Softw. 10(4), 587–637 (1999)

    Article  MATH  MathSciNet  Google Scholar 

  18. Grippo, L., Sciandrone, M.: On the convergence of the block nonlinear Gauss–Seidel method under convex constraints. Oper. Res. Lett. 26(3), 127–136 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  19. Hale, E.T., Yin, W., Zhang, Y.: A numerical study of fixed-point continuation applied to compressed sensing. J. Comput. Math. 28(2), 170194 (2010)

    MathSciNet  Google Scholar 

  20. Haynsworth, E.V.: On the Schur Complement. Basel Mathematical Notes, BNB 20. Wiley, Hoboken (1968)

    Google Scholar 

  21. Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Methods Appl. 4, 303–320 (1969)

    Article  MATH  MathSciNet  Google Scholar 

  22. Kim, S.-J., Koh, K., Lustig, M., Boyd, S., Gorinevsky, D.: An interior-point method for large-scale l1-regularized least squares. IEEE J. Sel. Top. Signal Process. 1(4), 606–617 (2007)

    Article  Google Scholar 

  23. Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, New York (1969)

    Google Scholar 

  24. Tibshirani, R.: Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B 58(1), 267–288 (1996)

    MATH  MathSciNet  Google Scholar 

  25. Tseng, P., Yun, S.: A coordinate gradient descent method for nonsmooth separable minimization. Math. Program. 117(1), 387–423 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  26. Wen, Z., Yin, W., Goldfarb, D., Zhang, Y.: A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation. SIAM J. Sci. Comput. 32(4), 1832–1857 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  27. Wen, Z., Yin, W., Zhang, H., Goldfarb, D.: On the convergence of an active-set method for \(\ell _1\) minimization. Optim. Methods Softw. 27(6), 1127–1146 (2012)

  28. Wright, S., Nowak, R., Figueiredo, M.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57, 2479–2493 (2009)

    Article  MathSciNet  Google Scholar 

  29. Zhang, J.J., Morini, B.: Solving regularized linear least-squares problems by alternating direction methods with applications to image restoration. Electron. Trans. Numer. Anal. (ETNA) 40, 356–372 (2013)

    MATH  MathSciNet  Google Scholar 

Download references

Acknowledgments

The authors thank anonymous referees for useful comments that helped to improve the manuscript. The second author would also like to thank Professor Luigi Grippo for kindly sharing with him many ideas and insights that led to this paper. The work of the first author was supported by “National Group of Computing Science (GNCS-INDAM)”.

Author information

Authors and Affiliations

  1. Dipartimento di Matematica, Università di Bologna, Piazza di Porta S. Donato 5, 40127, Bologna, Italy

    Margherita Porcelli

  2. Dipartimento di Matematica, Università di Padova, Via Trieste 63, 35121, Padua, Italy

    Francesco Rinaldi

Authors
  1. Margherita Porcelli
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Francesco Rinaldi
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Margherita Porcelli.

Appendix

Appendix

In Tables 1, 2 and 3, we report the average CPU time (Av-CPU), the average relative error (Av-rel.err.) and the average number on nonzero components of the computed solution (Av-nnz), corresponding to runs summarized in the performance profiles in Figs. 2 and 3. Problems where the false termination of FPC-AS with the criterion (40) occurs 1 over 10 runs is denoted with the symbol ‘*’ and the average values reported in the tables do not take into account these cases.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Porcelli, M., Rinaldi, F. A variable fixing version of the two-block nonlinear constrained Gauss–Seidel algorithm for \(\ell _1\)-regularized least-squares. Comput Optim Appl 59, 565–589 (2014). https://doi.org/10.1007/s10589-014-9653-0

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-014-9653-0

Keywords