Skip to main content
SpringerLink
Log in

Ways to sparse representation: An overview

  • Published:
Science in China Series F: Information Sciences Aims and scope Submit manuscript

Abstract

Many algorithms have been proposed to find sparse representations over redundant dictionaries or transforms. This paper gives an overview of these algorithms by classifying them into three categories: greedy pursuit algorithms, l p norm regularization based algorithms, and iterative shrinkage algorithms. We summarize their pros and cons as well as their connections. Based on recent evidence, we conclude that the algorithms of the three categories share the same root: l p norm regularized inverse problem. Finally, several topics that deserve further investigation are also discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Mallat S. A Wavelet Tour of Signal Processing. San Diego: Academic Press, 1998

    MATH  Google Scholar 

  2. Mallat S, Zhang Z. Matching pursuits with time-frequency dictionaries. IEEE Trans Signal Process, 1993 41(3): 3397–3415

    Article  MATH  Google Scholar 

  3. Davis G, Mallat S, Zhang Z. Adaptive time-frequency decompositions. Opt Engin, 1994 33(7): 2183–2191

    Article  Google Scholar 

  4. Chen S, Donoho D, Saunders M. Atomic decomposition by basis pursuit. SIAM J Sci Comput, 1999 20: 33–61

    Article  MATH  MathSciNet  Google Scholar 

  5. Elad M. Why simple shrinkage is still relevant for redundant representations?. IEEE Trans Inf Theory, 2006 52(12): 5559–5569

    Article  MathSciNet  Google Scholar 

  6. Tibshirani R. Regression shrinkage and selection via the lasso. J Royal Statist Soc Ser B (Methodological), 1996 58(1): 267–288

    MATH  MathSciNet  Google Scholar 

  7. Osborne M R, Presnell B, Turlach B A. A new approach to variable selection in least squares problems. IMA J Num Anal, 2000, 20(3): 389–403

    Article  MATH  MathSciNet  Google Scholar 

  8. Efron B, Hastie T, Johnstone I, et al.Least angle regression. Annals Statist, 2004 32: 407–499

    Article  MATH  MathSciNet  Google Scholar 

  9. Starck J L, Nguyen M K, Murtagh F. Wavelets and curvelets for image deconvolution: A combined approach. Signal Process, 2003 83: 2279–2283

    Article  MATH  Google Scholar 

  10. Figueiredo M A T, Nowak R D. An EM algorithm for waveletbased image restoration. IEEE Trans Image Process, 2003 12(8): 906–916

    Article  MathSciNet  Google Scholar 

  11. Fischer S, Crist’obal G, Redondo R. Sparse overcomplete Gabor wavelet representation based on local competitions. IEEE Trans Signal Process, 2006 15(2): 265–272

    Google Scholar 

  12. Wang B, Wang Y, Selesnick I, et al. Video coding using 3-D dual-tree wavelet transform. EURASIP J Image Video Process, Special issue on “Wavelets in Source Coding, Communications, and Networks”, Volume 2007 (2007), Article ID. 42761, 15 pages, doi: 10.1155/2007/42761

  13. Yang J Y, Xu W L, Dai Q H, et al. Image compression using 2-D dual-tree discrete wavelet transform (DDWT). In: Proc. IEEE International Symposium on Circuits and Systems, New Orleans, 2007. 297–300

  14. Figueiredo M A T, Nowak R D, Wright S J. Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse Problems. Technical Report, University of Wisconsin, 2007

  15. DeVore R A. Nonlinear Approximation, Acta Numerica. Cambridge: Cambridge Univ. Press, 1998. 51–150

    Google Scholar 

  16. Pati Y C, Rezaiifar R, Krishnaprasad P S. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In: Proc. 27th Annu. Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, USA, 1993. 40–44

  17. Cotter S, Rao B. Application of tree-based searches to matching pursuit. In: Proc. International Conference on Acoustics, Speech, and Signal Processing, Salt Lake City, UT, USA, 2001. 3933–3936

  18. Jost P, Vandergheynst P, Frossard P. Tree-based pursuit: Algorithm and properties. IEEE Trans Signal Process, 2006 54(12): 4685–4697

    Article  Google Scholar 

  19. Gorodnitsky I F, Rao B D. Sparse signal reconstructions from limited data using FOCUSS: A re-weighted minimum norm algorithm. IEEE Trans Signal Process, 1997 45(3): 600–616

    Article  Google Scholar 

  20. Rao B D, Kreutz-Delgado K. An affine scaling methodology for best basis selection. IEEE Trans Signal Process, 1999 47(1): 187–200

    Article  MATH  MathSciNet  Google Scholar 

  21. Huggins P S, Zucker S W. Greedy basis pursuit. IEEE Trans Signal Process, 2007 55(7): 3760–3772

    Article  MathSciNet  Google Scholar 

  22. Reeves T H, Kingsbury N G. Overcomplete image coding using iterative projection-based noise shaping. In: Proc. International Conference on Image Processing, Rochester, NY, 2002. 597–600

  23. Fischer S, Cristóbal G. Minimum entropy transform using Gabor wavelets for image compression. In: Proc. International Conference on Image Analysis and Processing, Palermo, Italy, 2001. 428–433

  24. Fadili M J, Starck J L, Murtagh F. Inpainting and zooming using sparse representations. Computer J, 2007, doi:10.1093/comjnl/bxm055

  25. Pece A E C. The problem of sparse coding. J Math Imag Vision, 2002 17: 89–108

    Article  MATH  MathSciNet  Google Scholar 

  26. Zhang C, Yin Z, Chen X, et al.Signal overcomplete representation and sparse decomposition based on redundant dictionaries. Chin Sci Bull, 2005 50(23): 2672–2677

    Article  MathSciNet  Google Scholar 

  27. Donoho D, Huo X. Uncertainty principles and ideal atomic decompositions. IEEE Trans Information Theory, 2001 47(7): 2845–2862

    Article  MATH  MathSciNet  Google Scholar 

  28. Donoho D, Tsaig Y, Drori I, et al. Sparse solution of underdetermined linear equations of stagewise orthogonal matching pursuit. Technical Report, Department of Statistics, Stanford University, 2006

  29. Neff R, Zakhor A. Matching pursuit video coding: I. dictionary approximation. IEEE Trans Circuits Syst Video Tech, 2002 12(1): 13–26

    Article  Google Scholar 

  30. Ventura R F, Vandergheynst P, Frossard P. Low-rate and flexible image coding with redundant representations. IEEE Trans Image Process, 2006 15(3): 726–739

    Article  Google Scholar 

  31. Gribonval R. Fast matching pursuit with a multiscale dictionary of Gaussian chirps. IEEE Trans Signal Process, 2001 49(5): 994–1001

    Article  MathSciNet  Google Scholar 

  32. Donoho D L. For most large underdetermined systems of linear equations, the minimal ell-1 norm solution is also the sparsest solution. Commun Pure Appl Math, 2006 59(7): 797–829

    Article  MATH  Google Scholar 

  33. Chartrand R. Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Process Lett, 2007 14(1): 707–710

    Article  Google Scholar 

  34. Donoho D. De-noising by soft-thresholding. IEEE Trans Inf Theory, 1995 41(3): 613–627

    Article  MATH  MathSciNet  Google Scholar 

  35. Daubechies I, Defrise M, De Mol C. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun Pure Appl Math, 2004 57(11): 1413–1457

    Article  MATH  Google Scholar 

  36. Daubechies I, Teschke G., Vese L. Iteratively solving linear inverse problems under general convex constraints. Inverse Problem Imag, 2007 1(1): 29–46

    MATH  MathSciNet  Google Scholar 

  37. Yang J Y, Peng Y G, Xu W L, et al.Way to sparse representation: a comparative study. Tsinghua Sci Tech, 2009 14(4): in press

  38. Donoho D, Tsaig Y. Fast solution of l 1-norm minimization problems when the solution may be sparse. Technical Report, Stanford University, 2006

  39. Blumensath T, Davies M E. Iterative thresholding for sparse approximations. Technical Report, The University of Edinburgh, 2004

  40. Olshausen B A, Field D J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 1996 381: 607–609

    Article  Google Scholar 

  41. Candes E J, Donoho D. Curvelets-a surprisingly effective nonadaptive representation for objects with edges. In: Laurent P J, Le Mehaute A, Schumaker L L, eds. Curves and Surfaces. Nashville: Vanderbilt University Press, TN, 2000. 105–120

    Google Scholar 

  42. Pennec E L, Mallat S. Sparse geometric image representations with bandelets. IEEE Trans Image Process, 2005 14: 423–438

    Article  MathSciNet  Google Scholar 

  43. Do M N, Vetterli M. The contourlet transform: an efficient directional multiresolution image representation. IEEE Trans Image Process, 2005 14(12): 2091–2106

    Article  MathSciNet  Google Scholar 

  44. Gribonval R, Nielsen R. Sparse decompositions in unions of bases. IEEE Trans Inf Theory, 2003 49: 3320–3325

    Article  MathSciNet  Google Scholar 

  45. Tropp J. Greed is good: Algorithmic results for sparse approximation. IEEE Trans Inf Theory, 2004 50(10): 2231–2242

    Article  MathSciNet  Google Scholar 

  46. Donoho D, Elad M. Maximal sparsity representation via l 1 minimization. In: Proc. the National Academy of Sciences, 2003 100: 2197–2202

    Article  MATH  MathSciNet  Google Scholar 

  47. Elad M, Bruckstein A M. A generalized uncertainty principle and sparse representation in pairs of bases. IEEE Trans Inf Theory, 2002 48: 2558–2567

    Article  MATH  MathSciNet  Google Scholar 

  48. Herrity K K, Gilbert A C, Tropp J A. Sparse approximation via iterative thresholding. In: Proc. International Conference on Acoustics, Speech and Signal Processing, 2006, 3: 624–627

    Google Scholar 

  49. Sharon Y, Wright J, Ma Y. Computation and relaxation of conditions for equivalence between l1 and l0 minimization. Technical Report, University of Illinois, 2007

  50. Donoho D, Elad M, Temlyakov V. Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Trans Inf Theory, 2006 52: 6–18

    Article  MathSciNet  Google Scholar 

  51. Fadili M J, Starck J L. Sparse representation-based image deconvolution by iterative thresholding. Astronomical Data Analysis, France: Marseille, 2006

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Automation, Tsinghua University, Beijing, 100084, China

    JingYu Yang, YiGang Peng, WenLi Xu & QiongHai Dai

Authors
  1. JingYu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. YiGang Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. WenLi Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. QiongHai Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to JingYu Yang.

Additional information

Supported by the Joint Research Fund for Overseas Chinese Young Scholars of the National Natural Science Foundation of China (Grant No. 60528004) and the Key Project of the National Natural Science Foundation of China (Grant No. 60528004)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Yang, J., Peng, Y., Xu, W. et al. Ways to sparse representation: An overview. Sci. China Ser. F-Inf. Sci. 52, 695–703 (2009). https://doi.org/10.1007/s11432-009-0045-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11432-009-0045-5

Keywords

Search

Navigation