Skip to main content
SpringerLink
Log in

The Problem of Sparse Image Coding

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

Linear expansions of images find many applications in image processing and computer vision. Overcomplete expansions are often desirable, as they are better models of the image-generation process. Such expansions lead to the use of sparse codes. However, minimizing the number of non-zero coefficients of linear expansions is an unsolved problem. In this article, a generative-model framework is used to analyze the requirements, the difficulty, and current approaches to sparse image coding.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. H. Attias, “Independent factor analysis,” Neural Comp., Vol. 11, No. 4, pp. 803–851, 1999.

    Google Scholar 

  2. F. Attneave, “Informational aspects of visual perception,” Psychol. Review, Vol. 61, pp. 183–193, 1954.

    Google Scholar 

  3. H.B. Barlow, “Possible principles underlying the transformation of sensory messages,” in Sensory Communication, W. Rosenblith (Ed.), PMIT Press: Cambridge, 1961, pp. 217–234.

    Google Scholar 

  4. H.B. Barlow, “Single units and sensation: A neuron doctrine for perceptual psychology?” Perception, Vol. 1, pp. 371–394, 1972.

    Google Scholar 

  5. H.B. Barlow, “Unsupervised learning,” Neural Comp., Vol. 1, pp. 295–311, 1989.

    Google Scholar 

  6. H.B. Barlow, “What is the computational goal of the neocortex?” in Large Scale Neuronal Theories of the Brain, C. Koch (Ed.), MIT Press: Cambridge, MA, 1994, pp. 1–22.

    Google Scholar 

  7. E.B. Baum, J. Moody, and F. Wilczeck, “Internal representations for associative memory,” Biol. Cybern., Vol. 59, pp. 217–228, 1988.

    Google Scholar 

  8. A.J. Bell and T.J. Sejnowski, “An information-maximization approach to blind separation and blind deconvolution,” Neural Computation, Vol. 7, pp. 1129–1159, 1995.

    Google Scholar 

  9. A.J. Bell and T.J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Res., Vol. 37, pp. 3327–3338, 1997.

    Google Scholar 

  10. C.E. Bredfeldt and D.L. Ringach, “Dynamics of spatial frequency tuning in macaque VI,” J. Neurosci., Vol. 22, No. 5, pp. 1976–1984, 2002.

    Google Scholar 

  11. E.J. Candes and D.L. Donoho, “Curvelets—A surprisingly effective nonadaptive representation for objects with edges,” in Curves and Surfaces, L.L. Schumaker et al. (Eds.), Vanderbilt University Press, 1999.

  12. S. Celebrini, S. Thorpe, Y. Trotter, and M. Imbert, “Dynamics of orientation coding in area V1 of the awake monkey,” Visual Neurosci., Vol. 10, pp. 811–825, 1993.

    Google Scholar 

  13. S. Chen, S.A. Billings, and W. Luo, “Orthogonal least squares methods and their application to non-linear system identification,” Int. J. Control, Vol. 50, No. 5, pp. 1873–1896, 1989.

    Google Scholar 

  14. S. Chen and D.L. Donoho, “Examples of basis pursuit,” in Proc. of the SPIE, Vol. 2569, No. 2, pp. 564–574, 1995.

    Google Scholar 

  15. R. Coifman and V. Wickerhauser, “Entropy-based algorithms for best basis selection,” IEEE Trans. Info. Theory, Vol. 38, No. 2, pp. 713–718, 1992.

    Google Scholar 

  16. I. Daubechies, “The wavelet transform, time-frequency localization and signal analysis,” IEEE Trans. Info. Theory, Vol. 36, pp. 961–1005, 1990.

    Google Scholar 

  17. J.G. Daugman, “Uncertainty relations for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters,” J. Opt. Soc. of Am. A, Vol. 2, pp. 1160–1169, 1985.

    Google Scholar 

  18. G. Davis, S. Mallat, and M. Avellaneda, “Greedy adaptive approximation,” J. Constructive Approximation, Vol. 13, pp. 57–98, 1997.

    Google Scholar 

  19. D.L. Donoho, “De-noising by soft thresholding,” IEEE Trans Info. Theory, Vol. 41, pp. 613–627, 1995.

    Google Scholar 

  20. R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification, 2nd edn., Wiley: New York, 2001.

    Google Scholar 

  21. D.J. Field, “Relations between the statistics of natural images and the response properties of cortical cells,” J. Opt. Soc. Am. A, Vol. 4, pp. 2379–2394, 1987.

    Google Scholar 

  22. D.J. Field, “What is the goal of sensory coding?” Neural Comp., Vol. 6, pp. 559–601, 1994.

    Google Scholar 

  23. P. Foldiak, “Adaptive network for optimal linear feature extraction,” in Proc. Int. Joint Conf. Neural Net., Washington, DC, June 18-22, 1989, pp. 401–405.

  24. P. Foldiak and M. Young, “Sparse coding in the primate cortex,” in The Handbook of Brain Theory and Neural Networks, Michael A. Arbib (Ed.), 1995, pp. 895–898. The Problem of Sparse Image Coding 107

  25. J.H. Friedman and W. Stuetzle, “Projection pursuit regression,” J. Am. Stat. Assoc., Vol. 76, No. 376, pp. 817–823, 1981.

    Google Scholar 

  26. E.I. George and R.E. McCulloch, “Approaches for Bayesian variable selection,” Statistica Sinica, Vol. 7, pp. 339–373, 1997.

    Google Scholar 

  27. A. Gersho and R.M. Gray, Vector Quantization and Signal Compression, Kluwer Academic Publishers: Boston, MA, 1992.

    Google Scholar 

  28. G. Golub and C. van Loan, Matrix Computations, 3rd edn., Johns Hopkins University Press: Baltimore, 1996.

    Google Scholar 

  29. G.F. Harpur and R.W. Prager, “Development of low entropy coding in a recurrent network,” Network, Vol. 7, No. 2, pp. 277–284, 1996.

    Google Scholar 

  30. P.A.d.F.R. Højen-Sørensen, O. Winther, and L.K. Hansen, “Mean field approaches to independent component analysis,” Neural Computation, Vol. 14, pp. 889–918, 2002.

    Google Scholar 

  31. J. Huang and D. Mumford, “Statistics of natural images and models,” in Proc. IEEE Conf. Computer Vis. Pattern Rec., Fort Collins, CO, 2000, Vol. 1, pp. 541–547.

    Google Scholar 

  32. A. Hyvärinen, “Gaussian moments for noisy independent component analysis,” IEEE Signal Proc. Letters, Vol. 6, No. 6, pp. 145–147, 1999.

    Google Scholar 

  33. A. Hyvärinen, P.O. Hoyer, and M. Inki, “Topographic independent component analysis,” Neural Computation, Vol. 13, No. 7, pp. 1525–1558, 2001.

    Google Scholar 

  34. A. Hyvärinen and M. Inki, “Estimating overcomplete independent component bases for image windows,” Vol. 17, No. 2, pp. 139–152, 2002.

    Google Scholar 

  35. A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis, Wiley: New York, 2001.

    Google Scholar 

  36. S. Jaggi, W. Karl, S. Mallat, and A. Willsky, “Silhouette recognition using high-resolution pursuit,” Pattern Recognition,Vol. 32, No. 5, pp. 753–771, 1999.

    Google Scholar 

  37. J. Jones and L. Palmer, “An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex,” J. Neurophysiol., Vol. 58, No. 6, pp. 1233–1258, 1987.

    Google Scholar 

  38. R.E. Kalman and R.S. Bucy, “New results in linear filtering and prediction theory,” Trans. Am. Soc. Mech. Eng. D and J. Basic Eng., Vol. 83, pp. 95–108, 1961.

    Google Scholar 

  39. S. Kaski, “Dimensionality reduction by random mapping: Fast similarity computation for clustering,” in Proc. of IEEE Int. Joint Conf. Neural Net., IJCNN'98, Anchorage, Alaska, 1998, Vol. 1, pp. 413–418.

    Google Scholar 

  40. F.P. Kourouniotis, R.F. Kubichek, N. Boyd, III, and A.K. Majumdar, “Application of the wavelet transform and matching pursuit algorithm in seismic data processing for the development of new noise reduction techniques,” in Int. Symposium on Optical Sci. Eng. and Instrumentation, Denver, CO, Aug. 1996.

  41. D.D. Lee and H.S. Seung, “Learning the parts of objects by nonnegative matrix factorization,” Nature, Vol. 401, pp. 788–791, 1999.

    Google Scholar 

  42. M. Lewicki and B. Olshausen, “A probabilistic framework for the adaptation and comparison of image codes,” J. Opt. Soc. Am. A, Vol. 16, No. 7, pp. 1587–1601, 1998.

    Google Scholar 

  43. S. Mallat, “A theory for multiresolution signal decomposition: The wavelet representation,” IEEE Trans. Pat. Anal. Mach. Intell., Vol. 11, pp. 674–693, 1989.

    Google Scholar 

  44. S. Mallat, A Wavelet Tour of Image Processing, Academic Press: San Diego, 1999.

    Google Scholar 

  45. S. Mallat and Z. Zhang, “Matching pursuit with time-frequency dictionaries,” IEEE Trans. Signal Processing, Vol. 41, No. 12, pp. 3397–3415, 1993.

    Google Scholar 

  46. A.J. Miller, Subset Selection in Regression, Chapman and Hall: London, 1990.

    Google Scholar 

  47. J. Miskin and D. MacKay, “Ensemble learning for blind source separation,” in Independent Component Analysis: Principle and Practice, S. Roberts and R. Everson (Eds.), Cambridge UniversityPress, 2001, pp. 209–233.

  48. T.J. Mitchell and J.J. Beauchamp, “Bayesian variable selection in linear regression,” J. Am. Stat. Assoc.,Vol. 83, pp. 1023–1032, 1988.

    Google Scholar 

  49. D. Mumford and B. Gidas, “Stochastic models for generic images,” Quarterly of Applied Mathematics, Vol. 59, No. 11, pp. 85–111, 2001.

    Google Scholar 

  50. P.C. Murphy and A.M. Sillito, “Corticofugal feedback influences the generation of length tuning in the visual pathway,” Nature, Vol. 329, pp. 727–729, 1987.

    Google Scholar 

  51. M. Nafie, M. Ali, and A.H. Tewfik, “Optimal subset selection for adaptive signal representation,” in Proc. of the IEEE Int. Conf. on Acoust. Speech and Signal Proc., 1996, pp. 2511–2514.

  52. B.K. Natarajan, “Sparse approximate solutions to linear systems,” SIAM J. Computing, Vol. 24, No. 2, pp. 227–234, 1995.

    Google Scholar 

  53. R. Neff and A. Zakhor, “Very low bit-rate video coding based on matching pursuits,” IEEE Trans. Circuits and Syst. for Video Technology, Vol. 7, No. 1, pp. 158–171, 1997.

    Google Scholar 

  54. A.N. Netravali and B.G. Haskell, Digital Pictures: Representation and Compression, Plenum Press: New York, 1988.

    Google Scholar 

  55. B.A. Olshausen and D.J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature, Vol. 381, pp. 607–609, 1996.

    Google Scholar 

  56. B.A. Olshausen and D.J. Field, “Sparse coding with an overcomplete basis set: A strategy employed by V1?” Vision Res., Vol. 37, pp. 3311–3325, 1997.

    Google Scholar 

  57. B.A. Olshausen and K.J. Millman, “Learning sparse codes with a mixture-of-Gaussians prior,” in Advances in Neural Information Processing Systems 12, S.A. Solla, T.K. Leen, and K.R. Muller, (Eds.), MIT Press: Cambridge, MA, 2000, pp. 841–847.

    Google Scholar 

  58. B.A. Olshausen, P. Sallee, and M.S. Lewicki, “Learning sparse image codes using a wavelet pyramid architecture,” in Advances in Neural Information Processing Systems 13, T.K. Leen, T.G. Dietterich, V. Tresp (Eds.), MIT Press: Cambridge, MA, 2000, pp. 887–893.

    Google Scholar 

  59. G. Palm, “On associative memory,” Biol. Cybern., Vol. 36, pp. 19–31, 1980.

    Google Scholar 

  60. A.E.C. Pece, “Redundancy reduction of a Gabor representation: A possible computational role for feedback from primary visual cortex to lateral geniculate nucleus,” in Artificial Neural Networks 2, I. Aleksander and J. Taylor (Eds.), Elsevier Science Publishers: Amsterdam, 1992, pp. 865–868.

    Google Scholar 

  61. A.E.C. Pece and N. Petkov, “Fast atomic decomposition by the inhibition method,” in Proceedings of the 15th International Conference on Pattern Recognition (ICPR 2000), Barcelona, Spain, Sept. 2-8, 2000, pp. 215–218.

  62. P.J. Phillips, “Matching pursuit filters applied to face identification,” IEEE Trans. Image Proc., Vol. 7, No. 8, pp. 1150–1164, 1998.

    Google Scholar 

  63. M.D. Plumbley, “Information processing in negative feedback neural networks,” Network, Vol. 7, No. 2, pp. 301–305, 1996.

    Google Scholar 

  64. W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery, Numerical Recipes in C, 2nd edn., Cambridge University Press: Cambridge, 1992. 108 Pece

    Google Scholar 

  65. J.G. Robson, “Frequency domain visual processing,” in Physical and Biological Processing of Images, O.J. Braddick and A.C. Sleigh (Eds.), Springer, 1983, pp. 73–87.

  66. S. Roweis and Z. Ghahramani, “A unifying review of linear Gaussian models,” Neural Computation,Vol. 11, No. 2, pp. 305–345, 1999.

    Google Scholar 

  67. D.L. Ruderman, “The statistics of natural images,” Network, Vol. 5, pp. 517–548, 1994.

    Google Scholar 

  68. A. Said and W.A. Pearlman, “A new, fast, and efficient image codec based on set partitioning in hierarchical trees,” IEEE Trans. Circuits and Syst. for Video Technol., Vol. 6, No. 3, pp. 243–250, 1996.

    Google Scholar 

  69. A. Shmilovici and O. Maimon, “Application of adaptive matching pursuit to adaptive control of nonlinear dynamic systems,” IEE Proceedings—Control Theory and Application, Vol. 145, No. 6, pp. 575–582, 1998.

    Google Scholar 

  70. E.P. Simoncelli and E.H. Adelson, “Noise removal via bayesian wavelet coring,” in Proc. Int. Conf. Im. Proc., Lausanne (CH), Sept. 1996, pp. 379–383.

  71. E.P. Simoncelli, W.T. Freeman, E.H. Adelson, and D.J. Heeger, “Shiftable multiscale transforms,” IEEE Trans. Info. Theory, Vol. 38, No. 2, pp. 587–607, 1992.

    Google Scholar 

  72. M.J. Wainwright, E.P. Simoncelli, and A.S. Willsky, “Random cascades on wavelet trees and their use in modeling and analyzing natural images,” Applied and Computational Harmonic Analysis, Vol. 11, No. 1, pp. 89–123, 2001.

    Google Scholar 

  73. S. Watanabe, Pattern Recognition: Human and Mechanical, Wiley: New York, 1985.

    Google Scholar 

  74. D.A. Winfeld, K.C. Gatter, and T.P.S. Powell, “An electron microscopic study of the types and proportions of neurons in the cortex of the motor and visual areas of the cat and rat,” Brain, Vol. 103, pp. 245–258, 1990.

    Google Scholar 

  75. Y.N. Wu, S.C. Zhu, and C. Guo, “Statistical modeling of texture sketch,” in Proceedings of the European Conference on Computer Vision (ECCV 2002), Springer, 2002.

  76. H. Yoshida, “Matching pursuit with optimally weighted wavelet packets for extraction of microcalcifications in mammograms,” Applied Signal Processing, Vol. 5, No. 3, pp. 127–141, 1999.

    Google Scholar 

  77. C. Zetzsche, “Sparse coding: The link between low level vision and associative memory,” in Parallel Processing in Neural Systems and Computers, R. Eckmiller, G. Hartmann, and G. Hauske (Eds.), North-Holland: Amsterdam, 1990, pp. 273–276.

    Google Scholar 

  78. M. Zibulevsky and B.A. Pearlmutter, “Blind source separation by sparse decomposition in a signal dictionary,” Neural Computation, Vol. 13, No. 4, pp. 863–882, 2001.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Computer Science, University of Copenhagen, Universitetsparken 1, DK-2100, Copenhagen, Denmark

    Arthur E.C. Pece

Authors
  1. Arthur E.C. Pece
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pece, A.E. The Problem of Sparse Image Coding. Journal of Mathematical Imaging and Vision 17, 89–108 (2002). https://doi.org/10.1023/A:1020677318841

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1020677318841

Search

Navigation