Skip to main content
SpringerLink
Log in

Estimating Overcomplete Independent Component Bases for Image Windows

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

Estimating overcomplete ICA bases for image windows is a difficult problem. Most algorithms require the estimation of values of the independent components which leads to computationally heavy procedures. Here we first review the existing methods, and then introduce two new algorithms that estimate an approximate overcomplete basis quite fast in a high-dimensional space. The first algorithm is based on the prior assumption that the basis vectors are randomly distributed in the space, and therefore close to orthogonal. The second replaces the conventional orthogonalization procedure by a transformation of the marginal density to gaussian.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. S.-I. Amari, A. Cichocki, and H.H. Yang, “A new learning algorithm for blind source separation,” in Advances in Neural Information Processing Systems 8, MIT Press, 1996, pp. 757–763.

  2. H. Attias, “Independent factor analysis,” Neural Computation, Vol. 11, No. 4, pp. 803–851, 1999.

    Google Scholar 

  3. A.J. Bell and T.J. Sejnowski, “An information-maximization approach to blind separation and blind deconvolution,” Neural Computation, Vol. 7, pp. 1129–1159, 1995.

    Google Scholar 

  4. A.J. Bell and T.J. Sejnowski, “The ‘independent components’ of natural scenes are edge filters,” Vision Research, Vol. 37, pp. 3327–3338, 1997.

    Google Scholar 

  5. J.-F. Cardoso and B. Hvam Laheld, “Equivariant adaptive source separation,” IEEE Trans. on Signal Processing, Vol. 44, No. 12, pp. 3017–3030, 1996.

    Google Scholar 

  6. P. Comon, “Independent component analysis: A new concept?” Signal Processing, Vol. 36, pp. 287–314, 1994.

    Google Scholar 

  7. N. Delfosse and P. Loubaton, “Adaptive blind separation of independent sources: A deflation approach,” Signal Processing, Vol. 45, pp. 59–83, 1995.

    Google Scholar 

  8. J.H. Friedman, “Exploratory projection pursuit,” J. American Statistical Association, Vol. 82, No. 397, pp. 249–266, 1987.

    Google Scholar 

  9. P.J. Huber, “Projection pursuit,” The Annals of Statistics,Vol. 13, No. 2, pp. 435–475, 1985.

    Google Scholar 

  10. A. Hyvärinen, “Independent component analysis in the presence of gaussian noise by maximizing joint likelihood,” Neurocomputing, Vol. 22, pp. 49–67, 1998.

    Google Scholar 

  11. A. Hyvärinen, “Fast and robust fixed-point algorithms for independent component analysis,” IEEE Trans. on Neural Networks, Vol. 10, No. 3, pp. 626–634, 1999.

    Google Scholar 

  12. A. Hyvärinen, “Sparse code shrinkage: Denoising of nongaussian data by maximum likelihood estimation,” Neural computation, Vol. 11, No. 7, pp. 1739–1768, 1999.

    Google Scholar 

  13. A. Hyvärinen, R. Cristescu, and E. Oja, “A fast algorithm for estimating overcomplete ICA bases for image windows,” in Proc. Int. Joint Conf. on Neural Networds, Washington, D.C., 1999, pp. 894–899.

  14. A. Hyvärinen and P.O. Hoyer, “Emergence of phase and shift invariant features by decomposition of natural images into independent feature sub-spaces,” Neural Computation, Vol. 12, No. 7, pp. 1705–1720, 2000.

    Google Scholar 

  15. A. Hyvärinen, P.O. Hoyer, and M. Inki, “Topographic independent component analysis,” Neural Computation, Vol. 13, No. 7, pp. 1527–1558, 2001.

    Google Scholar 

  16. A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis, Wiley Interscience, 2001.

  17. A. Hyvärinen and E. Oja, “Independent component analysis by general nonlinear Hebbian-like learning rules,” Signal Processing, Vol. 64, No. 3, pp. 301–313, 1998.

    Google Scholar 

  18. A. Hyvärinen and E. Oja, “Independent component analysis: Algorithms and applications,” Neural Networks, Vol. 13, Nos. 4, 5, pp. 411–430, 2000.

    Google Scholar 

  19. M. Inki and A. Hyvärinen, “Two methods for estimating overcomplete independent component basis,” in Proc. Int.Workshop on Independent Component Analysis and Blind Signal Separation (ICA2001), San Diego, California, 2001.

  20. M. Inki and A. Hyvärinen, “Two approaches to estimation of overcomplete independent component bases,” in Proc. Int. Joint Conference on Neural Networks (IJCNN 2002), Honolulu, Hawaii, 2002.

  21. C. Jutten and J. Hérault, “Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture,” Signal Processing, Vol. 24, pp. 1–10, 1991.

    Google Scholar 

  22. P. Kanerva, Sparse Distributed Memory, MIT Press, 1998.

  23. J. Karhunen, E. Oja, L Wang, R. Vigärio, and J. Joutsensalo, “A class of neural networks for independent component analysis,” IEEE Trans. on Neural Networks, Vol. 8, No. 3, pp. 486–504, 1997.

    Google Scholar 

  24. S. Kaski, “Dimensionality reduction by random mapping: Fast similarity computation for clustering,” in Proc. Int. Joint Conf. on Neural Networks (IJCNN'98), Anchorage, Alaska, 1998, pp. 413–418.

  25. T. Kohonen, Self-Organizing Maps, Springer, 1995.

  26. H. Lappalainen and A. Honkela, “Bayesian nonlinear independent component analysis by multi-layer perceptrons,” in Advances in Independent Component Analysis, M. Girolami (Ed.), Springer-Verlag: Berlin, 2000, pp. 93–121.

    Google Scholar 

  27. T.-W. Lee, M.S. Lewicki, M. Girolami, and T.J. Sejnowski, “Blind source separation of more sources than mixtures using overcomplete representations,” IEEE Signal Processing Letters, Vol. 4, No. 5, 1999.

  28. M. Lewicki and B. Olshausen, “A probabilistic framework for the adaptation and comparison of image codes,” J. Opt. Soc. Am. A: Optics, Image Science, and Vision, Vol. 16, No. 7, pp. 1587–1601, 1998.

    Google Scholar 

  29. M. Lewicki and T.J. Sejnowski, “Learning overcomplete representations,” Neural Computation, Vol. 12, pp. 337–365, 2000.

    Google Scholar 

  30. E. Moulines, J.-F. Cardoso, and E. Gassiat, “Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP'97), Munich, Germany, 1997, pp. 3617–3620.

  31. E. Oja, “The nonlinear PCA learning rule in independent component analysis, Neurocomputing, Vol. 17, No. 1, pp. 25–46, 1997.

    Google Scholar 

  32. B.A. Olshausen and D.J. Field, “Emergence of simple-cell receptive field properties by learning a sparse code for natural images,” Nature, Vol. 381, pp. 607–609, 1996.

    Google Scholar 

  33. B.A. Olshausen and D.J. Field, “Sparse coding with an overcomplete basis set: A strategy employed by V1?” Vision Research, Vol. 37, pp. 3311–3325, 1997.

    Google Scholar 

  34. B.A. Olshausen and K.J. Millman, “Learning sparse codes with a mixture-of-gaussians prior,” in Advances in Neural Information Processing Systems, Vol. 12, MIT Press, 2000, pp. 841–847.

    Google Scholar 

  35. P. Pajunen, “Blind separation of binary sources with less sensors than sources,” in Proc. Int. Conf. on Neural Networks, Houston, Texas, 1997.

  36. A. Pece, “The problem of sparse image coding,” Journal of Mathematical Imaging and Vision, 2002. In this issue.

  37. E.P. Simoncelli, W.T. Freeman, E.H. Adelson, and D.J. Heeger, “Shiftable multiscale transforms,” IEEE Transactions on Information Theory, Vol. 38, pp. 587–607, 1992.

    Google Scholar 

  38. E.P. Simoncelli and O. Schwartz, “Modeling surround suppression in V1 neurons with a statistically-derived normalization model,” in Advances in Neural Information Processing Systems 11, MIT Press, 1999, pp. 153–159.

  39. C. Zetzsche and G. Krieger, “Nonlinear neurons and high-order statistics: Newapproaches to human vision and electronic image processing,” in Human Vision and Electronic Imaging IV (Proc. SPIE Vol. 3644), B. Rogowitz and T.V. Pappas (Eds.), SPIE, 1999, pp. 2–33.

Download references

Author information

Authors and Affiliations

  1. Neural Networks Research Centre, Helsinki University of Technology, P.O. Box 9800, FIN-02015, HUT, Finland

    Aapo Hyvärinen & Mika Inki

Authors
  1. Aapo Hyvärinen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mika Inki
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hyvärinen, A., Inki, M. Estimating Overcomplete Independent Component Bases for Image Windows. Journal of Mathematical Imaging and Vision 17, 139–152 (2002). https://doi.org/10.1023/A:1020633503820

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1020633503820

Search

Navigation