Skip to main content
SpringerLink
Log in

Ensemble of Independent Factor Analyzers with Application to Natural Image Analysis

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

In this paper the ensemble of independent factor analyzers (EIFA) is proposed. This new statistical model assumes that each data point is generated by the sum of outputs of independently activated factor analyzers. A maximum likelihood (ML) estimation algorithm for the parameter is derived using a Monte Carlo EM algorithm with a Gibbs sampler. The EIFA model is applied to natural image data. With the progress of the learning, the independent factor analyzers develop into feature detectors that resemble complex cells in mammalian visual systems. Although this result is similar to the previous one obtained by independent subspace analysis, we observe the emergence of complex cells from natural images in a more general framework of models, including overcomplete models allowing additive noise in the observables.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Lawley, D. N., and Maxwell, A. E.: Factor Analysis as a Statistical Method. Butterworths, London, 1963.

    Google Scholar 

  2. Rubin, D. B., and Thayer, D. T.: EM algorithms for ML factor analysis, Psychometrika, 47 (1982), 69–76.

    Google Scholar 

  3. Roweis, S.: EM algorithms for PCA and SPCA. In: M. I. Jordan, M. J. Kearns, and S. A. Solla (eds), Advances in Neural Information Processing Systems 10, MIT Press, Cambridge (1998), pp. 626–632.

    Google Scholar 

  4. Tipping, M. E., and Bishop, C. M.: Mixtures of probabilistic principal component analyzers. Neural Computation, 11 (1999), 443–482.

    Google Scholar 

  5. Bell, A. J., and Sejnowski, T. J.: An information-maximization approach to blind separation and blind deconvolution. Neural Computation, 7 (1995), 1129–1159.

    Google Scholar 

  6. Lewicki, M. S., and Sejnowski, T. J.: Learning overcomplete representations. Neural Computation, 12 (2000), 337–365.

    Google Scholar 

  7. Attias, H.: Independent factor analysis. Neural Computation, 11 (1999), 803–852.

    Google Scholar 

  8. Olshausen, B. A., and Field, D. J.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381 (1996), 607–609.

    Google Scholar 

  9. Bell, A. J., and Sejnowski, T. J.: The ‘independent components’ of natural scenes are edge filters. Vision Research, 37 (1997), 3327–3338.

    Google Scholar 

  10. Lewicki, M. S., and Olshausen, B. A.: Probabilistic framework for the adaptation and comparison of image codes. J. Opt. Soc. of Am. A, 16 (1999), 1587–1607.

    Google Scholar 

  11. Olshausen, B. A., and Millman, K. J.: Learning sparse codes with a mixture-of-gaussians prior. In: S. A. Solla, T. K. Leen, and K.-R. Müller (eds), Advances in Neural Information Processing Systems 12, MIT Press, Cambridge (2000), pp. 841–847.

    Google Scholar 

  12. Hinton, G. E., Dayan, P., and Revow, M.: Modeling the manifolds of images of handwritten digits. IEEE Transactions on Neural Networks, 8 (1997), 65–74.

    Google Scholar 

  13. Ghahramani, Z., and Hinton, G. E.: The EM algorithm for mixtures of factor analyzers. Technical Report CRG-TR–96–1, University of Toronto, Dept. of Computer Science, 1997.

  14. Utsugi, A., and Kumagai, T.: Bayesian analysis of mixtures of factor analyzers. Neural Computation, 13 (2001), in press.

  15. Hyvärinen, A., and Hoyer, P.: Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces. Neural Computation, 12 (2000), 1705–1720.

    Google Scholar 

  16. Hyvärinen, A., and Hoyer, P.: Emergence of topography and complex cell properties from natural images using extensions of ICA. In: S. A. Solla, T. K. Leen, and K.-R. Müller (eds), Advances in Neural Information Processing Systems 12, MIT Press, Cambridge (2000), pp. 827–833.

    Google Scholar 

  17. Cardoso, J.-F.: Multidimensional independent component analysi. In: Proc. IEEE int. Conf. on Acoustics, Speech and Signal Processing (ICASSP'98), Seattle, 1998.

  18. Tanner, M. A.: Tools for Statistical Inference. 3rd Edn., Springer-Verlag, New York, 1996.

    Google Scholar 

  19. Clyde, M., Parmigiani, G., and Vidakovic, B.: Multiple shrinkage and subset selection in wavelets. Biometrika, 85 (1998), 391–402.

    Google Scholar 

  20. Kohonen, T.: Self-Organizing Maps. Springer, Berlin, 1995.

    Google Scholar 

  21. Ghahramani, Z., and Jordan, M. I.: Factorial hidden markov models. Machine Learning, 29 (1997), 245–273.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National Institute of Advanced Industrial Science and Technology, 1-1-1 Higashi, Tsukuba Ibaraki, 305-8566, Japan

    Akio Utsugi

Authors
  1. Akio Utsugi
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Utsugi, A. Ensemble of Independent Factor Analyzers with Application to Natural Image Analysis. Neural Processing Letters 14, 49–60 (2001). https://doi.org/10.1023/A:1011330208458

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1011330208458

Search

Navigation