Skip to main content
SpringerLink
Log in

Gaussianization: An Efficient Multivariate Density Estimation Technique for Statistical Signal Processing

  • Published:
Journal of VLSI signal processing systems for signal, image and video technology Aims and scope Submit manuscript

Abstract

Multivariate density estimation is an important problem that is frequently encountered in statistical learning and signal processing. One of the most popular techniques is Parzen windowing, also referred to as kernel density estimation. Gaussianization is a procedure that allows one to estimate multivariate densities efficiently from the marginal densities of the individual random variables. In this paper, we present an optimal density estimation scheme that combines the desirable properties of Parzen windowing and Gaussianization, using minimum Kullback–Leibler divergence as the optimality criterion for selecting the kernel size in the Parzen windowing step. The utility of the estimate is illustrated in classifier design, independent components analysis, and Prices’ theorem.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. R. O. Duda, P. E. Hart, and D. G. Stork, “Pattern Classification,” 2nd ed., Wiley, New York, 2001.

    MATH  Google Scholar 

  2. S. Theodoridis and K. Koutroumbas, “Pattern Recognition,” Academic, New York, 2003.

    Google Scholar 

  3. L. Devroye and G. Lugosi, “Combinatorial Methods in Density Estimation,” Springer, Berlin Heidelberg New York, 2001.

    MATH  Google Scholar 

  4. A. Papoulis, “Probability, Random Variables, Stochastic Processes,” McGraw-Hill, New York, 1991.

    Google Scholar 

  5. T. M. Cover and J. A. Thomas, “Elements of Information Theory,” Wiley, New York, 1991.

    MATH  Google Scholar 

  6. E. Parzen, “On Estimation of a Probability Density Function and Mode,” in Time Series Analysis Papers, Holden-Day, CA, 1967.

  7. R. Jenssen, D. Erdogmus, J. C. Principe, and T. Eltoft, “Towards a Unification of Information Theoretic Learning and Kernel Methods,” in Proceedings of MLSP’04, Sao Luis, Brazil, 2004.

  8. K. E. Hild II, D. Erdogmus, and J. C. Principe, “Blind Source Separation Using Renyi’s Mutual Information,” in IEEE Signal Processing Letters, no. 8, 2001, pp. 174–176.

  9. K. Torkkola, “Visualizing Class Structure in Data Using Mutual Information,” in Proceedings of NNSP’00, Sydney, Australia, 2000, pp. 376–385.

  10. D. Erdogmus, “Information Theoretic Learning: Renyi’s Entropy and its Applications to Adaptive System Training,” Ph.D. Dissertation, University of Florida, Gainesville, Florida, 2002.

  11. M. M. Van Hulle, “Kernel-Based Topographic Map Formation Achieved with an Information–Theoretic Approach,” Neural Netw., vol. 15, 2002, pp. 1029–1039.

    Article  Google Scholar 

  12. N. N. Schraudolph, “Gradient-Based Manipulation of Nonparametric Entropy Estimates,” IEEE Trans. Neural Netw., vol. 15, no. 4, 2004, pp. 828–837.

    Article  Google Scholar 

  13. K. Fukunaga, “Statistical Pattern Recognition,” Academic, New York, 1990.

    MATH  Google Scholar 

  14. B. W. Silverman, “Density Estimation for Statistics and Data Analysis,” Chapman & Hall, London, 1986.

    MATH  Google Scholar 

  15. M. C. Jones, J. S. Marron, and S. J. Sheather, “A Brief Survey of Bandwidth Selection for Density Estimation,” J. Am. Stat. Assoc., vol. 87, 1996, pp. 227–233.

    MathSciNet  Google Scholar 

  16. R. P. W. Duin, “On the Choice of the Smoothing Parameters for Parzen Estimators of Probability Density Functions,” IEEE Trans. Comput., vol. 25, no. 11, 1976, pp. 1175–1179.

    MathSciNet  Google Scholar 

  17. S. Amari, “Differential–Geometrical Methods in Statistics,” Springer, Berlin Heidelberg New York, 1985.

    MATH  Google Scholar 

  18. P. Viola, N. Schraudolph, and T. Sejnowski, “Empirical Entropy Manipulation for Real-World Problems,” in Proceedings of NIPS’95, 1996, pp. 851–857.

  19. T. Bell and T. Sejnowski, “An Information–Maximization Approach to Blind Separation and Blind Deconvolution,” Neural Comput., vol. 7, 1995, pp. 1129–1159.

    Google Scholar 

  20. A. Hyvarinen, J. Karhunen, and E. Oja, “Independent Component Analysis,” Wiley, New York, 2001.

    Google Scholar 

  21. A. Cichocki and S. I. Amari, “Adaptive Blind Signal and Image Processing: Learning Algorithms and Applications,” Wiley, New York, 2002.

    Google Scholar 

  22. T. W. Lee, “Independent Component Analysis: Theory and Applications,” Kluwer, New York, 1998.

    MATH  Google Scholar 

  23. A. Hyvarinen, “Survey on Independent Component Analysis,” Neural Comput. Surv., vol. 2, 1999, pp. 94–128.

    Google Scholar 

  24. C. Jutten and J. Karhunen, “Advances in Nonlinear Blind Source Separation,” in Proceedings of ICA’03, Nara, Japan, 2003, pp. 245–256.

  25. J. Karhunen and J. Joutsensalo, “Representation and Separation of Signals Using Nonlinear PCA Type Learning,” Neural Netw., vol. 7, 1994, pp. 113–127.

    Article  Google Scholar 

  26. D. Erdogmus, Y. N. Rao, and J. C. Principe, “Gaussianizing Transformations for ICA,” in Proceedings of ICA’04, Granada, Spain, 2004, pp. 26–32.

  27. D. Erdogmus, Y. N. Rao, and J. C. Principe, “Nonlinear Independent Component Analysis by Homomorphic Transformation of the Mixtures,” in Proceedings of IJCNN’04, Budapest, Hungary, 2004, pp. 47–52.

  28. A. Hyvarinen and P. Pajunen, “Nonlinear Independent Component Analysis: Existence and Uniqueness Results,” Neural Netw., vol. 12, no. 3, 1999, pp. 429–439.

    Article  Google Scholar 

  29. L. B. Almeida, “MISEP—Linear and Nonlinear ICA Based on Mutual Information,” J. Mach. Learn. Res., vol. 4, 2003, pp. 1297–1318.

    Article  Google Scholar 

  30. H. Valpola, E. Oja, A. Ilin, A. Honkela, and J. Karhunen, “Nonlinear Blind Source Separation by Variational Bayesian Learning,” IEICE Trans. Fundam. Electron. Commun. Comput. Sci., vol. 86, no. 3, 2003, pp. 532–541.

    Google Scholar 

  31. L. Parra, “Symplectic Nonlinear Independent Component Analysis,” in Proceedings of NIPS’96, 1997, pp. 437–443.

  32. Y. Tan and J. Wang, “Nonlinear Blind Source Separation Using Higher Order Statistics and a Genetic Algorithm,” IEEE Trans. Evol. Comput., vol. 5, no. 6, 2001.

  33. A. Ziehe, M. Kawanabe, S. Harmeling, and K. R. Muller, “Blind Separation of Post-Nonlinear Mixtures Using Linearizing Transformations and Temporal Decorrelation,” J. Mach. Learn. Res., vol. 4, 2003, pp. 1319–1338.

    Article  MathSciNet  Google Scholar 

  34. S. Harmeling, A. Ziehe, M. Kawanabem, and K. R. Müller, “Kernel-Based Nonlinear Blind Source Separation,” Neural Comput., vol. 15, 2003, pp. 1089–1124.

    Article  Google Scholar 

  35. S. Fiori, “A Theory for Learning by Weight Flow on Stiefel–Grassman Manifold,” Neural Comput., vol. 13, 2001, pp. 1625–1647.

    Article  Google Scholar 

  36. L. Xu, “Least Mean Square Error Reconstruction Principle for Self-Organizing Neural Nets,” Neural Netw., vol. 6, 1993, pp. 627–648.

    Article  Google Scholar 

  37. R. Price, “A Useful Theorem for Nonlinear Devices Having Gaussian Inputs,” IRE Trans. Inf. Theory, vol. 4, 1958, pp. 69–72.

    Article  MathSciNet  Google Scholar 

  38. E. L. McMahon, “An Extension of Price’s Theorem,” IEEE Trans. Inf. Theory, vol. 10, 1964, p. 168.

    Article  Google Scholar 

  39. R. F. Pawula, “A Modified Version of Price’s Theorem,” IEEE Trans. Inf. Theory, vol. 13, no. 2, 1967, pp. 285–288.

    Article  MathSciNet  Google Scholar 

  40. A. Papoulis, “Comment on ‘An Extension of Price’s Theorem’,” IEEE Trans. Inf. Theory, vol. 11, 1965, p. 154.

    Article  Google Scholar 

  41. J. L. Brown, “A Generalized Form of Price’s Theorem and Its Converse,” IEEE Trans. Inf. Theory, vol. 13, no. 1, 1967, pp. 27–30.

    Article  Google Scholar 

  42. A. van den Bos, “Price’s Theorem for Complex Variates,” IEEE Trans. Inf. Theory, vol. 42, no. 1, 1996, pp. 286–287.

    Article  Google Scholar 

  43. D. McGraw and J. Wagner, “Elliptically Symmetric Distributions,” IEEE Trans. Inf. Theory, vol. 14, no. 1, 1968, pp. 110–120.

    Article  Google Scholar 

  44. S. S. Chen and R. A. Gopinath, “Gausianization,” in Proceedings of NIPS, 2000.

Download references

Author information

Authors and Affiliations

  1. CSEE Department, Oregon Health and Science University, Portland, OR, USA

    Deniz Erdogmus

  2. Department of Physics, University of Tromso, Tromso, Norway

    Robert Jenssen

  3. Motorola Corporation, Plantation, FL, USA

    Yadunandana N. Rao

  4. CNEL, ECE Department, University of Florida, Gainesville, FL, USA

    Jose C. Principe

Authors
  1. Deniz Erdogmus
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert Jenssen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yadunandana N. Rao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jose C. Principe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert Jenssen.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Erdogmus, D., Jenssen, R., Rao, Y.N. et al. Gaussianization: An Efficient Multivariate Density Estimation Technique for Statistical Signal Processing. J VLSI Sign Process Syst Sign Image Video Technol 45, 67–83 (2006). https://doi.org/10.1007/s11265-006-9772-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11265-006-9772-7

Keywords

Search

Navigation