Skip to main content
SpringerLink
Log in

Estimation of generalized entropies with sample spacing

  • Theoretical Advances
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

A Publisher’s Erratum to this article was published on 13 October 2005

Abstract

In addition to the well-known Shannon entropy, generalized entropies, such as the Renyi and Tsallis entropies, are increasingly used in many applications. Entropies are computed by means of nonparametric kernel methods that are commonly used to estimate the density function of empirical data. Generalized entropy estimation techniques for one-dimensional data using sample spacings are proposed. By means of computational experiments, it is shown that these techniques are robust and accurate, compare favorably to the popular Parzen window method for estimating entropies, and, in many cases, require fewer computations than Parzen methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Silverman BW (1986) Density estimation for statistics and data analysis. Chapman and Hall, London

    Google Scholar 

  2. Renyi A (1970) Probability theory. North-Holland, Amsterdam

    Google Scholar 

  3. Tsallis C (1988) Possible generalization of Boltzmann-Gibbs statistics. J Stat Phys 52:479–487

    Article  Google Scholar 

  4. Smolíkoví R, Wachowiak MP, Zurada JM (2004) An information-theoretic approach to estimating ultrasound backscatter characteristics. Comput Biol Med 34:355–370

    PubMed  Google Scholar 

  5. Krishnamachari A, Mandal VM, Karmeshu (2004) Study of DNA binding sites using the Renyi parametric entropy measure. J Theor Biol 27:429–436

    MathSciNet  Google Scholar 

  6. Tonga S, Bezerianosa A, Amit Malhotraa A, Zhub Y, Thakor N (2003) Parameterized entropy analysis of EEG following hypoxic-ischemic brain injury. Phys Lett A 314:354–361

    Article  Google Scholar 

  7. Havrda J, Charvát F (1967) Quantification method of classification processes: concept of structural α-entropy. Kybernetika 3:30–35

    Google Scholar 

  8. Rosso OA, Martin MT, Plastino A (2003) Brain electrical activity analysis using wavelet-based informational tools (II): Tsallis non-extensivity and complexity measures. Physica A 320:497–511

    Google Scholar 

  9. Wachavia KMP, Smolíkoví R, Peters TM (2003) Multiresolution biomedical image registration using generalized information measures. Lecture notes in computer science 2899 (MICCAI 2003), pp 846–853

  10. Vasicek O (1976) A test for normality based on sample entropy. J Roy Stat Soc B 38:54–59

    Google Scholar 

  11. Dudewicz E, van der Meulen EC (1987) The empiric entropy, a new approach to nonparametric entropy estimation. In: Puri ML, Vilaplana JP, Wertz W (eds) New perspectives in theoretical and applied statistics. Wiley, NY

    Google Scholar 

  12. van Es B (1992) Estimating functionals related to a density by a class of statistics based on spacings. Scand J Stat 19:61–72

    Google Scholar 

  13. Correa JC (1995) A new estimator of entropy. Commun Stat Theor 24:2439–2449

    MathSciNet  Google Scholar 

  14. Beirlant J, Dudewicz E, Gyorfi L, van der Meulen EC (1997) Nonparametric entropy estimation: an overview. Int J Math Stat Sci 6(1):17–39

    Google Scholar 

  15. Wieczorkowski R, Grzegorzewski P (1999) Entropy estimators—improvements and comparisons. Commun Stat Simul 28(2):541–567

    MathSciNet  Google Scholar 

  16. Grassberger P. Entropy estimates from insufficient samplings. ArXiv:physics/0307138 2003

  17. Hero A, Ma B, Michel O, Gorman J (2002) Applications of entropic spanning graphs. IEEE Signal Proc Mag 19(5):85–95

    Article  Google Scholar 

  18. Erdogmus D, Principe JC (2001) Entropy minimization algorithm for multilayer perceptrons. In: Proceedings of INNS-IEEE conference on neural networks (IJCNN), Washington, DC, pp 3003–3008

  19. Holste D, Grosse I, Herzel H (1998) Bayes’ estimators of generalized entropies. J Phys A 31:2551–2566

    Google Scholar 

  20. Wolpert DH, Wolf DR (1995) Estimating functions of probability distributions from a finite set of samples. Phys Rev E 52(6):6841–6854

    MathSciNet  Google Scholar 

  21. Paninski L (2003) Estimation of entropy and mutual information. Neural Comput 15:1191–1253

    Article  Google Scholar 

  22. Tadikamalla PR (1980) Random sampling from the exponential power distribution. J Am Stat Assoc 75(371):683–686

    Google Scholar 

  23. Golan A, Perloff JM (2002) Comparison of maximum entropy and higher-order entropy estimators. J Econom 107(1–2):195–211

    Article  Google Scholar 

  24. Eggermont PB, LaRiccia VN (1999) Best asymptotic normality of the kernel density entropy estimator for smooth densities. IEEE T Inform Theory 45(4):1321–1325

    Article  Google Scholar 

Download references

Acknowledgments

The authors thank the anonymous reviewers for helpful criticisms and suggestions.

Author information

Authors and Affiliations

  1. Imaging Research Laboratories, Robarts Research Institute, London, ON, N6A 5K8, Canada

    Mark P. Wachowiak & Renata Smolíková

  2. Department of Computer Engineering and Computer Science, University of Louisville, Louisville, KY, 40292, USA

    Georgia D. Tourassi & Adel S. Elmaghraby

Authors
  1. Mark P. Wachowiak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Renata Smolíková
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Georgia D. Tourassi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Adel S. Elmaghraby
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mark P. Wachowiak.

Additional information

An erratum to this article is available at http://dx.doi.org/10.1007/s10044-005-0012-8.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wachowiak, M.P., Smolíková, R., Tourassi, G.D. et al. Estimation of generalized entropies with sample spacing. Pattern Anal Applic 8, 95–101 (2005). https://doi.org/10.1007/s10044-005-0247-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-005-0247-4

Keywords

Search

Navigation