Elsevier

Information Sciences

Volume 235, 20 June 2013, Pages 214-223
Information Sciences

Measures of statistical dispersion based on Shannon and Fisher information concepts

https://doi.org/10.1016/j.ins.2013.02.023Get rights and content

Abstract

We propose and discuss two information-based measures of statistical dispersion of positive continuous random variables: the entropy-based dispersion and Fisher information-based dispersion. Although standard deviation is the most frequently employed dispersion measure, we show, that it is not well suited to quantify some aspects that are often expected intuitively, such as the degree of randomness. The proposed dispersion measures are not entirely independent, though each describes the quality of probability distribution from a different point of view. We discuss relationships between the measures, describe their extremal values and illustrate their properties on the Pareto, the lognormal and the lognormal mixture distributions. Application possibilities are also mentioned.

Introduction

In recent years, information-based measures of randomness (or “regularity”) of signals have gained popularity in various branches of science [2], [3], [10], [12]. In this paper we construct measures of statistical dispersion based on Shannon and Fisher information concepts and we describe their properties and mutual relationships. The effort was initiated in [20], where the entropy-based dispersion was employed to quantify certain aspects of neuronal timing precision. Here we extend the previous effort by taking into account the concept of Fisher information (FI), which was employed in different contexts [5], [12], [33], [37], [38]. In particular, FI about the location parameter has been employed in the analysis of EEG [25], [37], of the atomic shell structure [32] (together with Shannon entropy) or in the description of variations among the two-electron correlated wavefunctions [14].

The goal of this paper is to propose different dispersion measures and to justify their usefulness. Although the standard deviation is used ubiquitously for characterization of variability, it is not well suited to quantify certain “intuitively intelligible” properties of the underlying probability distribution. For example highly variable data might not be random at all if it only consists of “very small” and “very large” values. Although the probability density function (or histogram of data) provides a complete view, one needs quantitative methods in order to make a comparison between different experimental scenarios.

The methodology investigated here does not adhere to any specific field of applications. We believe, that the general results are of interest to a wide group of researchers who deal with positive continuous random variables, in theory or in experiments.

Section snippets

Generic case: standard deviation

We consider a continuous positive random variable (r.v.) T with a probability density function (p.d.f.) f(t) and finite first two moments. Generally, statistical dispersion is a measure of “variability” or “spread” of the distribution of r.v. T, and such a measure has the same physical units as T. There are different dispersion measures described in the literature and employed in different contexts, e.g., standard deviation, inter-quartile range, mean difference or the LV coefficient [6], [8],

Extrema of variability

Generally, the value CV can be any non-negative real number, 0CV<. The lower bound, CV0, is approached by a p.d.f. highly peaked at the mean value, in the limit corresponding to the Dirac’s delta function, f(t)=δ(t-ET). There is, however, no unique upper bound distribution for which CV. For example, the p.d.f. examples analyzed in the next section allow arbitrarily high values of CV and yet their shapes are different.

Extrema of entropy and its relation to variability

The relation between CV and entropy was investigated in a series of

Lognormal and Pareto distributions

Both lognormal and Pareto distributions appear in a broad range of scientific applications [16]. The lognormal distribution is found in the description of, e.g., concentration of elements in the Earth’s crust, distribution of organisms in environment or in human medicine, see [24] for a review. The Pareto distribution is often described as an alternative model in situations similar as in the lognormal case, e.g, the sizes of human settlements, sizes of particle or allocation of wealth among

Discussion and conclusions

We propose and discuss two measures of statistical dispersion for continuous positive random variables: the entropy-based dispersion (Ch) and the Fisher information-based dispersion (CJ). Both Ch and CJ describe the overall spread of the distribution differently than the coefficient of variation. While Ch is most sensitive to the concentration of the probability mass (the predictability of random variable outcomes), CJ is sensitive to the modes of the p.d.f. or any non-smothness in the p.d.f.

Acknowledgements

This work was supported by the Institute of Physiology RVO:67985823, the Centre for Neuroscience P304/12/G069 and by the Grant Agency of the Czech Republic Projects P103/11/0282 and P103/12/P558.

References (38)

  • G.S. Bhumbra et al.

    Assessment of spike activity in the supraoptic nucleus

    J. Neuroendocrinol.

    (2004)
  • S.R. Chakravarty

    Ethical Social Index Numbers

    (1990)
  • T.M. Cover et al.

    Elements of Information Theory

    (1991)
  • H. Cramer

    Mathematical Methods of Statistics

    (1946)
  • B.C. DeBusk et al.

    Stimulus-dependent modulation of spike burst length in cat striate cortical cells

    J. Neurophysiol.

    (1997)
  • S.A. Della Pietra et al.

    Inducing features of random fields

    IEEE Trans. Pattern Anal. Mach. Int.

    (1997)
  • S. Ditlevsen et al.

    Firing variability is higher than deduced from the empirical coefficient of variation

    Neural Comput.

    (2011)
  • B.R. Frieden

    Physics from Fisher Information: A Unification

    (1998)
  • I.J. Good et al.

    Nonparametric roughness penalties for probability densities

    Biometrika

    (1971)
  • Cited by (30)

    • Total cost of risk for privatized electric power generation under pipeline vandalism

      2018, Heliyon
      Citation Excerpt :

      The lost MWh standard deviation, δ, of the case study plant as calculated from Eq. (14) is 48,176.776 MWh. The realized value of standard deviation indicates how off-centered the series of lost MWh power is from the mean value [44]. This resultant risk value is large and amounts to unserved electricity.

    View all citing articles on Scopus
    View full text