Elsevier

Neurocomputing

Volume 249, 2 August 2017, Pages 86-94
Neurocomputing

Exponential weighted entropy and exponential weighted mutual information

https://doi.org/10.1016/j.neucom.2017.03.075Get rights and content

Abstract

In this paper, the exponential weighted entropy (EWE) and exponential weighted mutual information (EWMI) are proposed as the more generalized forms of Shannon entropy and mutual information (MI), respectively. They are position-related and causal systems that redefine the foundations of information-theoretic metrics. As the special forms of the weighted entropy and the weighted mutual information, EWE and EWMI have been proved that they preserve nonnegativity and concavity properties similar to Shannon frameworks. They can be adopted as the information measures in spatial interaction modeling. Paralleling with the normalized mutual information (NMI), the normalized exponential weighted mutual information (NEWMI) is also investigated. Image registration experiments demonstrate that EWMI and NEWMI algorithms can achieve higher aligned accuracy than MI and NMI algorithms.

Introduction

Since Shannon’s original work in [1], entropy has been viewed as a kernel information measure of the uncertainty associated with a random variable. It has gained wide interest in signal processing, coding, data compression, and the other various fields. Mutual information (MI) is a quantity that measures the mutual dependence of the two variables. It is often recognized as an effective similarity measure in signal processing.

In this paper, we consider (X, Y) as the discrete random variable (r.v.) over a state space Ω×Ω with the joint probability distribution function (pdf) p(i, j), marginal pdfs pX(i) and pY(j). We also consider the conditional pdf pX|Y(i|j) of X given Y defined over Ω. Note that p(i), p(j), and p(i|j) are also used to mean pX(i), pY(j), and pX|Y(i|j), respectively.

Shannon entropy [1] of r.v. X was defined by H(X)=i=1p(i)logp(i)Based on Shannon entropy, MI and the normalized mutual information (NMI) were proposed as the similar measure in [1], [2].

Shannon frameworks have the drawbacks of being position-free and memoryless [3]. They characterize the objective information of events occurrence. In other terms, they only consider the pdf of an event, regardless the distribution of r.v.s. As a consequence, a random variable X possesses the same Shannon entropy as X+a, for any aR. In some applications, besides the objective information, the subjective information or utility about a goal of events occurrence should also be taken into account. This leads to the propositions of the weighted entropy [3] and the weighted MI [4].

Under the axiomatic framework of entropy [5], the weighted entropy [3] of r.v. Xp(i) was defined as Hw(X)=i=1ip(i)logp(i)There are different versions of the weighted MI and the normalized weighted MI. A typical version of the weighted MI of r.v.s (X, Y) ∼ p(i, j) was given by [4] Iw(X,Y)=i=1j=1w(i,j)p(i,j)logp(i,j)p(i)p(j)Since the additivity hypothesis in thermodynamics, Shannon entropy neglects the correlations between the subsystems, whereas non-extensive processes are common at many physical levels in statistical mechanics and atomic physics [6]. There are two ways to overcome this intrinsic drawbacks. The one way is to extend the additivity to nonadditivity, such as Rényi entropy [7] and Tsallis entropy[6], [8], [9]. The other way is taking some prior statistical information into account [10]. In (2), the weight function i is too simple. In (3), a weight w(i, j) is placed on the probability of each variable value co-occurrence p(i, j), which leads it to be difficult to study their mathematical properties. Till now, there still lack of theoretic research on the weighted entropy and the weighted MI. Since the exponential weighted method has been viewed as an efficient tool in engineering [11], in this paper, the exponential weighted entropy is proposed as the generalized form of the (weighted) entropy [1], [3] and exponential weighted mutual information is proposed as the special form of the weighted mutual information introduced in [4]. They are the extensions of Shannon frameworks and generalize the corresponding concepts in [12] that defined in a generalized Euclidean metric space based on fractional calculus.

The rest of this paper is organized as follows. EWE, EWMI, and NEWMI are proposed in Section 2. The concavity properties are studied in Section 3. Section 4 provides applications in image registration. We conclude with a summary of content in Section 5.

Section snippets

Exponential weighted entropy and exponential weighted mutual information

Let f(i) be bounded a function and r.v. Xp(i),i=1,2,, we use WXα as the exponential weighted summation operator with order α on p(i) to mean that WXα(p(i))=i=1f1α(i)p(i)(αR)It is worth noting that the exponential weighted summation operator WXα in Eq. (4) works on the probability distribution function of r.v. Xp(i),i=1,2,, and not on a single p(i).

We call f(i) the weight function and f1α(i) the exponential weight function. To keep consistent with the formula in [12], we use f1α rather

The concavity properties

This section will prove that EWE and EWMI inherit some concavity properties of probability distributions as Shannon entropy and MI, respectively. The following theorems generalize the corresponding conclusions in [13].

Applications in image registration

Image registration is to find a deformation function that can establish the spatial correspondence between an image pair. The kernel problem is to find an ideal similarity measure. Generally, MI and NMI are widely adopted as the well-established similarity measures [15]. some other information, such as gradient information, space information, and edge information, etc. of a image pair are incorporated to achieve higher registration accuracy.

In following experiments, when EWMI and NEWMI are used

Conclusions

EWE and EWMI are proposed as the more generalized forms of Shannon frameworks. They are the information measures supplied by a probabilistic experiment whose elementary events are characterized both by their objective probabilities and by some qualitative (objective or subjective) weights. Some mathematical properties of EWE and EWMI have been investigated. The experiment in image registration demonstrates that, using EWMI and NEWMI as the similarity measures would lead to higher aligned

Acknowledgment

This research is supported by 973 Program (2013CB329404), the Fundamental Research Funds for the Central Universities (ZYGX2013Z005), NSFC (61370147, 61170311), Sichuan Province Sci. & Tech. Research Project (2012GZX0080). The author would like to thank the anonymous reviewers and the editor in chief for their useful comments.

Shiwei Yu was born in Sichuan Province, China, in 1974. He received the B.S. degree and the M.S. degree both in basic mathematics from Sichuan Normal University in 1996 and 1999, respectively. Currently, he is a Ph.D. candidate in School of Mathematical Sciences, University of Electronic Science and Technology of China. His research interests include information theorem and image processing.

References (22)

  • A. Rényi

    On measures of entropy and information

    Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability

    (1961)
  • Cited by (0)

    Shiwei Yu was born in Sichuan Province, China, in 1974. He received the B.S. degree and the M.S. degree both in basic mathematics from Sichuan Normal University in 1996 and 1999, respectively. Currently, he is a Ph.D. candidate in School of Mathematical Sciences, University of Electronic Science and Technology of China. His research interests include information theorem and image processing.

    Ting-Zhu Huang received the B. S., M. S., and Ph.D. degrees in Computational Mathematics from the Department of Mathematics, Xi’an Jiaotong University, Xi’an, China. He is currently a professor in the School of Mathematical Sciences, UESTC. He is currently an editor of The Scientific World Journal, Advances in Numerical Analysis, J. Appl. Math., J. Pure and Appl. Math.: Adv. and Appl., J. Electronic Sci. and Tech. of China, etc. His current research interests include scientific computation and applications, numerical algorithms for image processing, numerical linear algebra, preconditioning technologies, and matrix analysis with applications, etc.

    View full text