Exponential weighted entropy and exponential weighted mutual information
Introduction
Since Shannon’s original work in [1], entropy has been viewed as a kernel information measure of the uncertainty associated with a random variable. It has gained wide interest in signal processing, coding, data compression, and the other various fields. Mutual information (MI) is a quantity that measures the mutual dependence of the two variables. It is often recognized as an effective similarity measure in signal processing.
In this paper, we consider (X, Y) as the discrete random variable (r.v.) over a state space Ω×Ω with the joint probability distribution function (pdf) p(i, j), marginal pdfs pX(i) and pY(j). We also consider the conditional pdf pX|Y(i|j) of X given Y defined over Ω. Note that p(i), p(j), and p(i|j) are also used to mean pX(i), pY(j), and pX|Y(i|j), respectively.
Shannon entropy [1] of r.v. X was defined by Based on Shannon entropy, MI and the normalized mutual information (NMI) were proposed as the similar measure in [1], [2].
Shannon frameworks have the drawbacks of being position-free and memoryless [3]. They characterize the objective information of events occurrence. In other terms, they only consider the pdf of an event, regardless the distribution of r.v.s. As a consequence, a random variable X possesses the same Shannon entropy as for any a ∈ R. In some applications, besides the objective information, the subjective information or utility about a goal of events occurrence should also be taken into account. This leads to the propositions of the weighted entropy [3] and the weighted MI [4].
Under the axiomatic framework of entropy [5], the weighted entropy [3] of r.v. X ∼ p(i) was defined as There are different versions of the weighted MI and the normalized weighted MI. A typical version of the weighted MI of r.v.s (X, Y) ∼ p(i, j) was given by [4] Since the additivity hypothesis in thermodynamics, Shannon entropy neglects the correlations between the subsystems, whereas non-extensive processes are common at many physical levels in statistical mechanics and atomic physics [6]. There are two ways to overcome this intrinsic drawbacks. The one way is to extend the additivity to nonadditivity, such as Rényi entropy [7] and Tsallis entropy[6], [8], [9]. The other way is taking some prior statistical information into account [10]. In (2), the weight function i is too simple. In (3), a weight w(i, j) is placed on the probability of each variable value co-occurrence p(i, j), which leads it to be difficult to study their mathematical properties. Till now, there still lack of theoretic research on the weighted entropy and the weighted MI. Since the exponential weighted method has been viewed as an efficient tool in engineering [11], in this paper, the exponential weighted entropy is proposed as the generalized form of the (weighted) entropy [1], [3] and exponential weighted mutual information is proposed as the special form of the weighted mutual information introduced in [4]. They are the extensions of Shannon frameworks and generalize the corresponding concepts in [12] that defined in a generalized Euclidean metric space based on fractional calculus.
The rest of this paper is organized as follows. EWE, EWMI, and NEWMI are proposed in Section 2. The concavity properties are studied in Section 3. Section 4 provides applications in image registration. We conclude with a summary of content in Section 5.
Section snippets
Exponential weighted entropy and exponential weighted mutual information
Let f(i) be bounded a function and r.v. we use as the exponential weighted summation operator with order α on p(i) to mean that It is worth noting that the exponential weighted summation operator in Eq. (4) works on the probability distribution function of r.v. and not on a single p(i).
We call f(i) the weight function and the exponential weight function. To keep consistent with the formula in [12], we use rather
The concavity properties
This section will prove that EWE and EWMI inherit some concavity properties of probability distributions as Shannon entropy and MI, respectively. The following theorems generalize the corresponding conclusions in [13].
Applications in image registration
Image registration is to find a deformation function that can establish the spatial correspondence between an image pair. The kernel problem is to find an ideal similarity measure. Generally, MI and NMI are widely adopted as the well-established similarity measures [15]. some other information, such as gradient information, space information, and edge information, etc. of a image pair are incorporated to achieve higher registration accuracy.
In following experiments, when EWMI and NEWMI are used
Conclusions
EWE and EWMI are proposed as the more generalized forms of Shannon frameworks. They are the information measures supplied by a probabilistic experiment whose elementary events are characterized both by their objective probabilities and by some qualitative (objective or subjective) weights. Some mathematical properties of EWE and EWMI have been investigated. The experiment in image registration demonstrates that, using EWMI and NEWMI as the similarity measures would lead to higher aligned
Acknowledgment
This research is supported by 973 Program (2013CB329404), the Fundamental Research Funds for the Central Universities (ZYGX2013Z005), NSFC (61370147, 61170311), Sichuan Province Sci. & Tech. Research Project (2012GZX0080). The author would like to thank the anonymous reviewers and the editor in chief for their useful comments.
Shiwei Yu was born in Sichuan Province, China, in 1974. He received the B.S. degree and the M.S. degree both in basic mathematics from Sichuan Normal University in 1996 and 1999, respectively. Currently, he is a Ph.D. candidate in School of Mathematical Sciences, University of Electronic Science and Technology of China. His research interests include information theorem and image processing.
References (22)
- et al.
An overlap invariant entropy measure of 3D medical image alignment
Pattern Recognit.
(1999) - et al.
A non-rigid registration approach for quantifying myocardial contraction in tagged MRI using generalized information measures
Med. Image Anal.
(2005) - et al.
Information measures based on fractional calculus
Inf. Process. Lett.
(2012) - et al.
A reproducible evaluation of ANTs similarity metric performance in brain image registration
Neuroimage
(2011) - et al.
Evaluation of atlas selection strategies for atlas-based image segmentation with application to confocal microscopy images of bee brains
Neuroimage
(2004) Mathematical theory of communication
Bell Syst. Tech. J.
(1948)- et al.
A quantitative-qualitative measure of information in cybernetic systems
IEEE Trans. Inf. Theory
(1968) - et al.
Information Theory with Applications
(1977) - et al.
Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy
IEEE Trans. Inf. Theory
(1980) Possible generalization of Boltzmann–Gibbs statistics
J. Stat. Phys.
(1988)
On measures of entropy and information
Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability
Cited by (0)
Shiwei Yu was born in Sichuan Province, China, in 1974. He received the B.S. degree and the M.S. degree both in basic mathematics from Sichuan Normal University in 1996 and 1999, respectively. Currently, he is a Ph.D. candidate in School of Mathematical Sciences, University of Electronic Science and Technology of China. His research interests include information theorem and image processing.
Ting-Zhu Huang received the B. S., M. S., and Ph.D. degrees in Computational Mathematics from the Department of Mathematics, Xi’an Jiaotong University, Xi’an, China. He is currently a professor in the School of Mathematical Sciences, UESTC. He is currently an editor of The Scientific World Journal, Advances in Numerical Analysis, J. Appl. Math., J. Pure and Appl. Math.: Adv. and Appl., J. Electronic Sci. and Tech. of China, etc. His current research interests include scientific computation and applications, numerical algorithms for image processing, numerical linear algebra, preconditioning technologies, and matrix analysis with applications, etc.