Elsevier

Information Sciences

Volume 512, February 2020, Pages 50-63
Information Sciences

On the f-divergence for discrete non-additive measures

https://doi.org/10.1016/j.ins.2019.09.033Get rights and content

Abstract

In this paper we study the definition of the f-divergence and the Hellinger distance for non-additive measures in the discrete case. As these measures are based on the derivatives of the measures, we consider the problem of defining the Radon–Nikodym derivative of a non-additive measure.

While Radon–Nikodym derivatives for additive measures exist for absolutely continuous measures, this is not the case in the non-additive case. In this paper we will define set-directional and upper, lower and interval derivatives. We will also define when two measures have the same sign. These definitions will be used to introduce alternative definitions of the f-divergence, all extending the classical definition to non-additive measures.

Introduction

In statistics and information theory, f-divergences are extensively used. Recall that the Hellinger distance, the Kullback-Leibler divergence, the Rényi distance and the variation distance are all examples of f-divergences. They are used to compare probability distributions, and the Kullback-Leibler divergence can also be used to define the entropy.

Non-additive measures, also known as fuzzy measures, capacities and monotonic games, are a generalization of additive measures in which the additivity axiom is replaced by a monotonicity with respect to set inclusion. Then, functions can be integrated with respect to these measures. In this paper we will consider integration of functions with respect to non-additive measures using the Choquet integral. Definitions for non-additive measures and integrals exist for both discrete and infinite reference sets.

In two recent papers [24], [25] we introduced the Hellinger distance and the f-divergence for continuous additive measures. The definition is based on a Radon–Nikodym-like derivative.

In this paper, we consider the same problem when we deal with the discrete case. That is, we introduce definitions for the f-divergence and the Hellinger distance. From the f-divergence for non-additive measures we can derive other distances and divergences as it is done in the additive case. The approach we use here for introducing these divergences is similar to our approach in [24], [25]. That is, to use the Radon–Nikodym-like derivative. For additive measures, the Radon–Nikodym theorem establishes that the derivative exists for absolutely continuous measures. Nevertheless, this is not the case for non-additive ones. As we will show in this paper, when we require that such derivative exists, we are making strong assumptions on the function.

In order to provide solutions to this problem, we introduce a few new definitions. Some of them generalize and extend the concept of derivative as set-directional and upper, lower and interval derivatives. Others refer to properties for measures. In particular, we introduce the property of homogeneous monotonicity for a pair of measures, and the concept of two measures with the same sign.

The structure of the paper is as follows. In Section 2 we review some definitions that we need in the rest of our work. They are mainly results on measures and divergences. In Section 3 we discuss the computation of the Radon–Nikodym-like derivative for discrete Choquet integrals and we propose several alternatives. In Section 4 we propose divergences for the discrete case. The paper finishes with some conclusions.

Section snippets

Preliminaries

In this section we review some definitions that are needed later in this paper. We start with some definitions related to non-additive measures, and then definitions related to the f-divergence.

On the discrete Radon–Nikodym derivative for discrete Choquet integrals

In this section we discuss the problem of finding the Radon–Nikodym derivative when f is defined on a discrete set Ω=X={x1,,xn}. So, we consider a function on this reference set f:XR+, and measures μ1, μ2: ℘(X) → [0, 1]. Nevertheless, the two initial definitions to formalize what a derivative is are valid for any measurable space (Ω,F) and, so, we use Ω and F. For details on the Choquet integral and the Radon–Nikodym derivative see e.g. [11], [14], [17], [23].

Definition 9

We say that μ1 is a Choquet

Divergences for the discrete case

In this section we introduce alternative definitions of the f-divergence and the Hellinger for non-additive measures for the discrete case. We begin with definitions based on chains and then on the Möbius transform. In both cases we introduce some additional definitions that we need later.

Conclusions

In this paper we have considered the problem of defining the f-divergence and the Hellinger distance, which are defined for additive measures, to non-additive ones. We have focused on measures on discrete domains. In previous papers [24], [25] we considered the problem for continuous domains.

These definitions rely on the Radon–Nikodym derivative. Because of that we have considered the problem of computing the Radon–Nikodym derivative for non-additive measures in the discrete case. While the

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (25)

  • S.S. Dragomir

    A generalization of f-divergence measure to convex functions defined on linear spaces

    Commun. Math. Anal.

    (2013)
  • K. Fujimoto

    Cooperative game as non-additive measure

  • Cited by (0)

    View full text