Loading [MathJax]/extensions/MathEvents.js
Limit Distribution Theory for f-Divergences | IEEE Journals & Magazine | IEEE Xplore

Limit Distribution Theory for f-Divergences


Abstract:

f -divergences, which quantify discrepancy between probability distributions, are ubiquitous in information theory, machine learning, and statistics. While there are n...Show More

Abstract:

f -divergences, which quantify discrepancy between probability distributions, are ubiquitous in information theory, machine learning, and statistics. While there are numerous methods for estimating f -divergences from data, a limit distribution theory, which quantifies fluctuations of the estimation error, is largely obscure. As limit theorems are pivotal for valid statistical inference, to close this gap, we develop a general methodology for deriving distributional limits for f -divergences based on the functional delta method and Hadamard directional differentiability. Focusing on four prominent f -divergences—Kullback-Leibler divergence, \chi ^{2} divergence, squared Hellinger distance, and total variation distance—we identify sufficient conditions on the population distributions for the existence of distributional limits and characterize the limiting variables. These results are used to derive one- and two-sample limit theorems for Gaussian-smoothed f -divergences, both under the null and the alternative. Finally, an application of the limit distribution theory to auditing differential privacy is proposed and analyzed for significance level and power against local alternatives.
Published in: IEEE Transactions on Information Theory ( Volume: 70, Issue: 2, February 2024)
Page(s): 1233 - 1267
Date of Publication: 13 October 2023

ISSN Information:

Funding Agency:


Contact IEEE to Subscribe

References

References is not available for this document.