Abstract
We consider the problem of finding the minimum and maximum values of f-divergence for discrete probability distributions P and Q provided that one of these distributions and the value of their coupling are given. An explicit formula for the minimum value of the f-divergence under the above conditions is obtained, as well as a precise expression for its maximum value. This precise expression is not explicit in the general case, but in many special cases it allows us to write out both explicit formulas and simple upper bounds, which are sometimes optimal. Similar explicit formulas and upper bounds are also obtained for the Kullback–Leibler and χ2 divergences, which are the most important cases of the f-divergence.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Csiszár, I., Information-type Measures of Difference of Probability Distributions and Indirect Observations, Studia Sci. Math. Hungar., 1967, vol. 2, pp. 299–318.
Ali, S.M. and Silvey, S.D., A General Class of Coefficients of Divergence of One Distribution from Another, J. Roy. Statist. Soc. Ser.B, 1966, vol. 28, no. 1, pp. 131–142. https://www.jstor.org/stable/2984279
Sason, I. and Verdú, S., f-Divergence Inequalities, IEEE Trans. Inform. Theory, 2016, vol. 62, no. 11, pp. 5973–6006. https://doi.org/10.1109/TIT.2016.2603151
Makur, A. and Zheng, L., Comparison of Contraction Coefficients for f-Divergences, Probl. Peredachi Inf., 2020, vol. 56, no. 2, pp. 3–62 [Probl. Inf. Transm. (Engl. Transl.), 2020, vol. 56, no. 2, pp. 103–156]. https://doi.org/10.1134/S0032946020020015
Prelov, V.V., On Coupling of Probability Distributions and Estimating the Divergence through Variation, Probl. Peredachi Inf., 2017, vol. 53, no. 3, pp. 16–22 [Probl. Inf. Transm. (Engl. Transl.), 2017, vol. 53, no. 3, pp. 215–221]. https://doi.org/10.1134/S0032946017030024
Prelov, V.V., On Some Optimization Problems for the Rényi Divergence, Probl. Peredachi Inf., 2018, vol. 54, no. 3, pp. 36–53 [Probl. Inf. Transm. (Engl. Transl.), 2018, vol. 54, no. 3, pp. 229–244]. https://doi.org/10.1134/S003294601803002X
Gilardoni, G.L., On the Minimum f-Divergence for Given Total Variation, C. R. Math. Acad. Sci. Paris, 2006, vol. 343, no. 11–12, pp. 763–766. https://doi.org/10.1016/j.crma.2006.10.027
Gilardoni, G.L., On Pinsker’s and Vajda’s Type Inequalities for Csiszár’s f-Divergences, IEEE Trans. Inform. Theory, 2010, vol. 56, no. 11, pp. 5377–5386. https://doi.org/10.1109/TIT.2010.2068710
Prelov, V.V., On the Maximum Values of f-Divergence and Rényi Divergence under a Given Variational Distance, Probl. Peredachi Inf., 2020, vol. 56, no. 1, pp. 3–15 [Probl. Inf. Transm. (Engl. Transl.), 2020, vol. 56, no. 1, pp. 1–12]. https://doi.org/10.1134/S0032946020010019
Funding
The research was supported in part by the Russian Foundation for Basic Research, project no. 19-01-00364.
Author information
Authors and Affiliations
Additional information
Translated from Problemy Peredachi Informatsii, 2021, Vol. 57, No. 1, pp. 64–80 https://doi.org/10.31857/S0555292321010034.
Rights and permissions
About this article
Cite this article
Prelov, V. The f-Divergence and Coupling of Probability Distributions. Probl Inf Transm 57, 54–69 (2021). https://doi.org/10.1134/S0032946021010038
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1134/S0032946021010038