Skip to main content
SpringerLink
Log in

Optimal Upper Bounds for the Divergence of Finite-Dimensional Distributions under a Given Variational Distance

  • Information Theory
  • Published:
Problems of Information Transmission Aims and scope Submit manuscript

Abstract

We consider the problem of finding the maximum values of divergences D(P‖Q) and D(Q‖P) for probability distributions P and Q ranging in the finite set \(\mathcal{N}=\left\{1,\;2,...,n\right\}\) provided that both the variation distance V (P,Q) between them and either the probability distribution Q or (in the case of D(P‖Q)) only the value of the minimal component q min of the probability distribution Q are given. Precise expressions for the maximum values of these divergences are obtained. In several cases these expressions allow us to write out some explicit formulas and simple upper and lower bounds for them. Moreover, explicit formulas for the maximum of D(P‖Q) for given V (P,Q) and q min and also for the maximum of D(Q‖P) for given Q and V (P,Q) are obtained for all possible values of these parameters.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Fedotov, A.A., Harremöes, P., and Topsøe, F., Refinements of Pinsker’s Inequality, IEEE Trans. Inform. Theory, 2003, vol. 49, no. 6, pp. 1491–1498.

    Article  MathSciNet  Google Scholar 

  2. Csiszár, I. and Körner, J., Information Theory: Coding Theorems for Discrete Memoryless Systems, New York: Academic; Budapest: Akad. Kiad´o, 1981. Translated under the title Teoriya informatsii: teoremy kodirovaniya dlya diskretnykh sistem bez pamyati, Moscow: Mir, 1985.

    MATH  Google Scholar 

  3. Vajda, I., Note on Discrimination Information and Variation, IEEE Trans. Inform. Theory, 1970, vol. 16, no. 6, pp. 771–773.

    Article  MathSciNet  Google Scholar 

  4. Ordentlich, E. and Weinberger, M.J., A Distribution Dependent Refinement of Pinsker’s Inequality, IEEE Trans. Inform. Theory, 2005, vol. 51, no. 5, pp. 1836–1840.

    Article  MathSciNet  Google Scholar 

  5. Prelov, V.V., On Coupling of Probability Distributions and Estimating the Divergence through Variation, Probl. Peredachi Inf., 2017, vol. 53, no. 3, pp. 16–22 [Probl. Inf. Transm. (Engl. Transl.), 2017, vol. 53, no. 3, pp. 215–221].

    MathSciNet  MATH  Google Scholar 

  6. Csiszár, I. and Talata, Z., Context Tree Estimation for Not Necessarily Finite Memory Processes via BIC and MDL, IEEE Trans. Inform. Theory, 2006, vol. 52, no. 3, pp. 1007–1016.

    Article  MathSciNet  Google Scholar 

  7. Sason, I. and Verdú, S., Upper Bounds on the Relative Entropy and Rényi Divergence as a Function of Total Variation Distance for Finite Alphabets, in Proc. 2015 IEEE Information Theory Workshop (ITW’2015), Jeju, Korea, Oct. 11–15, 2015, pp. 214–218.

    Google Scholar 

  8. Sason, I. and Verdú, S., f-Divergence Inequalities, IEEE Trans. Inform. Theory, 2016, vol. 62, no. 11, pp. 5973–6006.

    Article  MathSciNet  Google Scholar 

Download references

Funding

Funding The research was supported in part by the Russian Foundation for Basic Research, project no. 19-01-00364.

Author information

Authors and Affiliations

  1. Kharkevich Institute for Information Transmission Problems, Russian Academy of Sciences, Moscow, Russia

    V. V. Prelov

Authors
  1. V. V. Prelov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to V. V. Prelov.

Additional information

Russian Text © The Author(s), 2019, published in Problemy Peredachi Informatsii, 2019, Vol. 55, No. 3, pp. 21–29.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Prelov, V.V. Optimal Upper Bounds for the Divergence of Finite-Dimensional Distributions under a Given Variational Distance. Probl Inf Transm 55, 218–225 (2019). https://doi.org/10.1134/S0032946019030025

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0032946019030025

Key words

Search

Navigation