Skip to main content
SpringerLink
Log in
Book cover

International Conference on Geometric Science of Information

GSI 2021: Geometric Science of Information pp 677–684Cite as

Computing Statistical Divergences with Sigma Points

  • Conference paper
  • First Online:

  • 2282 Accesses

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12829))

Abstract

We show how to bypass the integral calculation and express the Kullback-Leibler divergence between any two densities of an exponential family using a finite sum of logarithms of density ratio evaluated at sigma points. This result allows us to characterize the exact error of Monte Carlo estimators. We then extend the sigma point method to the calculations of q-divergences between densities of a deformed q-exponential family.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Amari, S.I.: Information Geometry and Its Applications. Applied Mathematical Sciences. Springer, Japan (2016). https://doi.org/10.1007/978-4-431-55978-8

  2. Amari, S.I., Ohara, A.: Geometry of \(q\)-exponential family of probability distributions. Entropy 13(6), 1170–1185 (2011)

    Article  MathSciNet  Google Scholar 

  3. Azoury, K.S., Warmuth, M.K.: Relative loss bounds for on-line density estimation with the exponential family of distributions. Mach. Learn. 43(3), 211–246 (2001)

    Article  Google Scholar 

  4. Banerjee, A., Merugu, S., Dhillon, I.S., Ghosh, J.: Clustering with Bregman divergences. J. Mach. Learn. Res. 6, 1705–1749 (2005)

    MathSciNet  MATH  Google Scholar 

  5. Barndorff-Nielsen, O.: Information and Exponential Families. John Wiley & Sons, Hoboken (2014)

    Book  Google Scholar 

  6. Billingsley, P.: Probability and Measure, 3rd edn. Wiley, New York (1995)

    MATH  Google Scholar 

  7. Bregman, L.M.: The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phys. 7(3), 200–217 (1967)

    Article  MathSciNet  Google Scholar 

  8. Carathéodory, C.: Über den Variabilitätsbereich der Koeffizienten von Potenzreihen, die gegebene Werte nicht annehmen. Mathematische Annalen 64(1), 95–115 (1907)

    Article  MathSciNet  Google Scholar 

  9. Cover, T.M., Thomas, J.A.: Elements of Information Theory. John Wiley & Sons, Hoboken (2012)

    MATH  Google Scholar 

  10. Csiszár, I.: Information-type measures of difference of probability distributions and indirect observation. Studia Scientiarum Mathematicarum Hungarica 2, 229–318 (1967)

    MathSciNet  Google Scholar 

  11. Del Castillo, J.: The singly truncated normal distribution: a non-steep exponential family. Ann. Inst. Stat. Math. 46(1), 57–66 (1994)

    Article  Google Scholar 

  12. Gayen, A., Kumar, M.A.: Projection theorems and estimating equations for power-law models. J. Multivariate Anal. 184, 104734 (2021)

    Article  MathSciNet  Google Scholar 

  13. Goldberger, J., Greenspan, H.K., Dreyfuss, J.: Simplifying mixture models using the unscented transform. IEEE Trans. Pattern Anal. Mach. Intell. 30(8), 1496–1502 (2008)

    Article  Google Scholar 

  14. Julier, S., Uhlmann, J., Durrant-Whyte, H.F.: A new method for the nonlinear transformation of means and covariances in filters and estimators. IEEE Trans. Autom. Control 45(3), 477–482 (2000)

    Article  MathSciNet  Google Scholar 

  15. Kullback, S., Leibler, R.A.: On information and sufficiency. Ann. Math. Stat. 22(1), 79–86 (1951)

    Article  MathSciNet  Google Scholar 

  16. Lemieux, C.: Monte Carlo and quasi-Monte Carlo sampling. Springer Science & Business Media, New York (2009). https://doi.org/10.1007/978-0-387-78165-5

  17. Nielsen, F.: An elementary introduction to information geometry. Entropy 22(10), 1100 (2020)

    Article  MathSciNet  Google Scholar 

  18. Nielsen, F.: On Voronoi diagrams on the information-geometric Cauchy manifolds. Entropy 22(7), 713 (2020)

    Article  MathSciNet  Google Scholar 

  19. Nielsen, F., Garcia, V.: Statistical exponential families: A digest with flash cards. Technical Report. arXiv:0911.4863 (2009)

  20. Nielsen, F., Nock, R.: Patch matching with polynomial exponential families and projective divergences. In: Amsaleg, L., Houle, M.E., Schubert, E. (eds.) SISAP 2016. LNCS, vol. 9939, pp. 109–116. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46759-7_8

    Chapter  Google Scholar 

  21. Nielsen, F., Nock, R.: Cumulant-free closed-form formulas for some common (dis)similarities between densities of an exponential family. arXiv:2003.02469 (2020)

  22. Peyré, G., Cuturi, M.: Computational optimal transport: with applications to data science. Found. Trends® Mach. Learn. 11(5–6), 355–607 (2019)

    Google Scholar 

  23. Risch, R.H.: The solution of the problem of integration in finite terms. Bull. Am. Math. Soc. 76(3), 605–608 (1970)

    Article  MathSciNet  Google Scholar 

  24. Tanaya, D., Tanaka, M., Matsuzoe, H.: Notes on geometry of \(q\)-normal distributions. In: Recent Progress in Differential Geometry and Its Related Fields, pp. 137–149. World Scientific (2012)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Sony Computer Science Laboratories Inc., Tokyo, Japan

    Frank Nielsen

  2. Data61, CSIRO, Sydney, Australia

    Richard Nock

Authors
  1. Frank Nielsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Richard Nock
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Frank Nielsen .

Editor information

Editors and Affiliations

  1. Sony Computer Science Laboratories Inc., Tokyo, Japan

    Frank Nielsen

  2. THALES Land & Air Systems, Meudon, France

    Frédéric Barbaresco

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nielsen, F., Nock, R. (2021). Computing Statistical Divergences with Sigma Points. In: Nielsen, F., Barbaresco, F. (eds) Geometric Science of Information. GSI 2021. Lecture Notes in Computer Science(), vol 12829. Springer, Cham. https://doi.org/10.1007/978-3-030-80209-7_72

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-80209-7_72

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-80208-0

  • Online ISBN: 978-3-030-80209-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation