Skip to main content
Springer Nature Link
Log in

Some New Flexibilizations of Bregman Divergences and Their Asymptotics

  • Conference paper
  • First Online:
Geometric Science of Information (GSI 2017)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10589))

Included in the following conference series:

  • 2431 Accesses

Abstract

Ordinary Bregman divergences (distances) OBD are widely used in statistics, machine learning, and information theory (see e.g. [5, 18]; [4, 6, 7, 14,15,16, 22, 23, 25]). They can be flexibilized in various different ways. For instance, there are the Scaled Bregman divergences SBD of Stummer [20] and Stummer and Vajda [21] which contain both the OBDs as well the Csiszar-Ali-Silvey \(\phi -\)divergences as special cases. On the other hand, the OBDs are subsumed by the Total Bregman divergences of Liu et al. [12, 13], Vemuri et al. [24] and the more general Conformal Divergences COD of Nock et al. [17]. The latter authors also indicated the possibility to combine the concepts of SBD and COD, under the name “Conformal Scaled Bregman divergences” CSBD. In this paper, we introduce some new divergences between (non-)probability distributions which particularly cover the corresponding OBD, SBD, COD and CSBD (for separable situations) as special cases. Non-convex generators are employed, too. Moreover, for the case of i.i.d. sampling we derive the asymptotics of a useful new-divergence-based test statistics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    our concept can be analogously worked out for non-probability distributions (nonnegative measures) P, Q.

  2. 2.

    sigma-finite.

  3. 3.

    in (2), we can also extend \([ \ldots ]\) to \(G([ \ldots ])\) for some nonnegative scalar function G satisfying \(G(z) = 0\) iff \(z=0\).

  4. 4.

    (with respect to the one-dim. Lebesgue measure).

References

  1. Amari, S.-I.: Information Geometry and Its Applications. Springer, Tokyo (2016)

    Book  MATH  Google Scholar 

  2. Amari, S.-I., Nagaoka, H.: Methods of Information Geometry. Oxford University Press, Oxford (2000)

    MATH  Google Scholar 

  3. Ali, M.S., Silvey, D.: A general class of coefficients of divergence of one distribution from another. J. Roy. Statist. Soc. B–28, 131–140 (1966)

    MATH  MathSciNet  Google Scholar 

  4. Banerjee, A., Merugu, S., Dhillon, I.S., Ghosh, J.: Clustering with Bregman divergences. J. Mach. Learn. Res. 6, 1705–1749 (2005)

    MATH  MathSciNet  Google Scholar 

  5. Basu, A., Shioya, H., Park, C.: Statistical Inference: The Minimum Distance Approach. CRC Press, Boca Raton (2011)

    MATH  Google Scholar 

  6. Cesa-Bianchi, N., Lugosi, G.: Prediction, Learning and Games. Cambridge University Press, Cambridge (2006)

    Book  MATH  Google Scholar 

  7. Collins, M., Schapire, R.E., Singer, Y.: Logistic regression, AdaBoost and Bregman distances. Mach. Learn. 48, 253–285 (2002)

    Article  MATH  Google Scholar 

  8. Csiszar, I.: Eine informationstheoretische Ungleichung und ihre Anwendung auf den Beweis der Ergodizität von Markoffschen Ketten. Publ. Math. Inst. Hungar. Acad. Sci. A–8, 85–108 (1963)

    MATH  Google Scholar 

  9. Kißlinger, A.-L., Stummer, W.: Some decision procedures based on scaled Bregman distance surfaces. In: Nielsen, F., Barbaresco, F. (eds.) GSI 2013. LNCS, vol. 8085, pp. 479–486. Springer, Heidelberg (2013). doi:10.1007/978-3-642-40020-9_52

    Chapter  Google Scholar 

  10. Kißlinger, A.-L., Stummer, W.: New model search for nonlinear recursive models, regressions and autoregressions. In: Nielsen, F., Barbaresco, F. (eds.) GSI 2015. LNCS, vol. 9389, pp. 693–701. Springer, Cham (2015). doi:10.1007/978-3-319-25040-3_74

    Chapter  Google Scholar 

  11. Kißlinger, A.-L., Stummer, W.: Robust statistical engineering by means of scaled Bregman distances. In: Agostinelli, C., Basu, A., Filzmoser, P., Mukherjee, D. (eds.) Recent Advances in Robust Statistics: Theory and Applications, pp. 81–113. Springer, New Delhi (2016). doi:10.1007/978-81-322-3643-6_5

    Chapter  Google Scholar 

  12. Liu, M., Vemuri, B.C., Amari, S.-I., Nielsen, F.: Total Bregman divergence and its applications to shape retrieval. In: Proceedings 23rd IEEE CVPR, pp. 3463–3468 (2010)

    Google Scholar 

  13. Liu, M., Vemuri, B.C., Amari, S.-I., Nielsen, F.: Shape retrieval using hierarchical total Bregman soft clustering. IEEE Trans. Pattern Anal. Mach. Intell. 34(12), 2407–2419 (2012)

    Article  Google Scholar 

  14. Murata, N., Takenouchi, T., Kanamori, T., Eguchi, S.: Information geometry of U-boost and Bregman divergence. Neural Comput. 16(7), 1437–1481 (2004)

    Article  MATH  Google Scholar 

  15. Nock, R., Menon, A.K., Ong, C.S.: A scaled Bregman theorem with applications. In: Advances in Neural Information Processing Systems 29 (NIPS 2016), pp. 19–27 (2016)

    Google Scholar 

  16. Nock, R., Nielsen, F.: Bregman divergences and surrogates for learning. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 2048–2059 (2009)

    Article  Google Scholar 

  17. Nock, R., Nielsen, F., Amari, S.-I.: On conformal divergences and their population minimizers. IEEE Trans. Inf. Theory 62(1), 527–538 (2016)

    Article  MATH  MathSciNet  Google Scholar 

  18. Pardo, M.C., Vajda, I.: On asymptotic properties of information-theoretic divergences. IEEE Trans. Inf. Theory 49(7), 1860–1868 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  19. Pallaschke, D., Rolewicz, S.: Foundations of Mathematical Optimization. Kluwer Academic Publishers, Dordrecht (1997)

    Book  MATH  Google Scholar 

  20. Stummer, W.: Some Bregman distances between financial diffusion processes. Proc. Appl. Math. Mech. 7(1), 1050503–1050504 (2007)

    Article  Google Scholar 

  21. Stummer, W., Vajda, I.: On Bregman distances and divergences of probability measures. IEEE Trans. Inf. Theory 58(3), 1277–1288 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  22. Sugiyama, M., Suzuki, T., Kanamori, T.: Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimation. Ann. Inst. Stat. Math. 64, 1009–1044 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  23. Tsuda, K., Rätsch, G., Warmuth, M.: Matrix exponentiated gradient updates for on-line learning and Bregman projection. J. Mach. Learn. Res. 6, 995–1018 (2005)

    MATH  MathSciNet  Google Scholar 

  24. Vemuri, B.C., Liu, M., Amari, S.-I., Nielsen, F.: Total Bregman divergence and its applications to DTI analysis. IEEE Trans. Med. Imag. 30(2), 475–483 (2011)

    Article  Google Scholar 

  25. Wu, L., Hoi, S.C.H., Jin, R., Zhu, J., Yu, N.: Learning Bregman distance functions for semi-supervised clustering. IEEE Trans. Knowl. Data Eng. 24(3), 478–491 (2012)

    Article  Google Scholar 

Download references

Acknowledgement

We are grateful to 3 referees for their useful suggestions.

Author information

Authors and Affiliations

  1. Department of Mathematics, University of Erlangen–Nürnberg, Cauerstrasse 11, 91058, Erlangen, Germany

    Wolfgang Stummer

  2. Affiliated Faculty Member of the School of Business and Economics, University of Erlangen–Nürnberg, Lange Gasse 20, 90403, Nürnberg, Germany

    Wolfgang Stummer

  3. Chair of Statistics and Econometrics, University of Erlangen–Nürnberg, Lange Gasse 20, 90403, Nürnberg, Germany

    Anna-Lena Kißlinger

Authors
  1. Wolfgang Stummer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anna-Lena Kißlinger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anna-Lena Kißlinger .

Editor information

Editors and Affiliations

  1. Ecole Polytechnique, Palaiseau, France

    Frank Nielsen

  2. Thales Land and Air Systems, Limours, France

    Frédéric Barbaresco

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Stummer, W., Kißlinger, AL. (2017). Some New Flexibilizations of Bregman Divergences and Their Asymptotics. In: Nielsen, F., Barbaresco, F. (eds) Geometric Science of Information. GSI 2017. Lecture Notes in Computer Science(), vol 10589. Springer, Cham. https://doi.org/10.1007/978-3-319-68445-1_60

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-68445-1_60

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-68444-4

  • Online ISBN: 978-3-319-68445-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics