Skip to main content
SpringerLink
Log in

PAC-Bayesian Theory

  • Chapter
  • First Online:
Empirical Inference

Abstract

The PAC-Bayesian framework is a frequentist approach to machine learning which encodes learner bias as a “prior probability” over hypotheses. This chapter reviews basic PAC-Bayesian theory, including Catoni’s basic inequality and Catoni’s localization theorem.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin classifiers. In: Proceedings of the 5th Annual Workshop on Computational Learning Theory, Pittsburgh, pp. 144–152. ACM (1992)

    Google Scholar 

  2. Catoni, O.: PAC-Bayesian supervised classification: the thermodynamics of statistical learning. arXiv:0712.0248 (2007, preprint)

    Google Scholar 

  3. Germain, P., Lacasse, A., Laviolette, F., Marchand, M.: PAC-Bayesian learning of linear classifiers. In: Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, pp. 353–360. ACM (2009)

    Google Scholar 

  4. Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)

    Article  MathSciNet  MATH  Google Scholar 

  5. Langford, J.: Tutorial on practical prediction theory for classification. J. Mach. Learn. Res. 6(1), 273 (2006)

    MathSciNet  Google Scholar 

  6. Langford, J., Blum, A.: Microchoice bounds and self bounding learning algorithms. In: Proceedings of the 12th Annual Conference on Computational Learning Theory, Santa Cruz, pp. 209–214. ACM (1999)

    Google Scholar 

  7. Lever, G., Laviolette, F., Shawe-Taylor, J.: Distribution-dependent PAC-Bayes priors. In: Algorithmic Learning Theory, pp. 119–133. Springer, Berlin/Heidelberg (2010)

    Google Scholar 

  8. Maurer, A.: A note on the PAC Bayesian theorem. arXiv:cs/0411099 (2004, preprint)

    Google Scholar 

  9. McAllester, D.A.: PAC-Bayesian model averaging. In: COLT, Santa Cruz, pp. 164–170 (1999)

    Google Scholar 

  10. McAllester, D.: Simplified PAC-Bayesian margin bounds. In: Learning Theory and Kernel Machines, Washington, DC, pp. 203–215 (2003)

    Google Scholar 

  11. McAllester, D., Keshet, J.: Generalization bounds and consistency for latent structural probit and ramp loss. In: Proceedings of the 25th Annual Conference on Neural Information Processing Systems (NIPS), Granada (2011)

    Google Scholar 

  12. Seeger, M.: PAC-Bayesian generalisation error bounds for Gaussian process classification (English). J. Mach. Learn. Res. 3(2), 233–269 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  13. Shawe-Taylor, J., Langford, J.: PAC-Bayes & margins. In: Advances in Neural Information Processing Systems 15: Proceedings of the 2002 Conference, Vancouver, vol. 15, p. 439. MIT (2003)

    Google Scholar 

  14. Valiant, L.G.: A theory of the learnable. Commun. ACM 27(11), 1134–1142 (1984)

    Article  MATH  Google Scholar 

  15. Vapnik, V.N., Chervonenkis, A.Y.: On the uniform convergence of relative frequencies of events to their probabilities. Theory Probab. Appl. 16(2), 264–280 (1971)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Toyota Technological Institute at Chicago, 6045 S. Kenwood Avenue, Chicago, IL, 60637, USA

    David McAllester

  2. Statistics Department, University of Chicago, 5734 S. University Avenue, Chicago, IL, 60637, USA

    Takintayo Akinbiyi

Authors
  1. David McAllester
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Takintayo Akinbiyi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David McAllester .

Editor information

Editors and Affiliations

  1. Max Planck Institute for Intelligent Systems, Tübingen, Germany

    Bernhard Schölkopf

  2. Dept. of Computer Science, Royal Holloway, University of London, Egham, Surrey, United Kingdom

    Zhiyuan Luo

  3. Department of Computer Science, Royal Holloway, University of London, Egham, Surrey, United Kingdom

    Vladimir Vovk

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

McAllester, D., Akinbiyi, T. (2013). PAC-Bayesian Theory. In: Schölkopf, B., Luo, Z., Vovk, V. (eds) Empirical Inference. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-41136-6_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-41136-6_10

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-41135-9

  • Online ISBN: 978-3-642-41136-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation