Skip to main content
SpringerLink
Log in

Some Remarks on the Statistical Analysis of SVMs and Related Methods

  • Chapter
  • First Online:
Empirical Inference

Abstract

Since their invention by Vladimir Vapnik and his co-workers in the early 1990s, support vector machines (SVMs)Support vector machine (SVM)—( have attracted a lot of research activities from various communities. While at the beginning this research mostly focused on generalization bounds, the last decade witnessed a shift towards consistencyConsistency, oracle inequalities, and learning ratesLearning rate. We discuss some of these developments in view of binary classificationBinary classification and least squares regressionLeast squares regression.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    In [15] the authors actually give some credit to the 1987 paper [35] for the case d = 1.

References

  1. Aizerman, M., Braverman, E., Rozonoer, L.: Theoretical foundations of the potential function method in pattern recognition learning. Autom. Remote Control 25, 821–837 (1964)

    MathSciNet  Google Scholar 

  2. Bartlett, P., Jordan, M., McAuliffe, J.: Convexity, classification, and risk bounds. J. Am. Stat. Assoc. 101, 138–156 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bauer, H.: Measure and Integration Theory. De Gruyter, Berlin (2001)

    Book  MATH  Google Scholar 

  4. Bennett, C., Sharpley, R.: Interpolation of Operators. Academic, Boston (1988)

    MATH  Google Scholar 

  5. Bergh, J., Löfström, J.: Interpolation Spaces: An Introduction. Springer, New York (1976)

    Book  MATH  Google Scholar 

  6. Boser, B., Guyon, I., Vapnik, V.: A training algorithm for optimal margin classifiers. In: Proceedings of the 5th Annual ACM Workshop on Computational Learning Theory, Pittsburgh, pp. 144–152 (1992)

    Google Scholar 

  7. Caponnetto, A., De Vito, E.: Optimal rates for regularized least squares algorithm. Found. Comput. Math. 7, 331–368 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  8. Cortes, C., Vapnik, V.: Support vector networks. Mach. Learn. 20, 273–297 (1995)

    MATH  Google Scholar 

  9. Cucker, F., Smale, S.: On the mathematical foundations of learning. Bull. Am. Math. Soc. 39, 1–49 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  10. Devroye, L.: Any discrimination rule can have an arbitrarily bad probability of error for finite sample size. IEEE Trans. Pattern Anal. Mach. Intell. 4, 154–157 (1982)

    Article  MATH  Google Scholar 

  11. Devroye, L., Györfi, L., Lugosi, G.: A Probabilistic Theory of Pattern Recognition. Springer, New York (1996)

    Book  MATH  Google Scholar 

  12. Drucker, H., Burges, C., Kaufman, L., Smola, A., Vapnik, V.: Support vector regression machines. In: Advances in Neural Information Processing Systems, Denver, vol. 9, pp. 155–161. MIT, Cambridge (1997)

    Google Scholar 

  13. Eberts, M., Steinwart, I.: Optimal regression rates for SVMs using Gaussian kernels. Electron. J. Stat. 7, 1–42 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  14. Edmunds, D., Triebel, H.: Function Spaces, Entropy Numbers, Differential Operators. Cambridge University Press, Cambridge (1996)

    Book  MATH  Google Scholar 

  15. Györfi, L., Kohler, M., Krzyżak, A., Walk, H.: A Distribution-Free Theory of Nonparametric Regression. Springer, New York (2002)

    Book  MATH  Google Scholar 

  16. Kimeldorf, G., Wahba, G.: Some results on Tchebycheffian spline functions. J. Math. Anal. Appl. 33, 82–95 (1971)

    Article  MathSciNet  MATH  Google Scholar 

  17. Kohler, M., Krzyżak, A.: Nonparametric regression estimation using penalized least squares. IEEE Trans. Inform. Theory 47, 3054–3058 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  18. Kohler, M., Krzyżak, A., Schäfer, D.: Application of structural risk minimization to multivariate smoothing spline regression estimates. Bernoulli 4, 475–489 (2002)

    Google Scholar 

  19. Koltchinskii, V., Beznosova, O.: Exponential convergence rates in classification. In: Proceedings of the 18th Annual Conference on Learning Theory, Bertinoro, pp. 295–307 (2005)

    Google Scholar 

  20. Loustau, S.: Aggregation of SVM classifiers using Sobolev spaces. J. Mach. Learn. Res. 9, 1559–1582 (2008)

    MathSciNet  MATH  Google Scholar 

  21. Mendelson, S., Neeman, J.: Regularization in kernel learning. Ann. Stat. 38, 526–565 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  22. Micchelli, C., Xu, Y., Zhang, H.: Universal kernels. J. Mach. Learn. Res. 7, 2651–2667 (2006)

    MathSciNet  MATH  Google Scholar 

  23. Poggio, T., Girosi, F.: A theory of networks for approximation and learning. Proc. IEEE 78, 1481–1497 (1990)

    Article  Google Scholar 

  24. Silverman, B.: Some aspects of the spline smoothing approach to nonparametric regression. J. Royal Stat. Soc. B Stat. Methodol. 47, 1–52 (1985)

    MATH  Google Scholar 

  25. Smale, S., Zhou, D.: Estimating the approximation error in learning theory. Anal. Appl. 1, 17–41 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  26. Steinwart, I.: On the influence of the kernel on the consistency of support vector machines. J. Mach. Learn. Res. 2, 67–93 (2001)

    MathSciNet  Google Scholar 

  27. Steinwart, I.: Support vector machines are universally consistent. J. Complexity 18, 768–791 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  28. Steinwart, I., Christmann, A.: Support Vector Machines. Springer, New York (2008)

    MATH  Google Scholar 

  29. Steinwart, I., Christmann, A.: Estimating conditional quantiles with the help of the pinball loss. Bernoulli 17, 211–225 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  30. Steinwart, I., Scovel, C.: Fast rates for support vector machines using Gaussian kernels. Ann. Stat. 35, 575–607 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  31. Steinwart, I., Scovel, C.: Mercer’s theorem on general domains: on the interaction between measures, kernels, and RKHSs. Constr. Approx. 35, 363–417 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  32. Steinwart, I., Hush, D., Scovel, C.: Optimal rates for regularized least squares regression. In: Proceedings of the 22nd Annual Conference on Learning Theory, Montreal, pp. 79–93 (2009)

    Google Scholar 

  33. Stone, C.: Consistent nonparametric regression. Ann. Stat. 5, 595–645 (1977)

    Article  MATH  Google Scholar 

  34. Tsybakov, A.: Optimal aggregation of classifiers in statistical learning. Ann. Stat. 32, 135–166 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  35. van de Geer, S.: A new approach to least squares estimation, with applications. Ann. Stat. 15, 587–602 (1987)

    Article  MATH  Google Scholar 

  36. Vapnik, V.: The Nature of Statistical Learning Theory. Springer, New York (1995)

    Book  MATH  Google Scholar 

  37. Vapnik, V., Lerner, A.: Pattern recognition using generalized portrait method. Autom. Remote Control 24, 774–780 (1963)

    Google Scholar 

  38. Vapnik, V., Golowich, S., Smola, A.: Support vector method for function approximation, regression estimation, and signal processing. In: Advances in Neural Information Processing Systems, Denver, vol. 9, pp. 81–287. MIT, Cambridge (1997)

    Google Scholar 

  39. Wahba, G.: Spline Models for Observational Data. Series in Applied Mathematics, vol. 59. SIAM, Philadelphia (1990)

    Google Scholar 

  40. Zhang, T.: Statistical behaviour and consistency of classification methods based on convex risk minimization. Ann. Stat. 32, 56–134 (2004)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute for Stochastics and Applications, University of Stuttgart, Pfaffenwaldring 57, D-70569, Stuttgart, Germany

    Ingo Steinwart

Authors
  1. Ingo Steinwart
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ingo Steinwart .

Editor information

Editors and Affiliations

  1. Max Planck Institute for Intelligent Systems, Tübingen, Germany

    Bernhard Schölkopf

  2. Dept. of Computer Science, Royal Holloway, University of London, Egham, Surrey, United Kingdom

    Zhiyuan Luo

  3. Department of Computer Science, Royal Holloway, University of London, Egham, Surrey, United Kingdom

    Vladimir Vovk

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Steinwart, I. (2013). Some Remarks on the Statistical Analysis of SVMs and Related Methods. In: Schölkopf, B., Luo, Z., Vovk, V. (eds) Empirical Inference. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-41136-6_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-41136-6_4

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-41135-9

  • Online ISBN: 978-3-642-41136-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation