Skip to main content
SpringerLink
Log in

Approximation and Estimation Bounds for Subsets of Reproducing Kernel Kreǐn Spaces

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Reproducing kernel Kreǐn spaces are used in learning from data via kernel methods when the kernel is indefinite. In this paper, a characterization of a subset of the unit ball in such spaces is provided. Conditions are given, under which upper bounds on the estimation error and the approximation error can be applied simultaneously to such a subset. Finally, it is shown that the hyperbolic-tangent kernel and other indefinite kernels satisfy such conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. By a domain, we mean the closure of an open and connected set.

  2. For \(d=1\), such a set \(\varOmega \) is a closed and bounded interval.

  3. Note that [43, Theorem 8] actually provides an upper bound of order \(O(d^4)\) on the VC dimension of the family of binary-valued functions obtained by thresholding functions belonging to \(\mathcal{F}\). However, an upper bound of the same order \(O(d^4)\) is obtained on the VC dimension of the real-valued family of functions \(\mathcal{F}\) by recalling the relationship between the VC dimension of a family of real-valued functions and the one of the family of binary-valued functions obtained by adding a bias unit and thresholding the resulting output [6, Sect. 3.6].

References

  1. Shawe-Taylor J, Cristianini N (2004) Kernel methods for pattern analysis. Cambridge University Press, Cambridge

    Book  Google Scholar 

  2. Aronszajn N (1950) Theory of reproducing kernels. Trans Am Math Soc 68:337–404

    Article  MATH  MathSciNet  Google Scholar 

  3. Schölkopf B, Smola A (2002) Learning with kernels. MIT Press, Cambridge

    Google Scholar 

  4. Bishop CM (1995) Neural networks for pattern recognition. Oxford University Press, Oxford

    Google Scholar 

  5. Vapnik VP (1998) Statistical learning theory. Springer, Heidelberg

    MATH  Google Scholar 

  6. Vapnik VP (1995) The nature of statistical learning theory. Springer, Berlin

    Book  MATH  Google Scholar 

  7. Mendelson S (2003) A few notes on statistical learning theory. Advanced lectures on machine learning. Springer, New York, pp 1–40

  8. Smola AJ, Óvári ZL, Williamson RC (2001) Regularization with dot-product kernels. In: Leen T, Dietterich T, Tresp V (eds) Proceedings of neural information processing systems 13, pp 308–314

  9. Lin HT, Lin CJ (2003) A study on sigmoid kernels for SVM and the training of non-PSD kernels by SMO-type methods. National Taiwan University, Technical report

  10. Ramon J, Gärtner T (2003) Expressivity versus efficiency of graph kernels. In: Washio T, De Raedt L (eds) Proceedings of 1st international workshop on mining graphs, trees and sequences, pp 65–74

  11. Borgwardt KM, Kriegel HP (2005) Shortest path kernels on graphs. In: Proceedings of 5th IEEE international conference on data mining, Washington, pp 74–81

  12. Haasdonk B, Keysers D (2002) Tangent distance kernels for support vector machines. In: Proceedings of 16th international conference on pattern recognition, pp 864–868

  13. Saigo H, Vert J, Ueda N, Akutsu T (2004) Protein homology detection using string alignment kernels. Bioinformatics 20:1682–1689

    Article  Google Scholar 

  14. Wu G, Chang EY, Zhang Z (2005) An analysis of transformation on non-positive semidefinite similarity matrix for kernel machines. In: Proceedings of 22nd international conference on machine learning, pp 315–322

  15. Haasdonk B, Bahlmann C (2004) Learning with distance substitution kernels. In: Proceedings of 26th DAGM symposium on pattern recognition, pp 220–227

  16. Luss R, d’Aspremont A (2007) Support vector machine classification with indefinite kernels. In: Proceedings of NIPS 2007, pp 1–9

  17. Chen J, Ye J (2008) Training SVM with non indefinite kernels. In: Proceedings of 25th international conference on machine learning, New York, pp 136–143

  18. Ying Y, Campbell C, Girolami M (2009) Analysis of SVM with indefinite kernels. In: Proceedings of NIPS 2009, Vancouver, pp 2205–2213

  19. Luss R, d’Aspremont A (2009) Support vector machine classification with indefinite kernels. Math Prog Comput 1:97–118

    Article  MATH  MathSciNet  Google Scholar 

  20. Haasdonk B (2005) Feature space interpretation of SVMs with indefinite kernels. IEEE Trans Pattern Anal Mach Intell 27(4):482–492

    Google Scholar 

  21. Liwicki S, Zafeiriou S, Tzimiropoulos G, Pantic M (2012) Efficient online subspace learning with an indefinite kernel for visual tracking and recognition. IEEE Trans Neural Netw Learn Syst 3:1624–1636

    Article  Google Scholar 

  22. Bartlett PL, Mendelson S (2002) Rademacher and Gaussian complexities: risk bounds and structural results. J Mach Learn Res 3:463–482

    Google Scholar 

  23. Ong CS, Mary X, Canu S, Smola AJ (2004) Learning with non positive kernels. In: Proceedings of 21st international conference on machine learning, pp 639–646

  24. Gnecco G, Sanguineti M (2008) Approximation error bounds via Rademacher’s complexity. Appl Math Sci 2:153–176

    MATH  MathSciNet  Google Scholar 

  25. Gnecco G, Sanguineti M (2008) Estimates of the approximation error via Rademacher complexity: learning vector-valued functions. J Inequal Appl 2008(640758):16

    MathSciNet  Google Scholar 

  26. Gnecco G, Sanguineti M (2010) Regularization techniques and suboptimal solutions to optimization problems in learning from data. Neural Comput 22:793–829

    Article  MATH  MathSciNet  Google Scholar 

  27. Gnecco G, Gori M, Sanguineti M (2013) Learning with boundary conditions. Neural Comput 25 (in press)

  28. Anguita D, Ghio A, Oneto L, Ridella S (2011) Maximal discrepancy vs. Rademacher complexity for error estimation. In: Proceedings of ESANN 2011, pp 257–262

  29. Anguita D, Ghio A, Oneto L, Ridella S (2012) In-sample model selection for trimmed hinge loss support vector machine. Neural Process Lett 36:275–283

    Article  Google Scholar 

  30. Friedman A (1992) Foundations of modern analysis. Dover, New York

    Google Scholar 

  31. Bognar J (1974) Indefinite inner product spaces. Springer, Berlin

    Book  MATH  Google Scholar 

  32. Birman MS, Solomjak MZ (1987) Spectral theory of self-adjoint operators in Hilbert space. D. Reidel Publishing Company, Dordrecht

    MATH  Google Scholar 

  33. Tricomi FG (1985) Integral equations. Dover, New York

    Google Scholar 

  34. Smirnov VI (1964) A course of higher mathematics: integration and functional analysis, vol 5. Addison-Wesley, Reading

    Google Scholar 

  35. Girosi F (1995) Approximation error bounds that use VC-bounds. In: Proceedings of international conference on artificial neural networks, pp 295–302

  36. Barron AR (1992) Neural net approximation. In: Proceedings of 7th Yale workshop on adaptive and learning systems, New Haven, pp 69–72

  37. Barron AR (1993) Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans Inf Theory 39:930–945

    Article  MATH  MathSciNet  Google Scholar 

  38. K\(\mathring{\rm u}\)rková V, Kainen PC, Kreinovich V (1997) Estimates of the number of hidden units and variation with respect to half-spaces. Neural Netw 10:1061–1068

    Google Scholar 

  39. Tsybakov AB (2008) Introduction to nonparametric estimation. Springer, New York

    Google Scholar 

  40. Cucker F, Zhou DX (2007) Learning theory: an approximation theory viewpoint. Cambridge University Press, Cambridge

    Book  Google Scholar 

  41. Loosli G, Canu S (2011) Non positive SVM. In: OPT NIPS Workshop, pp 1–6

  42. Schwabik S, Ye G (2005) Topics in banach space integration. World Scientific, Singapore

    MATH  Google Scholar 

  43. Bartlett PL, Mass W (2003) The handbook of brain theory and neural networks. In: Arbib MA (ed) Vapnik–Chervonenkis dimension of neural nets, 2nd edn. MIT Press, Cambridge, pp 1188–1192

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. DIBRIS, University of Genoa, Via Opera Pia 13, 16145 , Genoa, Italy

    Giorgio Gnecco

Authors
  1. Giorgio Gnecco
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giorgio Gnecco.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gnecco, G. Approximation and Estimation Bounds for Subsets of Reproducing Kernel Kreǐn Spaces. Neural Process Lett 39, 137–153 (2014). https://doi.org/10.1007/s11063-013-9294-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-013-9294-9

Keywords

Search

Navigation