Abstract
Reproducing kernel Kreǐn spaces are used in learning from data via kernel methods when the kernel is indefinite. In this paper, a characterization of a subset of the unit ball in such spaces is provided. Conditions are given, under which upper bounds on the estimation error and the approximation error can be applied simultaneously to such a subset. Finally, it is shown that the hyperbolic-tangent kernel and other indefinite kernels satisfy such conditions.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
By a domain, we mean the closure of an open and connected set.
For \(d=1\), such a set \(\varOmega \) is a closed and bounded interval.
Note that [43, Theorem 8] actually provides an upper bound of order \(O(d^4)\) on the VC dimension of the family of binary-valued functions obtained by thresholding functions belonging to \(\mathcal{F}\). However, an upper bound of the same order \(O(d^4)\) is obtained on the VC dimension of the real-valued family of functions \(\mathcal{F}\) by recalling the relationship between the VC dimension of a family of real-valued functions and the one of the family of binary-valued functions obtained by adding a bias unit and thresholding the resulting output [6, Sect. 3.6].
References
Shawe-Taylor J, Cristianini N (2004) Kernel methods for pattern analysis. Cambridge University Press, Cambridge
Aronszajn N (1950) Theory of reproducing kernels. Trans Am Math Soc 68:337–404
Schölkopf B, Smola A (2002) Learning with kernels. MIT Press, Cambridge
Bishop CM (1995) Neural networks for pattern recognition. Oxford University Press, Oxford
Vapnik VP (1998) Statistical learning theory. Springer, Heidelberg
Vapnik VP (1995) The nature of statistical learning theory. Springer, Berlin
Mendelson S (2003) A few notes on statistical learning theory. Advanced lectures on machine learning. Springer, New York, pp 1–40
Smola AJ, Óvári ZL, Williamson RC (2001) Regularization with dot-product kernels. In: Leen T, Dietterich T, Tresp V (eds) Proceedings of neural information processing systems 13, pp 308–314
Lin HT, Lin CJ (2003) A study on sigmoid kernels for SVM and the training of non-PSD kernels by SMO-type methods. National Taiwan University, Technical report
Ramon J, Gärtner T (2003) Expressivity versus efficiency of graph kernels. In: Washio T, De Raedt L (eds) Proceedings of 1st international workshop on mining graphs, trees and sequences, pp 65–74
Borgwardt KM, Kriegel HP (2005) Shortest path kernels on graphs. In: Proceedings of 5th IEEE international conference on data mining, Washington, pp 74–81
Haasdonk B, Keysers D (2002) Tangent distance kernels for support vector machines. In: Proceedings of 16th international conference on pattern recognition, pp 864–868
Saigo H, Vert J, Ueda N, Akutsu T (2004) Protein homology detection using string alignment kernels. Bioinformatics 20:1682–1689
Wu G, Chang EY, Zhang Z (2005) An analysis of transformation on non-positive semidefinite similarity matrix for kernel machines. In: Proceedings of 22nd international conference on machine learning, pp 315–322
Haasdonk B, Bahlmann C (2004) Learning with distance substitution kernels. In: Proceedings of 26th DAGM symposium on pattern recognition, pp 220–227
Luss R, d’Aspremont A (2007) Support vector machine classification with indefinite kernels. In: Proceedings of NIPS 2007, pp 1–9
Chen J, Ye J (2008) Training SVM with non indefinite kernels. In: Proceedings of 25th international conference on machine learning, New York, pp 136–143
Ying Y, Campbell C, Girolami M (2009) Analysis of SVM with indefinite kernels. In: Proceedings of NIPS 2009, Vancouver, pp 2205–2213
Luss R, d’Aspremont A (2009) Support vector machine classification with indefinite kernels. Math Prog Comput 1:97–118
Haasdonk B (2005) Feature space interpretation of SVMs with indefinite kernels. IEEE Trans Pattern Anal Mach Intell 27(4):482–492
Liwicki S, Zafeiriou S, Tzimiropoulos G, Pantic M (2012) Efficient online subspace learning with an indefinite kernel for visual tracking and recognition. IEEE Trans Neural Netw Learn Syst 3:1624–1636
Bartlett PL, Mendelson S (2002) Rademacher and Gaussian complexities: risk bounds and structural results. J Mach Learn Res 3:463–482
Ong CS, Mary X, Canu S, Smola AJ (2004) Learning with non positive kernels. In: Proceedings of 21st international conference on machine learning, pp 639–646
Gnecco G, Sanguineti M (2008) Approximation error bounds via Rademacher’s complexity. Appl Math Sci 2:153–176
Gnecco G, Sanguineti M (2008) Estimates of the approximation error via Rademacher complexity: learning vector-valued functions. J Inequal Appl 2008(640758):16
Gnecco G, Sanguineti M (2010) Regularization techniques and suboptimal solutions to optimization problems in learning from data. Neural Comput 22:793–829
Gnecco G, Gori M, Sanguineti M (2013) Learning with boundary conditions. Neural Comput 25 (in press)
Anguita D, Ghio A, Oneto L, Ridella S (2011) Maximal discrepancy vs. Rademacher complexity for error estimation. In: Proceedings of ESANN 2011, pp 257–262
Anguita D, Ghio A, Oneto L, Ridella S (2012) In-sample model selection for trimmed hinge loss support vector machine. Neural Process Lett 36:275–283
Friedman A (1992) Foundations of modern analysis. Dover, New York
Bognar J (1974) Indefinite inner product spaces. Springer, Berlin
Birman MS, Solomjak MZ (1987) Spectral theory of self-adjoint operators in Hilbert space. D. Reidel Publishing Company, Dordrecht
Tricomi FG (1985) Integral equations. Dover, New York
Smirnov VI (1964) A course of higher mathematics: integration and functional analysis, vol 5. Addison-Wesley, Reading
Girosi F (1995) Approximation error bounds that use VC-bounds. In: Proceedings of international conference on artificial neural networks, pp 295–302
Barron AR (1992) Neural net approximation. In: Proceedings of 7th Yale workshop on adaptive and learning systems, New Haven, pp 69–72
Barron AR (1993) Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans Inf Theory 39:930–945
K\(\mathring{\rm u}\)rková V, Kainen PC, Kreinovich V (1997) Estimates of the number of hidden units and variation with respect to half-spaces. Neural Netw 10:1061–1068
Tsybakov AB (2008) Introduction to nonparametric estimation. Springer, New York
Cucker F, Zhou DX (2007) Learning theory: an approximation theory viewpoint. Cambridge University Press, Cambridge
Loosli G, Canu S (2011) Non positive SVM. In: OPT NIPS Workshop, pp 1–6
Schwabik S, Ye G (2005) Topics in banach space integration. World Scientific, Singapore
Bartlett PL, Mass W (2003) The handbook of brain theory and neural networks. In: Arbib MA (ed) Vapnik–Chervonenkis dimension of neural nets, 2nd edn. MIT Press, Cambridge, pp 1188–1192
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Gnecco, G. Approximation and Estimation Bounds for Subsets of Reproducing Kernel Kreǐn Spaces. Neural Process Lett 39, 137–153 (2014). https://doi.org/10.1007/s11063-013-9294-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-013-9294-9