Abstract
We address the problem of optimizing kernel parameters in Support Vector Machine modelling, especially when the number of parameters is greater than one as in polynomial kernels and KMOD, our newly introduced kernel. The present work is an extended experimental study of the framework proposed by Chapelle et al. for optimizing SVM kernels using an analytic upper bound of the error. However, our optimization scheme minimizes an empirical error estimate using a Quasi-Newton technique. The method has shown to reduce the number of support vectors along the optimization process. In order to assess our contribution, the approach is further used for adapting KMOD, RBF and polynomial kernels on synthetic data and NIST digit image database. The method has shown satisfactory results with much faster convergence in comparison with the simple gradient descent method.
Furthermore, we also experimented two more optimization schemes based respectively on the maximization of the margin and on the minimization of an approximated VC dimension estimate. While both of the objective functions are minimized, the error is not. The corresponding experimental results we carried out show this shortcoming.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
N. E. Ayat, M. Cheriet, and C. Y. Suen. Un système neuro-flou pour la reconnaissance de montants numériques de chèques arabes. In CIFED, pages 171–180, Lyon,France, Jul. 2000.
N. E. Ayat, M. Cheriet, and C. Y. Suen. Kmod-a new support vector machine kernel for pattern recognition. application to digit image recognition. In ICDAR, pages 1215–1219, Seattle,USA, Sept. 2001.
N. E. Ayat, M. Cheriet, and C. Y. Suen. Kmod-a two parameter svm kernel for pattern recognition. to appear in icpr 2002. quebec city, canada, 2002, 2002.
Y. Bengio. Gradient-based optimization of hyper-parameters. Neural Computation, 12(8):1889–1900, 2000.
B. Boser, I. Guyon, and V. Vapnik. A training algorithm for optimal margin classifiers. In Fifth Annual Workshop on Computational Learning Theory, Pittsburg, 1992.
C. Cortes and V. Vapnik. Support-vector networks. Machine Learning, 20(3):273–297, 1995.
Courant and R. Hilbert. Methods of Mathematical Physics. Interscience, 1953.
G. Wahba. The bias-variance trade-off and the randomized gacv. Advances in Neural Information Processing Systems, 11(5), November 1999.
T. Joachims. Making large-scale svm learning practical. In B. Scholkopf, C. Burges, and A. Smola, editors, Advances in Kernel Methods — Support Vector Learning, chapter 11. 1999.
J. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3), October 1999.
U. Kreβel. Pairwise classification and support vector machines. In B. Scholkopf, C. Burges, and A. Smola, editors, Advances in Kernel Methods — Support Vector Learning, chapter Chap.15, pages 255–268. 1999.
J. Larsen, C. Svarer, L. N. Andersen, and L. K. Hansen. Adaptive regularization in neural network modeling. In Neural Networks: Tricks of the Trade, pages 113–132, 1996.
O. Chapelle and V. Vapnik. Choosing multiple parameters for support vector machines. Advances in Neural Information Processing Systems, 03(5), March 2001.
W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling. Numerical Recipes in C, the art of scientific computing. Cambridge University Press, second edition, 1992.
G. Rätsch, T. Onoda, and K.-R. Müller. Soft margins for adaboost. Machine Learning, 43(3):287–320, 2001.
B. Scholkopf. Support vector learning. PhD thesis, Universität Berlin, Berlin, Germany, 1997.
B. Scholkopf, C. Burges, and A. Smola. Introduction to support vector learning. In B. Scholkopf, C. Burges, and A. Smola, editors, Advances in Kernel Methods — Support Vector Learning, chapter 1. 1999.
P. Sollich. Bayesian methods for support vector machines: Evidence and predictive class probabilities. Machine Learning, 46(1/3):21, 2002.
V. Vapnik. The Nature of Statistical Learning Theory. NY, USA, 1995.
V. Vapnik. An overview of statistical learning theory. IEEE Transactions on Neural Networks, 10(5), September 1999.
D. S. Watkins. Fundamentals of Matrix Computations. Wiley, New York, 1991.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ayat, NE., Cheriet, M., Suen, C.Y. (2002). Optimization of the SVM Kernels Using an Empirical Error Minimization Scheme. In: Lee, SW., Verri, A. (eds) Pattern Recognition with Support Vector Machines. SVM 2002. Lecture Notes in Computer Science, vol 2388. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45665-1_28
Download citation
DOI: https://doi.org/10.1007/3-540-45665-1_28
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44016-1
Online ISBN: 978-3-540-45665-0
eBook Packages: Springer Book Archive