Abstract
Designing supervised learning systems is in general a multi-objective optimization problem. It requires finding appropriate trade-offs between several objectives, for example between model complexity and accuracy or sensitivity and specificity. We consider the adaptation of kernel and regularization parameters of support vector machines (SVMs) by means of multi-objective evolutionary optimization. Support vector machines are reviewed from the multi-objective perspective, and different encodings and model selection criteria are described. The optimization of split modified radius-margin model selection criteria is demonstrated on benchmark problems. The MOO approach to SVM design is evaluated on a real-world pattern recognition task, namely the real-time detection of pedestrians in infrared images for driver assistance systems. Here the three objectives are the minimization of the false positive rate, the false negative rate, and the number of support vectors to reduce the computational complexity.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Hussein A. Abbass. An evolutionary artificial neural networks approach for breast cancer diagnosis. Artificial Intelligence in Medicine, 25(3):265–281, 2002.
Hussein A. Abbass. Speeding up backpropagation using multiobjective evolutionary algorithms. Neural Computation, 15(11):2705–2726, 2003.
Massimo Bertozzi, Alberto Broggi, Marcello Carletti, Alessandra Fascioli, Thorsten Graf, Paolo Grisleri, and Michael Meinecke. IR pedestrian detection for advanced driver assistance systems. In Proceedings of the 25th Pattern Recognition Symposium, volume 2781 of LNCS, pages 582–590. Springer-Verlag, 2003.
Massimo Bertozzi, Alberto Broggi, Alessandra Fascioli, and Massimiliano Sechi. Shape-based pedestrian detection. In Proceedings of the IEEE Intelligent Vehicles Symposium 2000, pages 215–220, 2000.
Massimo Bertozzi, Alberto Broggi, Thorsten Graf, Paolo Grisleri, and Michael Meinecke. Pedestrian detection in infrared images. In Proceedings of the IEEE Intelligent Vehicles Symposium 2003, pages 662–667, 2003.
J. Bi. Multi-objective programming in SVMs. In Tom Fawcett and Nina Mishra, editors, Machine Learning, Proceedings of the 20th International Conference (ICML 2003), pages 35–42. AAAI Press, 2003.
C.L. Blake and C.J. Merz. UCI repository of machine learning databases, 1998.
J Canny. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(6):679–698, 1986.
O. Chapelle, V. Vapnik, O. Bousquet, and S. Mukherjee. Choosing multiple parameters for support vector machines. Machine Learning, 46(1):131–159, 2002.
K.-M. Chung, W.-C. Kao, C.-L. Sun, and C.-J. Lin. Radius margin bounds for support vector machines with RBF kernel. Neural Computation, 15(11):2643–2681, 2003.
Carlos A. Coello Coello, David A. Van Veldhuizen, and Gary B. Lamont. Evolutionary Algorithms for Solving Multi-Objective Problems. Kluwer Academic Publishers, 2002.
Nello Cristianini and John Shawe-Taylor. An Introduction to Support Vector Machines and other kernel-based learning methods. Cambridge University Press, 2000.
C. Curio, J. Edelbrunner, T. Kalinke, C. Tzomakas, and W. von Seelen. Walking pedestrian recognition. IEEE Transactions on Intelligent Transportation Systems, 1(3):155–163, 2000.
Kalyanmoy Deb. Multi-Objective Optimization Using Evolutionary Algorithms. Wiley, 2001.
Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2):182–197, 2002.
K. Duan, S. S. Keerthi, and A.N. Poo. Evaluation of simple performance measures for tuning SVM hyperparameters. Neurocomputing, 51:41–59, 2003.
Damian R. Eads, Daniel Hill, Sean Davis, Simon J. Perkins, Junshui Ma, Reid B. Porter, and James P. Theiler. Genetic algorithms and support vector machines for time series classification. In Bruno Bosacchi, David B. Fogel, and James C. Bezdek, editors, Applications and Science of Neural Networks, Fuzzy Systems, and Evolutionary Computation V., volume 4787 of Proceedings of the SPIE, pages 74–85, 2002.
C. M. Fonseca, J. D. Knowles, L. Thiele, and E. Zitzler. A tutorial on the performance assessment of stochastic multiobjective optimizers. Presented at the Third International Conference on Evolutionary Multi-Criterion Optimization (EMO 2005), 2005.
Frauke Friedrichs and Christian Igel. Evolutionary tuning of multiple SVM parameters. Neurocomputing, 64(C):107–117, 2005.
H. Fröhlich, O. Chapelle, and B. Schölkopf. Feature selection for support vector machines using genetic algorithms. International Journal on Artificial Intelligence Tools, 13(4):791–800, 2004.
Tobias Glasmachers and Christian Igel. Gradient-based adaptation of general gaussian kernels. Neural Computation, 17(10):2099–2105, 2005.
Carl Gold and Peter Sollich. Model selection for support vector machine classification. Neurocomputing, 55(1–2):221–249, 2003.
Nikolaus Hansen. Invariance, self-adaptation and correlated mutations and evolution strategies. In Proceedings of the 6th International Conference on Parallel Problem Solving from Nature (PPSN VI), volume 1917 of LNCS, pages 355–364. Springer-Verlag, 2000.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning Data Mining, Inference, and Prediction. Springer Series in Statistics. Springer-Verlag, 2001.
C. Igel. Multi-objective model selection for support vector machines. In C. A. Coello Coello, E. Zitzler, and A. Hernandez Aguirre, editors, Proceedings of the Third International Conference on Evolutionary Multi-Criterion Optimization (EMO 2005), volume 3410 of LNCS, pages 534–546. Springer-Verlag, 2005.
Yaochu Jin, Tatsuya Okabe, and Bernhard Sendhoff. Neural network regularization and ensembling using multi-objective evolutionary algorithms. In Congress on Evolutionary Computation (CEC’04), pages 1–8. IEEE Press, 2004.
Kees Jong, Elena Marchiori, and Aad van der Vaart. Analysis of proteomic pattern data for cancer detection. In G. R. Raidl, S. Cagnoni, J. Branke, D. W. Corne, R. Drechsler, Y. Jin, C. G. Johnson, P. Machado, E. Marchiori, F. Rothlauf, G. D. Smith, and G. Squillero, editors, Applications of Evolutionary Computing, volume 3005 of LNCS, pages 41–51. Springer-Verlag, 2004.
S. S. Keerthi. Efficient tuning of SVM hyperparameters using radius/margin bound and iterative algorithms. IEEE Transactions on Neural Networks, 13(5):1225–1229, 2002.
M. T. Miller, A. K. Jerebko, J. D. Malley, and R. M. Summers. Feature selection for computer-aided polyp detection using genetic algorithms. In Anne V. Clough and Amir A. Amini, editors, Medical Imaging 2003: Physiology and Function: Methods, Systems, and Applications, volume 5031 of Proceedings of the SPIE, pages 102–110, 2003.
Anuj Mohan, Constantine Papageorgiou, and Thomas Poggio. Example-based object detection in images by components. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(4):349–361, 2001.
Katharina Morik, Peter Brockhausen, and Thorsten Joachims. Combining statistical learning with a knowledge-based approach - a case study in intensive care monitoring. In Proceedings of the 16th International Conference on Machine Learning, pages 268–277. Morgan Kaufmann, 1999.
M. Oren, C. P. Papageorgiou, P. Sinha, E. Osuna, and T. Poggio. Pedestrian detection using wavelet templates. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 193–199, 1997.
S. Pang and N. Kasabov. Inductive vs. transductive inference, global vs. local models: SVM, TSVM, and SVMT for gene expression classification problems. In International Joint Conference on Neual Networks (IJCNN), volume 2, pages 1197–1202. IEEE Press, 2004.
C. Papageorgiou, T. Evgeniou, and T. Poggio. A trainable pedestrian detection system. In Proceedings of the IEEE International Conference on Intelligent Vehicles Symposium 1998, pages 241–246, 1998.
C. P. Papageorgiou. A trainable system for object detection in images and video sequences. Technical Report AITR-1685, Massachusetts Institute of Technology, Artificial Intelligene Laboratory, 2000.
Constantine Papageorgiou and Tomaso Poggio. A trainable system for object detection. International Journal of Computer Vision, 38(1):15–33, 2000.
G. Rätsch, T. Onoda, and K.-R. Müller. Soft margins for AdaBoost. Machine Learning, 42(3):287–320, 2001.
Günther Rudolph. On correlated mutations in evolution strategies. In R. Männer and B. Manderick, editors, Parallel Problem Solving from Nature 2 (PPSN II), pages 105–114. Elsevier, 1992.
Thomas Philip Runarsson and Sven Sigurdsson. Asynchronous parallel evolutionary model selection for support vector machines. Neural Information Processing - Letters and Reviews, 3(3):59–68, 2004.
Y. Sawaragi, H. Nakayama, and T. Tanino. Theory of Multiobjective Optimization, volume 176 of Mathematics in Science and Engineering. Academic Press, 1985.
B. Schölkopf, C. J. C. Burges, and V. Vapnik. Extracting support data for a given task. In U. M. Fayyad and R. Uthurusamy, editors, Proceedings of the First International Conference on Knowledge Discovery & Data Mining, pages 252–257, Menlo Park, CA, 1995. AAAI Press.
B. Schölkopf and A. J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, 2002.
John Shawe-Taylor and Nello Cristianini. Kernel Methods for Pattern Analysis. Cambridge University Press, 2004.
S. Y. M. Shi, P. N. Suganthan, and K. Deb. Multi-class protein fold recognition using multi-objective evolutionary algorithms. In IEEE Symposium on Computational Intelligence in Bioinformatics and Computational Biology, pages 61–66, 2004.
Vladimir N. Vapnik. The Nature of Statistical Learning Theory. Springer- Verlag, 1995.
S. Wiegand, C. Igel, and U. Handmann. Evolutionary multi-objective optimization of neural networks for face detection. International Journal of Computational Intelligence and Applications, 4(3):237–253, 2004. Special issue on Neurocomputing and Hybrid Methods for Evolving Intelligence.
F. Xu, X. Liu, and K. Fujimura. Pedestrian detection and tracking with night vision. IEEE Transactions on Intelligent Transportation Systems, 6(1):63–71, 2005.
L. Zhao and C. Thorpe. Stereo- and neural network-based pedestrian detection. In Proceedings of the IEEE International Conference on Intelligent Transportation Systems’99, pages 298–303, 1999.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer
About this chapter
Cite this chapter
Suttorp, T., Igel, C. (2006). Multi-Objective Optimization of Support Vector Machines. In: Jin, Y. (eds) Multi-Objective Machine Learning. Studies in Computational Intelligence, vol 16. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-33019-4_9
Download citation
DOI: https://doi.org/10.1007/3-540-33019-4_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-30676-4
Online ISBN: 978-3-540-33019-6
eBook Packages: EngineeringEngineering (R0)