Elsevier

Neurocomputing

Volume 70, Issues 4–6, January 2007, Pages 1089-1095
Neurocomputing

Letters
Support vector perceptrons☆

https://doi.org/10.1016/j.neucom.2006.08.001Get rights and content

Abstract

Due to their excellent performance, support vector machines (SVMs) are now used extensively in pattern classification applications. In this paper we show that the standard sigmoidal kernel definition lacks the capability to represent the family of perceptrons, and we propose an improved SVM with a sigmoidal kernel called support vector perceptron (SVP). We show by means of both synthetic and real world data sets that the proposed SVP is able to provide very accurate results in many classification problems, providing maximal margin solutions when classes are separable, and also producing very compact architectures comparable to classical multilayer perceptrons.

Section snippets

Introduction: support vector machines with sigmoidal kernels

Recently support vector machines (SVMs) have been extensively used by the machine learning community because they effectively deal with high dimensional data, provide good generalization properties and define the classifier architecture in terms of the so-called support vectors (SVs), once the hyperparameters are set (usually by means of a cross-validation procedure) [14], [17]. Nonlinear SVMs are obtained by mapping input patterns to a feature space F, such that all operations comprising inner

The SVP algorithm

In the standard SVM formulation we have little control over the kernel hyperparameters once the QP optimization starts, since they are fixed beforehand. We therefore need a more flexible scheme to be able to select good kernel hyperparameters as learning progresses, and without the restriction δi=δ0,∀i. We propose to take advantage of a previously developed method to grow semiparametric models [10], [11]. Under this paradigm, the size of the classifier can be effectively controlled by

Experiments

We have benchmarked the proposed SVP algorithm against the standard SVM with sigmoidal kernel (sigmoid-LibSVM) trained with the LibSVM software on several data sets from the UCI Machine Learning repository, as well as other synthetic data sets. We also provide results with a Gaussian kernel (RBF-LibSVM), for reference, and with a linear model, for baseline comparison, since some of the data sets admit a reasonably good linear solution. For all algorithms hyperparameters have been selected using

Conclusions and further work

We have proposed an improved modification of the support vector machine (SVM) with sigmoidal kernel, the support vector perceptron (SVP) method, comprising a training algorithm based on iterated weighted least squares minimizations and a procedure for iteratively selecting the best basis elements to build up the machine architecture. The SVP method was shown to yield very accurate results and compact classifier architectures (analogous to single hidden layer multilayer perceptrons) in a variety

Angel Navia-Vázquez received his Degree in Telecommunications Engineering in 1992 (Universidad de Vigo, Spain), and finished his PhD, also in Telecommunications Engineering in 1997 (Universidad Politécnica de Madrid, Spain). He is now an Associate Professor at the Department of Signal Theory and Communications, Universidad Carlos III de Madrid, Spain. His research interests are focused on new architectures and algorithms for nonlinear processing, as well as their application to multimedia

References (17)

  • G. Camps-Valls et al.

    Fuzzy sigmoid kernel for support vector classifiers

    Neurocomputing

    (2004)
  • E. Parrado-Hernández et al.

    Growing support vector classifiers with controlled complexity

    Pattern Recognition

    (2003)
  • N. Barakat et al.

    Eclectic rule-extraction from support vector machines

    Int. J. Comp. Intelligence

    (2005)
  • G. Bologna, Rule extraction from a multi layer perceptron with staircase activation functions, IEEE-INNS-ENNS...
  • M. Costa, E. Filippi, E. Pasero, Multi-layer perceptron ensembles for pattern recognition: some experiments, IEEE World...
  • M. Costa, P. Gay, D. Palmisano, E. Pasero, A neural ensemble for speech recognition, in: Proceedings of the IEEE...
  • S. Delshadpour, Reduced size multi layer perceptron neural network for human chromosome classification, Proceedings of...
  • Y.H. Hu, Q. Xue, W.J. Tompkins, Structural simplification of a feed-forward, multilayer perceptron artificial neural...
There are more references available in the full text version of this article.

Cited by (6)

Angel Navia-Vázquez received his Degree in Telecommunications Engineering in 1992 (Universidad de Vigo, Spain), and finished his PhD, also in Telecommunications Engineering in 1997 (Universidad Politécnica de Madrid, Spain). He is now an Associate Professor at the Department of Signal Theory and Communications, Universidad Carlos III de Madrid, Spain. His research interests are focused on new architectures and algorithms for nonlinear processing, as well as their application to multimedia processing, communications, data mining, content management and E-learning. He has (co)authored 17 international refereed journal papers in these areas, several book chapters, more than 40 conference communications, and participated in more than 20 research projects. He is IEEE (Senior) Member since 1999 and Associate Editor of IEEE Trans. Neural Networks since January 2004.

☆

This work has been partially supported by Spain CICYT Grant TEC2005-04264/TCM and CAM Grant PRO.MULTIDIS S-0505/TIC/0223.

View full text
Copyright © 2006 Elsevier B.V. All rights reserved.