Integrating support vector machines and neural networks

Abstract

Support vector machines (SVMs) are a powerful technique developed in the last decade to effectively tackle classification and regression problems. In this paper we describe how support vector machines and artificial neural networks can be integrated in order to classify objects correctly. This technique has been successfully applied to the problem of determining the quality of tiles. Using an optical reader system, some features are automatically extracted, then a subset of the features is determined and the tiles are classified based on this subset.

Introduction

Support vector machines (SVMs) are a novel and popular learning technique for solving different classification problems (Changet al., 2004, Cristianini and Shawe-Taylor, 2000, Vapnik, 1995) and data mining problems in various areas such as image processing, signal processing, pattern recognition, regression and so on.

A classification task usually involves a training set containing “target values” (class labels) and several “attributes” (features). The goal of SVMs is to produce a model that predicts target values for new data instances. More specifically, given a training set of attributes–label pairs

$$S≔\{(x^1,y_1),(x^2,y_2),\dots ,(x^N,y_N)\}$$

where $x^i\in {\mathbb{R}}^n$, $i=1,\dots ,N$ and $y_i\in \{-1,1\}$, $i=1,\dots ,N$ and a function $\varphi :{\mathbb{R}}^n\to {\mathbb{R}}^m$, the optimization problem

$$\begin{array}{cc}\underset{w,b,\xi }{min}\hfill & \frac{1}{2}{w}^{T}w+C\sum _{i=1}^N{\xi }_i\hfill \\ \text{subject to}\hfill & y_i(w^T\varphi \left(x^i\right)+b)\ge 1-{\xi }_i\text{,}i=1,\dots ,N\hfill \\ \hfill & {\xi }_i\ge 0,i=1,\dots ,N\hfill \end{array}$$

must be solved in order to obtain the vector $w^{\ast }$ and the scalar $b^{\ast }$. Successively, the classification function

$$\mathrm{sgn}\{w^{\ast T}\varphi \left(x\right)+b^{\ast }\}$$

is used to discriminate between the two sets of elements.

In Problem (1), the slack variable $\xi$, will assume nonzero values only in correspondence with points that are misclassified. The objective function has two terms: the first attempts to maximize the distance between the bounding planes, while the other minimizes the classification errors. The parameter $C\ge 0$ is introduced to balance the emphasis between these two goals. A small value of $C$ indicates that most of the importance has to be placed on separating the hyperplanes. A large value of $C$, on the contrary, indicates that it is important to reduce classification error. Therefore, finding the correct value of $C$ is typically an experimental task (Cherkassky and Ma, 2002, Joachims, 2002, Wahba et al., 2000), accomplished via a training set and cross-validation (Stone, 1974).

In addition to classification, support vector machines are used to effectively solve the feature selection problem. Again, while discriminating between two different classes of data, the most important features that allow separation of the two classes are also selected.

In this paper we present a novel procedure that integrates support vector machines and artificial neural networks in order to solve a specific real-world problem. Using SVMs we select the “best” features that will be used as inputs to the artificial neural networks. In Section 2 the feature selection problem is presented (see Bradley, Mangasarian, and Street (1998) and references therein for a more detailed explanation of the problem) with a parametric objective function and linear constraints. We briefly describe artificial neural networks in Section 3. In Section 4 the problem of determining the quality of tiles is described. Finally, Section 5 contains numerical test results as well as comparisons with two different normalizations.

We briefly describe our notation now. All vectors will be column vectors and will be indicated by a lower case italic letter ($x,y,\dots$). The scalar (inner) product of two vectors $x$ and $y$ in the $n$-dimensional real space ${\mathbb{R}}^n$ will be denoted by $x^{T}y$. A column vector of ones of arbitrary dimension will be denoted by $e$. Matrices will be indicated by an upper case italic letter. For a matrix $A$ we will denote the transpose with $A^T$. For a vector $v\in {\mathbb{R}}^n$, $v_{\ast }$ is the vector with components

$${\left(v_{\ast }\right)}_j=\{\begin{array}{cc}1& \text{if }{v}_j>0\\ 0& \text{otherwise .}\end{array}$$