Nonparallel plane proximal classifier

Abstract

We observed that the two costly optimization problems of twin support vector machine (TWSVM) classifier can be avoided by introducing a technique as used in proximal support vector machine (PSVM) classifier. With this modus operandi we formulate a much simpler nonparallel plane proximal classifier (NPPC) for speeding up the training of it by reducing significant computational burden over TWSVM. The formulation of NPPC for binary data classification is based on two identical mean square error (MSE) optimization problems which lead to solving two small systems of linear equations in input space. Thus it eliminates the need of any specialized software for solving the quadratic programming problems (QPPs). The formulation is also extended for nonlinear kernel classifier. Our computations show that a MATLAB implementation of NPPC can be trained with a data set of 3 million points with 10 attributes in less than 3 s. Computational results on synthetic as well as on several bench mark data sets indicate the advantages of the proposed classifier in both computational time and test accuracy. The experimental results also indicate that performances of classifiers obtained by MSE approach are sufficient in many cases than the classifiers obtained by standard SVM approach.

Introduction

Support vector machine (SVM) algorithm is an excellent tool for binary data classification [1], [2], [3], [4]. This learning strategy introduced by Vapnik and co-worker [1] is a principled and very powerful method in machine learning algorithm. Within a few years after its introduction SVM has already outperformed most other systems in a wide variety of applications. These include a wide spectrum of research areas, ranging from pattern recognition [5], text categorization [6], biomedicine [7], [8], brain–computer interface [9], [10], and financial applications [11], [12], etc.

The theory of SVM, proposed by Vapnik, is based on the idea of structural risk minimization (SRM) principle [1], [2], [3]. In its simplest form, SVM for a linearly separable two class problem finds an optimal hyper plane that maximizes the separation between the two classes. The hyper plane is obtained by solving a quadratic optimization problem. For nonlinearly separable cases the input feature vectors are first mapped into a high dimensional feature space by using a nonlinear kernel function [4], [13], [14]. A linear classifier is then implemented in that feature space to classify the data. One of the main challenges of standard SVM is that it requires large training time for huge database as it has to optimize a computationally expensive cost function. The performance of a trained SVM classifier also depends on the optimal parameter set which is usually found by cross-validation on a tuning set [15]. The large training time of SVM also prevents one to locate optimal parameter set from a very fine grid of parameters over large span. To remove this drawback, various versions of SVM have been reported by many researchers with comparable classifications ability. Introduction of proximal type of SVMs [16], [17], [18] eradicate the above shortcoming of standard SVM classifier. These classifiers avoid the costly optimization problem of SVM and as a result they are very fast. Such formulations of SVM can be interpreted as regularized least squares and considered in the much more general context of regularized networks [19], [20].

All the above classifiers discriminate a pattern by determining in which half space it lies. Mangasarian and Wild [21] first proposed a classification method by the proximity of patterns to one of the two nonparallel planes. They named it as the generalized eigenvalue proximal support vector machine (GEPSVM) classifier. Instead of finding a single hyperplane, GEPSVM finds two nonparallel hyperplanes such that each plane is clustered around one particular class data. For this GEPSVM solves two allied generalized eigenvalue problems. Although this approach is called a SVM but it is more likely to discriminate patterns by using fisher information criterion [13], [15]. Because by changing the two class margin representation by “parallel” to “nonparallel” hyperplanes it switches from a binary to potentially many class approach. The linear kernel GEPSVM is very fast as it solves two generalized eigenvalue problems of the order of input space dimension. But performance of it is only comparable with standard SVM and in many cases it gives low classification rates. Recently, Jayadeva et al. [22] proposed twin support vector machine (TWSVM) classifier. In TWSVM also two nonparallel planes are generated similar to GEPSVM but in a different technique. For this purpose it solves two smaller sized quadratic programming problems (QPPs) instead of solving large one as in the standard SVM [2], [3], [4]. Although TWSVM and GEPSVM classify data by two nonparallel planes yet the former is more likely to a typical SVM problem which does not eliminate the basic assumption of selecting a minimum number of support vectors [23]. Although TWSVM achieves good classification accuracy but it is not desirable to solve two optimization problems in many cases, predominantly for large data sets due to higher learning time. This fact motivates us to formulate the proposed classifier such that it has good classification ability as TWSVM [22] and at the same time it should computationally efficient as PSVM [18] or linear GEPSVM [21].

In this paper, we recommend binary data classifier, named as nonparallel plane proximal classifier (NPPC). NPPC also classifies binary data by the proximity of it to one of two nonparallel planes. The formulation of NPPC is totally different from that of GEPSVM [21]. But the formulations of the objective functions of NPPC are similar to that of TWSVM [22] with a different loss function and equality constraints instead of inequality constraints. We call this formulation nonparallel proximal plane classifier (NPPC) rather than a SVM classifier as there is no SRM by margin maximization between the two classes like standard SVM. Thus it can be interpreted as a classifier obtained by regularized mean square error (MSE) optimization. At last the most important fact is that the computational results on several data sets show that the performance of such classifiers obtained by MSE optimization is comparable or even better than the SVM classifiers and eliminates the need of computational costly SVM classifier in many cases.

The rest of this paper is organized as follows. A brief introduction of all the SVM classifiers is given in Section 2. In Section 3, we have formulated NPPC for linear kernel with two visual examples in two dimensions and in Section 4; we have extended the formulation for the nonlinear kernels and demonstrated its performance visually by one example. In Section 5 performances of our proposed NPPC is compared with other SVM classifiers for linear and nonlinear kernels. Finally Section 6 concludes the paper.

A concise utterance regarding the notations used in this paper [21] is as follows. All vectors are considered as column vectors if not they are transposed by using a superscript T. Inner product of two vectors x and y in n-dimensional real space ${\Re }^n$ is denoted by x^Ty and the two-norm of x is indicated by ||x||. The vector e represents column vector of ones of proper dimension whereas I stands for identity matrix of subjective dimension. In case of a matrix, containing feature vectors, $A\in {\Re }^{m\times n}$ the ith row A_i is a row vector in ${\Re }^n$. Matrices A and B contain the feature vectors of classes +1 and −1, respectively. For $A\in {\Re }^{m_1\times n}$ and $C\in {\Re }^{n\times m}$, a kernel K(A,C) maps ${\Re }^{m_1\times n}\times {\Re }^{n\times m}$ into ${\Re }^{m_1\times m}$. Only the symmetric property of the kernel is assumed [21] without any use of Mercer's positive definiteness condition [2], [3], [4], [13], [14]. The ijth element of the assumed Gaussian kernel [2] for testing nonlinear classification is given by $(K(A,C){)}_{\mathit{ij}}={\epsilon }^{-\mu ||A_i^{\mathrm{T}}-C_j|{|}^2}$, where i=1,…,m₁, j=1,…,m and μ is a positive constant, ε is the base of the natural logarithm, and A and C are as described above.