Improvements on -Twin Support Vector Machine
Introduction
Support vector machine (SVM) has evolved as an efficient machine learning tool for binary classification problems (Cortes & Vapnik, 1995). SVM has its foundation in statistical learning theory and its formulation is based on SRM principle (Burges, 1998, Vapnik, 1999, Vapnik, 2000). The optimization task involves the minimization of a convex quadratic function subject to linear inequality constraints. SVM was initially developed to solve classification problems, but later it was extended to regression problems. Over the past few decades, various amendments to SVM have been suggested, such as Lagrangian support vector machine (LSVM) (Mangasarian & Musicant, 2001), a smooth support vector machine (SSVM) for classification (Lee & Mangasarian, 2001), least squares support vector machine (LS-SVM) (Suykens & Vandewalle, 1999), proximal support vector machine (PSVM) (Mangasarian & Wild, 2001) and generalized eigenvalue proximal SVM (GEPSVM) (Mangasarian & Wild, 2006).
GEPSVM is a nonparallel plane classifier that generates two hyperplanes instead of one, as opposed to SVM. Taking motivation from GEPSVM, Jayadeva et al. proposed TWSVM (Jayadeva et al., 2007, Khemchandani, 2008). TWSVM is a binary classifier that attempts to generate two nonparallel hyperplanes such that each plane is closer to its own class and is as far as possible from the other class. Thus, TWSVM comprises of a pair of QPPs such that, in each QPP, the objective function corresponds to a particular class and the constraints are determined by patterns of the other class. Thus, TWSVM gives rise to two smaller sized QPPs and makes it almost four times faster than standard SVM. Many extensions of TWSVM have been made, which have been discussed in the survey paper by Tian and Qi (2014). Shao, Zhang, Wang, and Deng (2011) proposed twin bounded support vector machine (TBSVM) that tries to minimize the structural risk by adding a regularization term, with the idea of maximizing the margin. Similar to TBSVM, Tian, Ju, Qi, and Shi (2014) proposed improved TWSVM (ITWSVM). Recently, Khemchandani, Goyal, and Chandra (2016) proposed TWSVR for regression problems using TWSVM framework.
Schölkopf et al. proposed new support vector machine (-SVM) for classification and regression (Scholkopf et al., 1999, Schölkopf et al., 2000), which is a modification of SVM. -SVM introduced a priori chosen parameter that determines an upper bound on the training error and a lower bound on the number of support vectors. Recently, Peng extended the concept of -SVM to TWSVM and proposed -TWSVM (Peng, 2010). In TWSVM, the patterns of one class are at least a unit distance away from the hyperplane of other class; this might increase the number of SVs which may lead to poor generalization ability. The parameter in -TWSVM controls the bounds on the number of SVs, similar to -SVM, and further the unit distance of TWSVM is modified to variable , which is optimized in the primal problem involved therein. Further, -TWSVM can be interpreted as a pair of minimum generalized Mahalanobis-norm problems on two reduced convex hulls (RCHs).
In this paper, we propose -TWSVM that generates two nonparallel hyperplanes and its key features are listed below:
- •
-TWSVM solves a smaller-sized QPP and a UMP, instead of a pair of QPPs as solved by -TWSVM and TWSVM-based classifiers. Therefore, the two implementations of -TWSVM have efficient training time as compared to -TWSVM.
- •
For linear case, the hyperplane for one of the twin problems of -TWSVM is obtained by solving a UMP in the feature dimension, while -TWSVM solves a QPP with constraints defined by number of data points in other class. Hence, -TWSVM solves a simpler optimization problem.
- •
Unlike -TWSVM and TWSVM, the formulation of -TWSVM is based on the principle of SRM and hence it has got good generalization ability, with an added advantage that -TWSVM is much faster than both of these classifiers.
- •
-TWSVM uses a single parameter to control the bounds on the training error and number of support vectors, whereas -TWSVM uses two such parameters— and .
- •
In -TWSVM (Fast), we have modified the first problem of -TWSVM as minimization of a unimodal function for which line search methods can be used; this further avoids solving the QPP. The other problem is formulated as a UMP, similar to -TWSVM. Hence, -TWSVM (Fast) is a faster version of our proposed work. It is experimentally proved to be more time-efficient than -TWSVM and -TWSVM.
The paper is organized as follows: Section 2 gives a brief description of TWSVM and -TWSVM and explains the notations used in the rest of the paper. Section 3 introduces “Improvements on -Twin Support Vector Machine” and is followed by experimental results on benchmark datasets in Section 4. The performance of -TWSVM for pixel classification is also investigated in this section. Finally, the paper is concluded in Section 5.
Section snippets
Twin support vector machines (TWSVM)
TWSVM (Jayadeva et al., 2007, Khemchandani, 2008) is a binary classifier that determines two nonparallel hyperplanes by solving two related SVM-type problems, each of which is smaller than the problem in a conventional SVM. The nonparallel hyperplanes of TWSVM are given by
The formulation of pair of QPPs in TWSVM is similar to that of a typical SVM, but all patterns do not appear in the constraints of either problem at the same time. Let the data points belonging to
Improvements on -twin support vector machine
In this section, we propose two novel classifiers “Improvements on -Twin Support Vector Machine, namely -TWSVM and -TWSVM (Fast)”, developed on the lines of TWSVM (Jayadeva et al., 2007) and further based on -TWSVM (Peng, 2010). (From this point onwards, we will refer to first implementation as -TWSVM and second as -TWSVM (Fast).) Unlike TWSVM, -TWSVM solves a smaller sized QPP and a UMP as compared to solving a related pair of QPPs for obtaining two nonparallel hyperplanes. The
Numerical experiments
To evaluate the performance of the proposed work, we compare -TWSVM and -TWSVM (Fast) with TBSVM (Shao et al., 2011) and -TWSVM (Peng, 2010). The performance is measured in terms of classification accuracy and computational efficiency of these algorithms. The experiments are performed in MATLAB version 8.0 under Microsoft Windows environment on a machine with 3.40 GHz CPU and 16 GB RAM.
Conclusions
In this paper, we have proposed two novel classifiers as “Improvements on -Twin Support Vector Machine: -TWSVM and -TWSVM (Fast)”, which improve the learning time of Twin support vector machine (TWSVM) based classifiers, specifically -TWSVM. In -TWSVM, we solve a smaller sized quadratic programming problem (QPP) and an unconstrained optimization problem (UMP), whereas TWSVM based classifiers solve a pair of QPPs. Hence, -TWSVM is computationally faster than TBSVM and -TWSVM and has
Acknowledgments
The authors would like to thank the editor and anonymous reviewers whose valuable comments and feedback have helped us to improve the content and presentation of the paper.
References (37)
Data clustering: 50 years beyond k-means
Pattern Recognition Letters
(2010)- et al.
A customized gabor filter for unsupervised color image segmentation
Image and Vision Computing
(2009) - et al.
TWSVR: Regression via twin support vector machine
Neural Networks
(2016) A -twin support vector machine (-tsvm) classifier and its geometric algorithms
Information Sciences
(2010)- et al.
Laplacian twin support vector machine for semi-supervised classification
Neural Networks
(2012) - Arbelaez, P., Fowlkes, C., & Martin, D. (2007). The berkeley segmentation dataset and benchmark. See...
- Blake, C., & Merz, C. J. (1998). {UCI} repository of machine learning databases. URL:...
A tutorial on support vector machines for pattern recognition
Data Mining and Knowledge Discovery
(1998)- et al.
Numerical optimization with applications
(2009) - et al.
Support-vector networks
Machine Learning
(1995)
Statistical comparisons of classifiers over multiple data sets
The Journal of Machine Learning Research
Pattern classification
Support vector machines for classification and regression
A survey of outlier detection methodologies
Artificial Intelligence Review
A simple sequentially rejective multiple test procedure
Scandinavian Journal of Statistics
Twin support vector machines for pattern classification
IEEE Transactions on Pattern Analysis and Machine Intelligence
Lognormal distributions
Mathematical programming applications in machine learning
Cited by (33)
A new algorithm for support vector regression with automatic selection of hyperparameters
2023, Pattern RecognitionAdaptive robust learning framework for twin support vector machine classification
2021, Knowledge-Based SystemsMulti-category ternion support vector machine
2019, Engineering Applications of Artificial IntelligenceFast Laplacian twin support vector machine with active learning for pattern classification
2019, Applied Soft Computing JournalCitation Excerpt :In order to show the effectiveness of proposed algorithm on non-mean centered data and with varying sizes, the NDC [33] datasets are skewed by making use of log-normal distribution. The exponential noise has thus been added to the usual NDC datasets and the resulting dataset has been termed as exp-NDC [26]. The feature dimension of exp-NDC data is 32.
Twin support vector machines: A survey
2018, NeurocomputingCitation Excerpt :The theoretical analysis showed that this method can be interpreted as a pair of minimum generalized Mahalanobis-norm problems on two reduced convex hulls. In order to further improve the performance of v-TWSVM, Khemchandani et al. [102] proposed two novel binary classifiers termed as “Improvements on v-twin support vector machine: Iv-TWSVM and Iv-TWSVM (Fast)” that were motivated by v-TWSVM. The significant advantage of Iv-TWSVM over v-TWSVM was that Iv-TWSVM solved one smaller-sized Quadratic Programming Problem (QPP) and one Unconstrained Minimization Problem (UMP).
Support spinor machine
2017, Digital Signal Processing: A Review Journal