Preprocessing unbalanced data using support vector machine

Abstract

This paper deals with the application of support vector machine (SVM) to deal with the class imbalance problem. The objective of this paper is to examine the feasibility and efficiency of SVM as a preprocessor. Our study analyzes different classification algorithms that are employed to predict the customers with caravan car policy based on his/her sociodemographic data and history of product ownership. A series of experiments was conducted to test various computational intelligence techniques viz., Multilayer Perceptron (MLP), Logistic Regression (LR), and Random Forest (RF). Various standard balancing techniques such as under-sampling, over-sampling and Synthetic Minority Over-sampling TEchnique (SMOTE) are also employed. Subsequently, a strategy of data balancing for handling imbalanced distribution in data is proposed. The proposed approach first employs SVM as a preprocessor and the actual target values of training data are then replaced by the predictions of trained SVM. Later, this modified training data is used to train techniques such as MLP, LR, and RF. Based on the measure of sensitivity, it is observed that the proposed approach not only balances the data effectively but also provides more number of instances for minority class, which in turn enhances the performance of the intelligence techniques.

Introduction

The class imbalance problem has been recognized in many real world applications [26] and is an evolving topic of machine learning research. It is observed from the literature that machine learning techniques tend to produce suboptimal classification models. The class imbalance problem where little or very less instances are available for the most important class of the study exists in many real world application domains, such as telecommunications [23], detection of oil spoils in satellite radar images [32], text classification [42], medical diagnosis [29], intrusion detection [34] and fraud detection [41].

Researchers have been attempting to deal with classification using unbalanced datasets. Methods to deal with imbalanced problems include, resizing training set that includes, oversampling minority class samples [35] and downsizing majority class samples [31], adjusting misclassification costs [11] and recognition based learning [32]. Detailed review reports [19], [30], [39], [51] have discussed the key issues related to problem solving with unbalanced training data using machine learning techniques. Research studies show that many standard machine learning approaches result in poor performance, specifically when dealing with medium and large scale unbalanced datasets [17], [26], [32], [49], [50]. One of the key problems when learning with imbalanced data sets is the lack of data where the number of samples is small or no sample is available for a particular class [50]. If there is a lack of data, the estimated decision boundary can be very far from the true boundary. Japkowicz and Stephen [26] reported that for simple data sets that were linearly separable, classifier performances were not susceptible to any amount of imbalance. Indeed, as the degree of data complexity increased, the class imbalance factor started affecting the generalization ability of the classifiers. Most accuracy-driven algorithms are biased toward the prevalent class. The machine learning approaches improved overall accuracy by assigning the overlapped area to the majority class, and ignored or treated the minority class as noise [49].

Since late 1960s, researchers have put their efforts toward developing strategies to deal with the class imbalance problem. In the earliest stage of this research, researchers used the condensed nearest neighbor method of under-sampling [22]. Wilson [52] proposed an Edited Nearest Neighbor (ENN) method of under-sampling. In this method, noisy samples from the majority class are removed in order to under-sample the data. Later, Kubat and Matwin [31] developed a concept of selective under-sampling by keeping the minority samples untouched. They introduced a data cleaning procedure using the Tomek–Links concept for under-sampling and removed the borderline majority samples. Based on Wilson's ENN method, the Neighbourhood Cleaning Rule is proposed to eliminate or to discard the majority class samples [33]. Later, Chawla et al. [9] proposed SMOTE (Synthetic Minority Over-sampling TEchnique), where synthetic (artificial) samples are generated rather than over-sampling by replacement. Maloof [37] reported that sampling has the same result as moving the decision threshold or adjusting the cost matrix. Barendela et al. [2] proposed a weighted distance function to be used in the classification phase of k-NN to compensate for the imbalance in the training samples without actually altering the distribution of classes. Efficiency of SVM is then analyzed to deal with the class imbalance problem [53]. They proposed SVM with a changed kernel function, which pushed the hyperplane closer to the positive class. Estabrooks et al. [13] concluded that combining different expressions of resampling approach was an effective solution. On the contrary, some researchers also reported that there was no further improvement to the predictive performance of SVM for text classification when it was preceded by strategies such as resampling in the presence of imbalanced training data [44].

Researchers have emphasized the use of clustering based preprocessing methods as an alternative for sampling of the data. Batista et al. [3], [4] proposed two hybrid sampling techniques, SMOTE + Tomek–Links and SMOTE + ENN for overlapping datasets, for better defined class clusters among majority and minority classes. Jo and Japkowicz [27] presented a cluster based over-sampling approach. Majority and minority class samples are clustered first and the clusters in the majority class are over-sampled to the largest cluster obtained for the majority class data. Han et al. [21] proposed borderline SMOTE, which identified minority samples at borderline and applied SMOTE. This is the only technique proposed to over-sample the borderline minority samples. Later, k-means based under-sampling method and the Agglomerative Hierarchical Clustering based over-sampling method to deal with unbalanced datasets are proposed [10]. Guo and Viktor [18] proposed boosting method with various over-sampling techniques to deal with hard to classify examples and concluded that boosting approach improved the prediction accuracy of the classifier. Huang et al. [25] presented Biased Minimax Probability Machine to resolve the imbalance problem.

Researchers then exerted their efforts toward developing hybrid approaches to deal with unbalanced data, where they combined over-sampling and under-sampling with different concepts into one approach. Some used a combination of under-sampling and over-sampling [35]. They used lift analysis instead of classification accuracy to measure a classifiers performance. Various hybrids, SMOTE-bootstrap hybrid [36] and a hybrid combining machine learning and unsupervised McCab feature selection method using SVM and maximum entropy method [12], and a hybrid balancing model using unsupervised clustering and decision tree boosting [6] are proposed. Later, Farquad et al. [14], [15] proposed a hybrid rule extraction from SVM approach for handling the class imbalance problem. They concluded that rules extracted using their proposed approach performed very well. Table 1 provides a chronological overview of the balancing approaches proposed by various researchers.

Researchers have never reported any preprocessing using intelligent methods to balance the data. In this paper we employ SVM as a preprocessor. SVM is one of the best intelligent algorithms used for classification and regression purposes. The best property of SVM is that it always yields global optimal solution, whereas other intelligent algorithms suffer from getting stuck with a local minimum. SVM tries to find the decision boundary between various classes without actually worrying about the number of instances available for a class. SVM is suitable for high dimensional problems and works with a small number of observations as well. Hence, trained SVM is proposed as a preprocessor in this paper.

The rest of the paper is organized as follows. Section 2 presents a brief overview of the method of SVM and motivation for the proposed approach. Section 3 explains the architecture of the proposed balancing approach. Section 4 presents a description of the dataset and the experimental method used in this research. Results and discussions are presented in Section 5. Section 6 concludes the paper. A brief overview of MLP, LR and RF is provided in Appendix A.