A new adaptive weighted imbalanced data classifier via improved support vector machines with high-dimension nature

Abstract

The standard support vector machine (SVM) models are widely used in various fields, but we show that they are not rationally defined from the perspective of geometric point, which is likely to degrade the models’ performances theoretically, especially under the high-dimensional cases. In this paper, we consider a composite penalty and propose an elastic net support vector machine (ENSVM). Unlike the doubly regularized support vector machine (DrSVM, Wang et al. (2006)), we impose the penalty to the slack variables rather than the normal vectors of the hyperplane. Then, we prove that ENSVM is more rationally defined than standard SVM and DrSVM (in section 3.2.1). Moreover, the ENSVM demonstrates a more stable and high-dimension nature inherently, while the simulation results cogently support these merits. Besides, we also combine fused weights with ENSVM and propose an adaptive weighted elastic net support vector machine (AWENSVM), to make the primal model more adaptive and robust to the imbalanced data. Compared with the other popular SVMs, the AWENSVM model proposed in this paper performs better obviously.

Introduction

Support vector machine (SVM) was first proposed in Cortes and Vapnik [1], which has become a powerful tool in many fields such as classification, detection, pattern recognition, gene selection and etc. For more related works, the information can be found in Chen and Wang [2], Waring and Liu [3], Zhang et al. [4] and Zhao et al. [5]. Considering a two-class classification issue, SVM attempts to find a proper hyperplane between these two classes of training points. Although SVM is succinct, efficient and more distinctly interpreted than artificial neural network (ANN), it still involves many weaknesses. One of them is that standard SVM models are limited in high-dimensional data classification problem. In fact, the SVM model works under the principle of structural risk minimization and penalized by an $l_1$ norm like LASSO (least absolute shrinkage and selection operator). LASSO was first proposed in Tibshirani [6], which shrinks the parameters by an $l_1$ penalty and it can produce exact zero coefficients. More information about model selection and estimation are listed in Fan and Li [7], Efron et al. [8], Zou and Hastie [9], Zhao and Yu [10], Pan and Xu [11] and Zhao et al. [12]. However, LASSO performs awfully in high-dimensional cases and lacks of Oracle properties (proposed in Fan and Li [7]). To promote the interpretation of LASSO in high-dimensional cases, Zou and Hastie [9] considered a mixed penalty called elastic net. They showed that the elastic net naturally has a potential excellent performance in high-dimensional cases. Meanwhile, the elastic net also encourages the grouping effects, which means it can select high correlated variables at the same time. Thus, Wang et al. [13] applied the elastic-net penalty to improve the support vector machine and proposed doubly regularized support vector machine (DrSVM) model. They also proved the grouping effects of DrSVM for correlated variables.

The other limitation is that SVM is severely influenced by the level of the imbalance between two classes training points. The imbalance has been being widely discussed for classifiers, such as Du and Chen [14], Sun et al. [15] and Zhang et al. [16]. Many studies have indicated that SVM prefers to assign the sample to the majority class. So the performance of the minority class deteriorates. A lot of researchers studied this problem and made varieties of improvements to the standard SVM, briefly reviewed in Li et al. [17], Tang et al. [18], Zou et al. [19], Akbani et al. [20], Chawla et al. [21], Huang and Du [22], Ji and Xing [23], Shao et al. [24], Xu et al. [25], Xu [26] and Xu et al. [27]. Recently, Hwang et al. [28] considered a new weighted SVM, which performs well in simulations for the imbalanced data classification. Moreover, Ji and Xing [23] combined the density and distance weights to produce a new adaptive weighted One-Class SVM which was turned out to be robust in simulation studies.

In this paper, we consider these problems in the geometric view of SVM model. We first point out the geometric irrationality of the standard SVM and propose the primal improved model, so called elastic net support vector machine (ENSVM). The most notable difference between DrSVM and ENSVM is that ENSVM uses elastic-net penalty to constrain the slack variables other than the feature variables in DrSVM. We prove that this little modification makes standard SVM be more rationally defined than DrSVM and the standard SVM. We further show the ENSVM model possesses high-dimension nature in Section 3.2.2 theoretically. Thus, the ENSVM model shows a good performance in high-dimensional cases. The simulation results powerfully support these properties. To improve the performance in imbalanced data classifying issues, we develop combinational weights and propose adaptive weighted elastic net support vector machine (AWENSVM), which performs well in imbalanced data classification problem and presents to be more robustly than the standard SVM and other existing weighted SVMs.

In the rest of the contents, this paper is organized as follows. The next section, we give a brief review to the standard SVM. Besides, we also introduce the elastic net and DrSVM together with some important properties. In Section 3, we propose the AWENSVM model and discuss its vital systematic properties in detail, including the rationality and high-dimension nature. In Section 4, we use synthetic data and UCI (The UCI Machine Learning Repository) datasets to compare the performance of AWENSVM with other famous SVMs. Finally, a conclusion is derived in Section 5.