On robust asymmetric Lagrangian $\nu$-twin support vector regression using pinball loss function

Highlights

• Proposed a novel approach as a pair of the unconstrained minimization problem.

• Applied generalized derivative and smoothing approaches to find the Hessian matrix.

• By using gradient based iterative schemes determine solution of the problem.

• Improved the stability in the dual formulations and makes the model well-posed.

• Performance results of URALTSVR are carried out on artificial & realworld datasets.

Abstract

The main objective of twin support vector regression (TSVR) is to find the optimum regression function based on the $\epsilon$-insensitive up- and down-bound with equal influences on the regression function where all the data points have a different location above the up-bound points and below the down-bound points. However, the effects of all data points must be distinct based on their distribution in the regression function. Recently, asymmetric $\nu$-twin support vector regression (Asy-$\nu$-TSVR) is encouraged on the same subject but still, the present matrices in the mathematical formulation have faced the problem of semi-definite. In order to handle this problem effectively, a new regressor model named as robust asymmetric Lagrangian $\nu$-twin support vector regression using pinball loss function (URALTSVR) proposes as a pair of the unconstrained minimization problem to handle not only the noise sensitivity and instability of re-sampling but also consist positive definite matrices. Here, we suggest the proposed model URALTSVR in such a way where the pinball loss function is playing a vital role to control the fitting error inside the asymmetric tube. One of the advantages is that unlike TSVR and Asy-$\nu$-TSVR, it considers the concept of structural risk minimization principle through the inclusion of regularization term as well as change the one-norm of the vector of the slack variable by two-norm, which yields the dual problem to be strongly convex, stable and well-posed. Aforementioned, the proposed formulation has a continuous and piecewise quadratic problem that is solved by their gradients based iterative approaches. Specifically, we analyze the three implementations of URALTSVR with the baselines approaches support vector regression (SVR), TSVR and Asy-$\nu$-TSVR, which discard the dependencies to solve a pair of quadratic programming problem (QPP) for obtaining the unique global solution. Overall, SRALTSVR1 based on smooth approximation function performs outstanding for artificial and real-world datasets.

Introduction

In the real world, we are facing many problems that can be solved with the concept using different machine learning approaches in consideration of the classification, regression, detection, tracking, and recognition. Support vector machine (SVM) is a powerful approach proposed by Vapnik [1] among all computational algorithms for classification [2], [3] and regression problems [4], [5], [6]. In classification, this algorithm tries to implement structural risk minimization (SRM) effectively and find the suitable hyperplane which classifies the unknown and unseen real data samples. Based on Vapnik’s concept [7], SVR does not rely on the dimensionality of the input space. Different variants of SVR [8] such as $\epsilon$-SVR [9], least-square approach of SVR named as least square support vector regression (LS-SVR) [10], SMO [11], $\nu$-SVR [12], active set SVR (ASVR) [13], smooth SVR (SSVR) [14] already suggested to overcome the demerit of SVM i.e. high computational complexity. Some of the researchers have also introduced new SVR models with heuristic training and geometric method [15], functional iterative method [15] and different loss functions like soft insensitive loss or unified loss [16], [17], Laplacian loss [17], Huber loss [5], [18] and many more [4], [6]. SVR is widely accepted by a range of research areas such as price prediction of electronic commerce products by using sentiment analysis [19], real estate as house pricing prediction [20], more occurred volatility prediction for traditional and cryptocurrencies [21], [22], [23], forecasting solar and wind energy resources [24], [25], [26], prediction of energy utilization of different data centers based on weather constraints [27], financial time series data [28], [29], [30] etc. However, several improvements are needed to boost the generalization capability and computation cost of the regression model.

Aforementioned the concept of parallel hyperplane classifier, Jayadeva et al. [31] have developed a flexible and convenient approach for binary classification called twin SVM (TWSVM) [31] by following the idea of GEPSVM [32]. As one can see that TWSVM model require two non-parallel supporting hyperplanes which are as much as possible nearer to one class datapoints and far from second class datapoints by solving a pair of quadratic programming problem (QPP), respectively. The twin model of SVM is far better than SVM based on several experimental results [31]. Inspired from the concept of TWSVM classifier, Peng [33] has proposed a popular algorithm for the regression problems, i.e. twin support vector regression (TSVR) where the optimal regressor is obtained for the training data points through calculating the $\epsilon$-insensitive up- and down-bound functions. TSVR is also seeking two QPPs with smaller size like TWSVM for getting the non-parallel bound functions. It receives good performance capability with fast training speed in comparison to standard SVR. One of the disadvantages of TSVR is that it considers only the gist of empirical risk minimization (ERM). A series of new enhanced versions of TSVR has been introduced with the innovative ideas that perform better in the case of outliers and noise to enhance the ability of generalization performance and computation time such as reduced TSVR [34] which apply the ideas of the rectangular kernel to find the notable changes in computation cost; Smooth twin support vector regression (STSVR) [35] has been reformulated as a strongly convex model that gives a generalized effective and efficient solution for each subproblem; A weighted TSVR (WTSVR) [36] has suggested for data samples that get the different penalties based on the locations to reduce the effect of over-fitting etc. In a similar context, twin parametric insensitive SVR (TPISVR) [37] is proposed to handle the heteroscedastic noise if present in the dataset. Shao et al. [38] have suggested a novel approach termed as $\epsilon$-insensitive TSVR ($\epsilon$-TSVR) which use the SRM principle for getting better generalization performance as well as improve the learning speed through the successive over relaxation (SOR). Zhao et al. [39] have given a least square approach of TSVR known as twin least square support vector regression (TLSSVR) to enhance the computational cost for large scale data sets but it loses the sparseness. Further, Lagrangian TSVR (LTSVR) has proposed as a strongly convex minimization problem with non-negativity constraints and solved an iterative approach to seeking the solution but LTSVR has focused only on ERM principle [40]. K-nearest neighbor based WTSVR (KNN-WTSVR) [41] has suggested which utilizes complete knowledge of data samples and assign the weight to the training sample based on the K-nearest neighbors.

To provide robustness and sparseness to the algorithm, a robust and sparse TSVR [42] has been introduced by adding regularization term and solve it by using linear programming (LP). Further, unconstrained Lagrangian TSVR [43] has proposed which solve as unconstrained minimization problems. Weighted Lagrangian $\epsilon$-TSVR (WL-$\epsilon$-TSVR) with quadratic loss functions [44] has reduced the effect of outliers by using the weight matrix with different penalties for the samples placed in different locations. Recently, the approach Twin SVR (TWSVR) via TSVM [45] uses the true spirit of TSVM where up- and down-bound regressor functions are solved and provide better accuracy and efficacy in comparison to TSVR. Pairing support vector algorithm [46] has focused on evaluating an insensitive zone of flexible structure to rigidly fit the training data sample that gives better results in respect of sparsity and generalization capability. For another variant of TSVR, see [47], [48], [49].

One can notice that in TSVR, same penalties are assigned to every point upper side of the up-bound and lower side of the down-bound but the effect on regression function cannot be identical for each sample. To handle this issue in the SVR, Huang has suggested the pinball loss which is the extension of $\epsilon$-insensitive loss to give different penalties [48], [49], [50] for each sample based on the distribution of the samples and then yields better generalization performance. Similarly for TSVR, a novel approach as asymmetric $\nu$-twin support vector regression (Asy-$\nu$-TSVR) is given [51] where an asymmetric tube is used with pinball loss function to improve the generalization performance but takes more computational time to solve the QPPs. According to the literature survey, very limited research is focused on SVR and TSVR with Pinball loss function for regression problems. Motivated from the literature [40], [43], [51], we suggest a model named asymmetric Lagrangian $\nu$-twin support vector regression with pinball loss (URALTSVR) as a pair of the unconstrained minimization problem where the pinball loss function is used to control the fitting error inside the asymmetric tube to handle not only the noise sensitivity and instability of re-sampling but also consist of positive definite matrices. Further, we find the solution of our proposed URALTSVR by solving the two unconstrained minimization problems which are having strongly convex, piece-wise quadratic and differentiable functions in dual space. Specifically, we analyze the three different implementations of URALTSVR by using the gradient-based iterative approaches which discard the dependencies to solve the QPPs for obtaining the unique global solution. The main attractive advantages of proposed URALTSVR are as follows:

• Rather than following the one-norm of slack variables, two-norm of slack variables is considered in our proposed URALTSVR approach which makes the model strongly convex hence, obtain the unique global solution.

• Addition of the regularization term in the objective functions improves the stability in the dual formulations and builds the model well-posed.

• The solution of our methods is easily determined by using gradient-based iterative schemes, so no essentiality of any additional toolbox to solve the QPPs unlike SVR, TSVR and Asy-$\nu$-TSVR.

• Dissimilar from the TSVR and Asy-$\nu$-TSVR, URALTSVR follows the SRM principle.

• The numerical experimental analysis on various synthetic and benchmark real-world datasets are also indicated the importance and superiority of the URALTSVR in contrast to SVR, TSVR and Asy-$\nu$-TSVR.

In Table 1, the important merit and demerit of reported approaches SVR, TSVR and Asy-$\nu$-TSVR with proposed URALTSVR are mentioned.

The work is written in this paper in such a way: Section 2 presented the mathematical formulation of SVR, TSVR and Asy-$\nu$-TSVR in details under related work. Our proposed approach URALTSVR is given in Section 3. Further, Comparative performance of proposed URALTSVR with others compared methods like SVR, TSVR and Asy-$\nu$-TSVR has been conducted on artificially generated datasets and real-world datasets in Section 4. Section 5 dwells on the conclusion of research work.