Robust Twin Support Vector Regression with Smooth Truncated Hε Loss Function

Abstract

Twin support vector regression (TSVR) is an important algorithm to handle regression problems developed on the basis of support vector regression (SVR). However, TSVR adopts ε-insensitive loss function which is not well capable to deal with noise or outliers. In order to reduce the influence of noise or outliers on the performance of TSVR, in this paper, we propose a novel robust twin support vector regression with smooth truncated Hε loss function, termed as THε-TSVR. At first, we construct smooth truncated Hε loss function by combining ε-insensitive loss function and Huber loss function. Then, a concave-convex programming (CCCP) is employed to solve the nonconvex optimization problem in the primal space. In addition, the convergence of THε-TSVR is also proved. The experimental results on some artificial datasets and UCI datasets with noise and without noise verified the effectiveness and robustness of the proposed THε-TSVR.

Appendices

Appendix 1 Symbols Description

Symbols	Implication
$A$	All training samples, $n \times m$ matrix
$A_{i} \in R^{m}$	The $i$-th training sample
$Y = (y_{1} ,\,y_{2} , \ldots ,y_{n} )$	The response vector of training samples
$\alpha ,\gamma$	Lagrange multipliers
$c_{1} ,c_{2} > 0$	Penalty parameters
$\varepsilon_{1} ,\varepsilon_{2} \ge 0$	Insensitive parameters
$\xi ,\eta \in R^{n}$	Slack variables
$I$	Identity matrix
$u$	Arbitrary variable
$L_{\varepsilon } \left( u \right)$	ε-insensitive loss function
$L_{h} \left( u \right)$	Huber loss function
$L_{h\varepsilon }^{v} \left( u \right)$	Smooth truncated Hε loss function
$u_{1} = [\omega_{1}^{T} \, b_{1} ]^{T}$, $u_{2} = [\omega_{2}^{T} \, b_{2} ]^{T}$	Optimal solutions
$s^{i} (u_{1} ) \in R^{n}$, $s^{i} (u_{2} ) \in R^{n}$	Sign vectors
$s_{j}^{i} (u_{1} ) \in R^{n} \, (j = 1,2, \cdots ,n)$	The $j$-th element of $s^{i} (u_{1} ) \in R^{n} \, (i = 1,2,3,4)$
$s_{j}^{i} (u_{2} ) \in R^{n} \, (j = 1,2, \cdots ,n)$	Thej-th element of $s^{i} (u_{2} ) \in R^{n} \;(i = 5,6,7,8)$

Appendix 2 Abbreviations Description

Abbreviations	Full name
SVR	Support vector regression
TSVR	Twin support vector regression
ε-TSVR	ε-Twin support vector regression
TWSVM	Twin support vector machine
THε-TSVR	Robust twin support vector regression with smooth truncated Hε loss function
HN-TSVR	Twin support vector regression with Huber loss
RHN-TSVR	Regularization based TSVR with Huber loss
CCCP	Concave-convex programming
SOR	Successive overrelaxation technique
SRM	Structural risk minimization