Robust twin support vector regression with correntropy-based metric

Abstract

Machine learning methods have been widely used control and information systems. Robust learning is an important issue in machine learning field. In this work, we propose a novel robust regression framework. Specifically, we propose a robust similarity measure induced by correntropy, and its important properties are demonstrated theoretically such as symmetry, boundedness, nonnegativity, consistency, smoothness and approximation behaviors. Moreover, the proposed robust metric extends the traditional metrics such as the $l_{2}$-norm and $l_{0}$-norm as the kernel parameter approaches different values. Then with the proposed similarity metric and $\epsilon$-insensitive pinball loss, a new robust twin support vector regression framework (RTSVR) is proposed to handle robust regression problems. The linear RTSVR model is first built, and a kernelled RTSVR version is developed to deal with nonlinear regressions. To handle the nonconvexity of the proposed RTSVR, we use DC (different of convex function) programming algorithm to iteratively solve the problem, and the resulting algorithm converges linearly. To test the proposed RTSVR, numerical experiments are implemented on two databases including a public benchmark database and a practical application database. Experiments on benchmark data with different types of noise illustrate that the proposed methods achieve better performance than the traditional methods in most cases. Experiments on the application database, the proposed RTSVR is combined with near-infrared (NIR) spectral technique to analyze the hardness of licorice seeds in low frequency,intermediate frequency and high frequency spectral regions respectively. Experiments on different spectral regions show that the performance of the RTSVR is better than that of the traditional methods in all spectral regions.

Appendix: DC programming and DCA

We outline the main algorithmic results for DC programming. The key to DC programs is to decompose an objective function into the difference of two convex functions, from which a sequence of approximations of the objective function yields a sequence of solutions converging to a stationary point, possibly an optimal solution. Generally speaking, a so-called DC program is to minimize a DC function:

$$\begin{aligned} f(x)=g(x)-h(x),x\in R^{n} \end{aligned}$$

(53)

with g(x) and h(x) being convex functions.

The DCA is an iterative algorithm based on local optimality conditions and duality [14-17]. The main scheme of DCA is: (for simplify, we omit here the dual part), at each iteration, one replaces in the primal DC problem ($P_{dc}$) the second component h by its affine minorization: $h(x^{k})+(x-x^{k})^{T}y^{k}$, to generate the convex program:

$$\begin{aligned} minimize: \{g(x)-h(x^{k})-(x-x^{k})^{T}y^{k},x\in R^{n},y^{k}\in \partial h(x^{k}) \} \end{aligned}$$

(54)

Where $\partial h$ is the subdifferential of convex function h. In practice, a simplified form of the DCA is used. Two sequences $\{x^{k}\}$ and $\{y^{k}\}$ satisfying $y^{k}\in \partial h(x^{k})$ are constructed, and $x^{k+1}$ is a solution to the convex program (54). The simplified DCA is described as follows. Initialization: Choose an initial point $x^{0}\in R^{n}$ and Let $k=0$

Repeat

Calculate $y^{k}\in \partial h(x^k)$

Solve convex program (54) to obtain $x^{k+1}$

Let k:=k+1

Until some stopping criterion is satisfied.

DCA is a descent method without line search, and it converges linearly for general DC programming.