Relaxed support vector regression

Abstract

Datasets with outliers pose a serious challenge in regression analysis. In this paper, a new regression method called relaxed support vector regression (RSVR) is proposed for such datasets. RSVR is based on the concept of constraint relaxation which leads to increased robustness in datasets with outliers. RSVR is formulated using both linear and quadratic loss functions. Numerical experiments on benchmark datasets and computational comparisons with other popular regression methods depict the behavior of our proposed method. RSVR achieves better overall performance than support vector regression (SVR) in measures such as RMSE and $R^2_{adj}$ while being on par with other state-of-the-art regression methods such as robust regression (RR). Additionally, RSVR provides robustness for higher dimensional datasets which is a limitation of RR, the robust equivalent of ordinary least squares regression. Moreover, RSVR can be used on datasets that contain varying levels of noise.

Appendices

Quadratic loss function formulation

1.1 Dual derivation

The Lagrangian function for Formulation (5) can be written as,

$$\begin{aligned}&\mathcal {L}(\mathbf {w}, \mathbf {\xi },\overline{\mathbf {\xi }}, b, \epsilon ,\alpha ^+,\alpha ^-, \beta , \lambda ,\overline{\lambda }, v,\overline{v}) \nonumber \\&\quad = \frac{1}{2} \langle \mathbf {w} , \mathbf {w} \rangle + \frac{C}{2} \sum _{i=1}^n \left( {{\xi _i}^2 +{\overline{\mathbf {\xi }}_i }^2} \right) - \sum _{i=1}^n {\alpha _i}^+ \left( \epsilon + \xi _i + v_i +\langle \mathbf {w} , \varPhi (\mathbf {x}_i) \rangle + b - y_{i} \right)&\nonumber \\&\qquad - \sum _{i=1}^n {\alpha _i}^- \left( \epsilon + \overline{\xi }_i +\overline{v_i } - \langle \mathbf {w} , \varPhi (\mathbf {x}_i) \rangle - b + y_{i} \right) - \beta \left( n \varUpsilon - \sum _{i=1}^n \left( v_i + \overline{v}_i \right) \right)&\nonumber \\&\qquad - \sum _{i=1}^n\lambda _i v_i - \sum _{i=1}^n \overline{\lambda }_i \overline{v}_i&\end{aligned}$$

(10a)

where $\mathbf {\alpha ^+}, \mathbf {\alpha ^-}, \beta , \mathbf {\lambda }$ and $\overline{\lambda }$ are the Lagrange multipliers. Since (5) is a convex problem, its Wolfe dual can be obtained from the following stationary first order conditions of the primal variables $\mathbf {w}, b, \mathbf {\xi }, \overline{\mathbf {\xi }}, v$ and $\overline{v}$.

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial \mathbf {w} }&= \mathbf {w} - \sum _{i=1}^n \left( {\alpha _i}^+ -{\alpha _i}^- \right) \varPhi (\mathbf {x}_i) = 0&\end{aligned}$$

(11a)

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial b}&= \sum _{i=1}^n \left( {\alpha _i}^- - {\alpha _i}^+ \right) = 0&\end{aligned}$$

(11b)

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial \xi _k}&= C {\xi }_k - {\alpha _k}^+ = 0,&\forall k=1,2, \ldots n&\end{aligned}$$

(11c)

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial \overline{\xi }_k}&= C \overline{{\xi }}_k - {\alpha _k}^- = 0,&\forall k=1,2, \ldots n&\end{aligned}$$

(11d)

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial v_k}&= - {\alpha _k}^+ + \beta - \lambda _k = 0,&\forall k=1,2, \ldots n \end{aligned}$$

(11e)

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial \overline{v}_k}&= - {\alpha _k}^- + \beta - \overline{\mathbf {\lambda }}_k = 0,&\forall k=1,2, \ldots n \end{aligned}$$

(11f)

Substituting the equivalent expressions for $\mathbf {w}$, b, $\mathbf {\xi }$, $\overline{\mathbf {\xi }}$, v and $\overline{v}$ from Eqs. (11a)–(11f) back in expression (10), the Wolfe dual can be written as shown in (6).

1.2 Optimal hyperplane parameters

The solution to (6) is used to evaluate,

$$\begin{aligned} {\mathbf {w}}^* = \sum _{j=1}^n \left( {\alpha _j}^+ -{\alpha _j}^- \right) \varPhi (\mathbf {x}_j) \end{aligned}$$

(12a)

Let ${S^+} = \Big \{ {\alpha _i}^+ | \, (0< {\alpha _i}^+ < \beta ) \Big \}$, $I^+ = \Big \{ i | \, {\alpha _i}^+ \in S^+ \Big \}$.

Let ${S^-} = \Big \{ {\alpha _i}^- | \, (0< {\alpha _i}^- < \beta ) \Big \}$, $I^- = \Big \{ i | \, {\alpha _i}^- \in S^- \Big \}$.

The bias can be computed as,

$$\begin{aligned} {b}^*&= \frac{1}{|S^+|} \sum _{i \in I^+} \left( - \epsilon -\frac{{\alpha _i}^+ }{C} - \sum _{j=1}^n \left( {\alpha _j}^+ -{\alpha _j}^- \right) K(\mathbf {x}_i,\mathbf {x}_j) + y_{i} \right) \nonumber \\ {}&\quad + \frac{1}{|S^-|} \sum _{i \in I^-}\left( \epsilon + \frac{{\alpha _i}^- }{C} -\sum _{j=1}^n \left( {\alpha _j}^+ -{\alpha _j}^- \right) K(\mathbf {x}_i,\mathbf {x}_j) + y_{i} \right) \end{aligned}$$

(13a)

Lemma 1 derivation

The proof follows from the Karush–Kuhn–Tucker complementary slackness condition $\beta \left( n \varUpsilon - \sum _{i=1}^n \left( v_i + \overline{v}_i \right) \right) = 0$. Using the fact that $||w||>0$ and constraints (6c) and (6d), Eq. (11a) implies that $\beta >0$. Thus, constraint (5d) is binding, i.e., the total free slack amount $n \varUpsilon$ is always consumed completely.

Linear loss function formulation

1.1 Dual derivation

The Lagrangian function for Formulation (8) can be written as,

$$\begin{aligned}&\mathcal {L}(\mathbf {w}, \mathbf {\xi },\overline{\mathbf {\xi }}, b, \epsilon ,\alpha ^+,\alpha ^-, \beta , {\mathbf {\gamma }},{\mathbf {\overline{\gamma }}},{\mathbf {\delta }},{\mathbf {\overline{\delta }}}, \lambda ,\overline{\lambda }, v,\overline{v}, s,\overline{s})&\nonumber \\&\quad =\frac{1}{2} \langle \mathbf {w} , \mathbf {w} \rangle + C \left( \sum _{i=1}^n \left( \xi _i +\overline{\mathbf {\xi }}_i \right) + s + \overline{s} \right) - \sum _{i=1}^n {\alpha _i}^+ \left( \epsilon + \xi _i + v_i +\langle \mathbf {w} , \varPhi (\mathbf {x}_i) \rangle + b - y_{i} \right)&\nonumber \\&\qquad - \sum _{i=1}^n {\alpha _i}^- \left( \epsilon + \overline{\xi }_i +\overline{v}_i - \langle \mathbf {w} , \varPhi (\mathbf {x}_i) \rangle - b + y_{i} \right) - \beta \left( n \varUpsilon - \sum _{i=1}^n \left( v_i +\overline{v}_i \right) \right) - \sum _{i=1}^n\lambda _i v_i&\nonumber \\&\qquad - \sum _{i=1}^n \overline{\lambda }_i \overline{v}_i - \sum _{i=1}^n {\gamma _i} \xi _i - \sum _{i=1}^n \overline{\mathbf {\gamma }}_i \overline{\mathbf {\xi }}_i - \sum _{i=1}^n {\delta _i} \left( s - \xi _i \right) - \sum _{i=1}^n \overline{\mathbf {\delta }}_i \left( \overline{s} - \overline{\mathbf {\xi }}_i \right) \end{aligned}$$

(14a)

where $\mathbf {\alpha ^+}, \mathbf {\alpha ^-}, \beta , \gamma , \overline{\gamma }, \mathbf {\delta }, \mathbf {\overline{\delta }}, \mathbf {\lambda }$ and $\overline{\lambda }$ are the Lagrange multipliers. Since (8) is a convex problem, its Wolfe dual can be obtained from the following stationary first order conditions of the primal variables $\mathbf {w}, b, \mathbf {\xi }, \overline{\mathbf {\xi }}, s, \overline{s}, v$ and $\overline{v}$ .

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial \mathbf {w} }&= \mathbf {w} - \sum _{i=1}^n \left( {\alpha _i}^+ -{\alpha _i}^- \right) \varPhi (\mathbf {x}_i)= 0&\end{aligned}$$

(15a)

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial b}&= \sum _{i=1}^n \left( {\alpha _i}^- - {\alpha _i}^+ \right) = 0&\end{aligned}$$

(15b)

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial \xi _k}&= C - {\alpha _k}^+ - {\gamma _k} + {\delta _k} = 0,&\forall k=1,2, \ldots n&\end{aligned}$$

(15c)

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial \overline{\xi }_k}&= C - {\alpha _k}^- - \overline{\gamma }_k + \overline{\delta }_k = 0,&\forall k=1,2, \ldots n&\end{aligned}$$

(15d)

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial v_k}&= - {\alpha _k}^+ + \beta - \lambda _k = 0,&\forall k=1,2, \ldots n \end{aligned}$$

(15e)

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial \overline{v}_k}&= - {\alpha _k}^- + \beta - \overline{\mathbf {\lambda }}_k = 0,&\forall k=1,2, \ldots n \end{aligned}$$

(15f)

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial s}&= C - \sum _{i=1}^n {\delta }_i = 0 \end{aligned}$$

(15g)

$$\begin{aligned} \frac{\partial \mathcal {L}}{\partial \overline{s}}&= C - \sum _{i=1}^n \overline{{\delta }}_i = 0 \end{aligned}$$

(15h)

Substituting the equivalent expressions for $\mathbf {w}, b,\mathbf {\xi },\overline{\mathbf {\xi }},v, \overline{v}$, s and $\overline{s}$ from Eqs. (15a)–(15h) back in expression (14), the Wolfe dual can be written as shown in (9).

1.2 Optimal hyperplane parameters

The solution to (9) is used to evaluate,

$$\begin{aligned} {\mathbf {w}}^* = \sum _{j=1}^n \left( {\alpha _j}^+ -{\alpha _j}^- \right) \varPhi (\mathbf {x}_j) \end{aligned}$$

(16a)

Let ${S^+} = \Big \{ {\alpha _i}^+ | \, (0< {\alpha _i}^+ < C) \Big \}$, ${I}^+ = \Big \{ i | \, {\alpha _i}^+ \in S^+ \Big \}$.

Let ${S^-} = \Big \{ {\alpha _i}^- | \, (0< {\alpha _i}^- < C) \Big \}$, ${I}^- = \Big \{ i | \, {\alpha _i}^- \in S^- \Big \}$.

The bias can be computed as,

$$\begin{aligned} {b}^*&= \frac{1}{|S^+|} \sum _{i \in {I}^+} \left( - \epsilon - \sum _{j=1}^n \left( {\alpha _j}^+ -{\alpha _j}^- \right) K(\mathbf {x}_i,\mathbf {x}_j) + y_{i} \right) \nonumber \\ {}&\quad + \frac{1}{|S^-|} \sum _{i \in {I}^-}\left( \epsilon -\sum _{j=1}^n \left( {\alpha _j}^+ -{\alpha _j}^- \right) K(\mathbf {x}_i,\mathbf {x}_j) + y_{i} \right) \end{aligned}$$

(17a)