Extreme vector machine for fast training on large data

Abstract

Quite often, different types of loss functions are adopted in SVM or its variants to meet practical requirements. How to scale up the corresponding SVMs for large datasets are becoming more and more important in practice. In this paper, extreme vector machine (EVM) is proposed to realize fast training of SVMs with different yet typical loss functions on large datasets. EVM begins with a fast approximation of the convex hull, expressed by extreme vectors, of the training data in the feature space, and then completes the corresponding SVM optimization over the extreme vector set. When hinge loss function is adopted, EVM is the same as the approximate extreme points support vector machine (AESVM) for classification. When square hinge loss function, least squares loss function and Huber loss function are adopted, EVM corresponds to three versions, namely, L2-EVM, LS-EVM and Hub-EVM, respectively, for classification or regression. In contrast to the most related machine AESVM, with the retainment of its theoretical advantage, EVM is distinctive in its applicability to a wide variety of loss functions to meet practical requirements. Compared with the other state-of-the-art fast training algorithms CVM and FastKDE of SVMs, EVM indeed relaxes the limitation of least squares loss functions, and experimentally exhibits its superiority in training time, robustness capability and number of support vectors.

Appendices

Appendix 1

1.1 Proof of Theorem 1

• 1. For the L2-SVM:

Let ${L_{L2 - 2}}{\text{(}}{\mathbf{w}},b,{{\mathbf{X}}^*})=(C/2N)\sum\nolimits_{{t=1}}^{M} {l({\mathbf{w}},b,{{\mathbf{z}}_t})} \sum\nolimits_{{i=1}}^{N} {{r_{i,t}}}$ and ${L_{L2 - 3}}{\text{(}}{\mathbf{w}},b,{{\mathbf{X}}^*})=(C/2N)\sum\nolimits_{{t=1}}^{M} {l({\mathbf{w}},b,{{\mathbf{u}}_i})}$, where ${{\mathbf{u}}_i}=\sum\nolimits_{{i=1}}^{M} {{r_{i,t}}} {{\mathbf{z}}_t}$. From the precondition of y_i = y_j in each subset, we have

$$\begin{aligned} L_{{L2 - 3}} ({\mathbf{w}},b,{\mathbf{X}}^{*} ) & = (C/2N)\sum\nolimits_{{i = 1}}^{N} {\{ max\;\{ 0,\;[1 - y_{i} } ({\mathbf{w}}^{T} \sum\nolimits_{{t = 1}}^{M} {r_{{i,t}} {\mathbf{z}}_{t} + b)]\} \} ^{2} } \\ & = (C/2N)\sum\nolimits_{{i = 1}}^{N} {\{ max\{ 0,\;} \sum\nolimits_{{t = 1}}^{M} {r_{{i,t}} } [1 - y_{t} ({\mathbf{w}}^{T} {\mathbf{z}}_{t} + b)]\} \} ^{2} . \\ \end{aligned}$$

According to $max(0,\;A+B) \leq max(0,\;A)+max(0,\;B)$,

$$\begin{aligned} L_{{L2 - 3}} ({\mathbf{w}},b,{\mathbf{X}}^{*} ) & \le (C/2N)\sum\nolimits_{{i = 1}}^{N} {\sum\nolimits_{{t = 1}}^{M} {\{ max\{ 0,} } \;r_{{i,t}} [1 - y_{i} ({\mathbf{w}}^{T} {\mathbf{z}}_{t} + b)]\} \} ^{2} \\ & \le (C/2N)\sum\nolimits_{{t = 1}}^{M} {\{ max\{ 0,\;1 - y_{i} ({\mathbf{w}}^{T} {\mathbf{z}}_{t} + b} )\} \} ^{2} \sum\nolimits_{{i = 1}}^{N} {r_{{i,t}} } = L_{{L2 - 2}} ({\mathbf{w}},b,{\mathbf{X}}^{*} ). \\ \end{aligned}$$

Adding $(1/2){\left\| {\mathbf{w}} \right\|^2}$ to both sides of the inequality above, we get ${F_{L2-3}}{\text{(}}{\mathbf{w}},{\text{ }}b{\text{)}} \leq {F_{L2-2}}{\text{(}}{\mathbf{w}},{\text{ }}b{\text{)}}$.

• 2. For the LS-SVM:

Let ${L_{LS - 2}}{\text{(}}{\mathbf{w}},b,{{\mathbf{X}}^*})=(C/2N)\sum\nolimits_{{t=1}}^{M} {l({\mathbf{w}},b,{{\mathbf{z}}_t})} \sum\nolimits_{{i=1}}^{N} {{r_{i,t}}}$ and ${L_{LS - 3}}{\text{(}}{\mathbf{w}},b,{{\mathbf{X}}^*})=(C/2N)\sum\nolimits_{{i=1}}^{N} {l({\mathbf{w}},b,{{\mathbf{u}}_i})}$. We have

$$\begin{gathered} {L_{LS - 3}}{\text{(}}{\mathbf{w}},b,{{\mathbf{X}}^*})=(C/2N)\sum\nolimits_{{i=1}}^{N} {({y_i} - ({{\mathbf{w}}^T}} \sum\nolimits_{{t=1}}^{M} {{r_{i,t}}{{\mathbf{z}}_t}+} b){)^2} \hfill \\ =(C/2N)\sum\nolimits_{{i=1}}^{N} {(\sum\nolimits_{{t=1}}^{M} {{r_{i,t}}(} } {y_t} - {{\mathbf{w}}^T}{{\mathbf{z}}_t}+b){)^2}. \hfill \\ \end{gathered}$$

According to Jensen’s inequality, if ${\lambda _1},{\lambda _2}, \ldots ,{\lambda _n}$ are nonnegative real numbers such that${\lambda _1}+{\lambda _2}+ \ldots +{\lambda _n}=1$, $\varphi ( \cdot )$is a real convex function, then $\varphi ({\lambda _1}{x_1}+{\lambda _2}{x_2}+ \cdots +{\lambda _n}{x_n}) \leq {\lambda _1}\varphi ({x_1})+{\lambda _2}\varphi ({x_2})+ \cdots +{\lambda _n}\varphi ({x_n})$, for any ${x_1}, \ldots ,{x_n}$. So, we can get

$$\begin{aligned} {L_{LS - 3}}{\text{(}}{\mathbf{w}},b,{{\mathbf{X}}^*}) &\leq (C/2N)\sum\nolimits_{{i=1}}^{N} {\left(\sum\nolimits_{{t=1}}^{M} {r_{{i,t}}^{{}}} {{({y_t} - {{\mathbf{w}}^T}{{\mathbf{z}}_t}+b)}^2}\right)} \hfill \\ &= (C/2N)\sum\nolimits_{{t=1}}^{M} {{{({y_t} - {{\mathbf{w}}^T}{{\mathbf{z}}_t}+b)}^2}} \sum\nolimits_{{i=1}}^{N} {r_{{i,t}}^{{}}} . \hfill \\ \end{aligned}$$

Adding $(1/2){\left\| {\mathbf{w}} \right\|^2}$to both sides of the inequality above, we get ${F_{LS{\text{-3}}}}{\text{(}}{\mathbf{w}},{\text{ }}b{\text{)}} \leq {F_{LS - 2}}{\text{(}}{\mathbf{w}},{\text{ }}b{\text{)}}$.

• 3. For the Hub-SVM:

Let ${L_{Hub - 2app}}{\text{(}}{\mathbf{w}},{{\mathbf{X}}^*})=(C/2N)\sum\nolimits_{{t=1}}^{M} {{l_H}({\mathbf{w}},{{\mathbf{z}}_t})} \sum\nolimits_{{i=1}}^{N} {{r_{i,t}}},$ and

$$\begin{aligned} {L_{Hub - 3app}}{\text{(}}{\mathbf{w}},{{\mathbf{X}}^*}) & =(C/2N)\sum\nolimits_{{i=1}}^{N} {{l_H}({\mathbf{w}},{{\mathbf{u}}_i})} \\ & =(hC/N)\sum\nolimits_{{t=1}}^{M} {(\sqrt {1+{{(1+h - (C/2N){y_t}{{\mathbf{w}}^T}{{\mathbf{z}}_t})}^2}/(4{h^2})} - 1)} \sum\nolimits_{{i=1}}^{N} {{r_{i,t}}} \\ & ={L_{Hub - 2app}}{\text{(}}{\mathbf{w}},{{\mathbf{X}}^*}). \\ \end{aligned}$$

Adding $(1/2){\left\| {\mathbf{w}} \right\|^2}$to both sides of the inequality above, then ${F_{Hub - {\text{3}}app}}{\text{(}}{\mathbf{w}}{\text{)}}={F_{Hub - 2app}}{\text{(}}{\mathbf{w}}{\text{)}}$.

Appendix 2

2.1 Proof of Theorem 2

• 1. For the L2-SVM:

Let ${L_{L2 - 1}}{\text{(}}{\mathbf{w}},b,{\mathbf{X}})=(C/2N)\sum\nolimits_{{i=1}}^{N} {l({\mathbf{w}},b,{{\mathbf{z}}_i})}$ denote the average square hinge loss that is minimized in (7) and let ${L_{L2 - 3}}{\text{(}}{\mathbf{w}},b,{{\mathbf{X}}^*})$ be defined in Theorem1. Then, we have

$$\begin{aligned} {L_{L2 - 1}}{\text{(}}{\mathbf{w}},b,{\mathbf{X}}) & =(C/2N)\sum\nolimits_{{i=1}}^{N} {{{\{ max\{ 0,1 - {y_i}({{\mathbf{w}}^T}{{\mathbf{z}}_i}+b)\} \} }^2}} \\ & \leq (C/2N)\sum\nolimits_{{i=1}}^{N} {\{ max\{ 0,} \sum\nolimits_{{t=1}}^{M} {{r_{i,t}}} (1 - {y_t}({{\mathbf{w}}^T}{{\mathbf{z}}_t}+b))\} {\} ^2}+(C/2N)\sum\nolimits_{{i=1}}^{N} {max{{\{ 0, - {y_i}{{\mathbf{w}}^T}{\tau _i}\} }^2}} \\ & \quad +(C/N)\sum\nolimits_{{t=1}}^{M} {max\{ 0,\;(1 - {y_t}({{\mathbf{w}}^T}{{\mathbf{z}}_t}+b))\} } \sum\nolimits_{{i=1}}^{N} {{r_{i,t}}} max\{ 0, - {y_i}{{\mathbf{w}}^T}{\tau _i}\} \\ \end{aligned}$$

By assuming ${\Delta _1}=\mathop {max}\limits_{{1 \leq t \leq M}} (max\{ 0,\;(1 - {y_t}({{\mathbf{w}}^T}{{\mathbf{z}}_t}+b))\} )$, we further have

$${L_{L2 - 1}}{\text{(}}{\mathbf{w}},b,{\mathbf{X}}) \leq {L_{L2 - 3}}{\text{(}}{\mathbf{w}},b,{{\mathbf{X}}^*})+(C/2N)\sum\nolimits_{{i=1}}^{N} {{{\{ max\{ 0, - {y_i}{{\mathbf{w}}^T}{\tau _i}\} \} }^2}} +(CM{\Delta _1}/N)\sum\nolimits_{{i=1}}^{N} {max\{ 0, - {y_i}{{\mathbf{w}}^T}{\tau _i}\} } .$$

Adding $(1/2){\left\| {\mathbf{w}} \right\|^2}$ to both sides of the inequality above, then this theorem is proved.

• 2. For the LS-SVM:

We have ${L_{LS - 1}}{\text{(}}{\mathbf{w}},b,{\mathbf{X}})=(C/2N)\sum\nolimits_{{i=1}}^{N} {{{({y_i} - ({{\mathbf{w}}^T}{{\mathbf{z}}_i}+b))}^2}}$.

Assume that the maximum value of ${y_t} - ({{\mathbf{w}}^T}{{\mathbf{z}}_t}+b)$ in the dataset X is ${\Delta _2}= {\hbox{max} }_{{1 \leq t \leq M}} \left| {{y_t} - {{\mathbf{w}}^T}({{\mathbf{z}}_t}+b)} \right|$. Then, we have

$$\begin{aligned} {L_{LS - 1}}{\text{(}}{\mathbf{w}},b,{\mathbf{X}}) & \leq (C/2N)\sum\nolimits_{{i=1}}^{N} {(\sum\nolimits_{{t=1}}^{M} {{r_{i,t}}} } ({y_t} - {{\mathbf{w}}^T}{{\mathbf{z}}_t} - b){)^2}+(CM/N)({{\mathbf{w}}^T}{\Delta _2})\sum\nolimits_{{i=1}}^{N} {({\tau _i}{r_{i,t}})} +(C/2N)\sum\nolimits_{{i=1}}^{N} {{{({{\mathbf{w}}^T}{\tau _i})}^2}} \\ & ={L_{LS - 3}}{\text{(}}{\mathbf{w}},b,{{\mathbf{X}}^*})+(CM/N)({{\mathbf{w}}^T}{\Delta _2})\sum\nolimits_{{i=1}}^{N} {{\tau _i}} +(C/2N)\sum\nolimits_{{i=1}}^{N} {{{({{\mathbf{w}}^T}{\tau _i})}^2}} \\ \end{aligned}$$

Adding $(1/2){\left\| {\mathbf{w}} \right\|^2}$ to both sides of the inequality above, and in terms of M ≪ N, this theorem is proved.

• 3. For the Hub-SVM:

We have

$$\begin{aligned} {L_{Hub - 1app}}({\mathbf{w}},{\mathbf{X}}) & =(hC/N)\sum\nolimits_{{i=1}}^{N} {(\sqrt {1+{{(1+h - yf({\mathbf{x}}))}^2}/(4{h^2})} - 1)} \\ & \leq (hC/N)\sum\nolimits_{{i=1}}^{N} {\{ (\sqrt {1+(1+h - (C/2N){y_i}{{\mathbf{w}}^T}(\sum\nolimits_{{t=1}}^{M} {{r_{i,t}}{{\mathbf{z}}_t}){)^2}} /(4{h^2})} - 1)} \\ & \quad +(C/Nh)\left| {{{\mathbf{w}}^T}{\tau _i}} \right|+\sqrt {(C/N)\left| {(1+h - (C/2N){y_i}{{\mathbf{w}}^T}(\sum\nolimits_{{t=1}}^{M} {{r_{i,t}}{{\mathbf{z}}_t}))({y_i}{{\mathbf{w}}^T}{\tau _i})} } \right|/(4{h^2})} \} \\ & ={L_{Hub - 3app}}({\mathbf{w}},{{\mathbf{X}}^*})+{(C/2N)^2}\sum\nolimits_{{i=1}}^{N} {\left| {{{\mathbf{w}}^T}{\tau _i}} \right|} +{(C/2N)^{\frac{3}{2}}}\sum\nolimits_{{t=1}}^{M} {\sqrt {2\left| {1+h - (C/2N){y_t}{{\mathbf{w}}^T}{{\mathbf{z}}_t}} \right|} } \sum\nolimits_{{i=1}}^{N} {{r_{i,t}}\sqrt {\left| {{{\mathbf{w}}^T}{\tau _i}} \right|} } . \\ \end{aligned}$$

Meanwhile, we can get:

$$\begin{aligned} {L_{Hub - 1app}}({\mathbf{w}},{\mathbf{X}}) & =\;(hC/N)\sum\nolimits_{{i=1}}^{N} {(\sqrt {1+(1+h - (C/2N){y_i}{{\mathbf{w}}^T}{{(\sum\nolimits_{{t=1}}^{M} {{r_{i,t}}{{\mathbf{z}}_t}+{\tau _i})} )}^2}/(4{h^2})} - 1)} \hfill \\ & \geq (hC/N)\sum\nolimits_{{i=1}}^{N} {\{ (\sqrt {1+(1+h - (C/2N){y_i}{{\mathbf{w}}^T}(\sum\nolimits_{{t=1}}^{M} {{r_{i,t}}{{\mathbf{z}}_t}){)^2}} /(4{h^2})} - 1)} \hfill \\ &\quad - (C/Nh)\left| {{{\mathbf{w}}^T}{\tau _i}} \right| - \sqrt {(C/N)\left| {(1+h - (C/2N){y_i}{{\mathbf{w}}^T}(\sum\nolimits_{{t=1}}^{M} {{r_{i,t}}{{\mathbf{z}}_t}))({y_i}{{\mathbf{w}}^T}{\tau _i})} } \right|/(4{h^2})} \} \hfill \\ & ={L_{Hub - 3app}}({\mathbf{w}},{{\mathbf{X}}^*}) - {(C/2N)^2}\sum\nolimits_{{i=1}}^{N} {\left| {{{\mathbf{w}}^T}{\tau _i}} \right|} - {(C/2N)^{\frac{3}{2}}}\sum\nolimits_{{t=1}}^{M} {\sqrt {2\left| {1+h - (C/2N){y_t}{{\mathbf{w}}^T}{{\mathbf{z}}_t}} \right|} } \sum\nolimits_{{i=1}}^{N} {{r_{i,t}}\sqrt {\left| {{{\mathbf{w}}^T}{\tau _i}} \right|} } . \hfill \\ \end{aligned}$$

Appendix 3

3.1 Proof of Corollary 1

• 1. For the L2-EVM, ${F_{L2 - 1}}({\mathbf{w}}_{1}^{*},\;b_{1}^{*}) - {F_{L2 - 2}}({\mathbf{w}}_{2}^{*},\;b_{2}^{*}) \leq {C^2}\varepsilon /2+CM{\Delta _1}\sqrt {C\varepsilon }$.

$$\begin{aligned} F_{{L2 - 1}} ({\mathbf{w}}_{2}^{*} ,\;b_{2}^{*} ) - F_{{L2 - 3}} ({\mathbf{w}}_{2}^{*} ,\;b_{2}^{*} ) & \le (C/2N)\sum\nolimits_{{i = 1}}^{N} {\{ max\{ 0, - y_{i} {\mathbf{w}}_{2}^{{*T}} \tau _{i} \} \} ^{2} + (CM\Delta _{1} /N)\sum\nolimits_{{i = 1}}^{N} {max\{ 0, - y_{i} {\mathbf{w}}_{2}^{{*T}} \tau _{i} \} } } \\ & \le (C/2N)\left( {\sum\nolimits_{{i = 1}}^{N} {\left\| {{\mathbf{w}}_{2}^{*} } \right\|^{2} } \left\| {\tau _{i} } \right\|^{2} } \right) + (CM\Delta _{1} /N)\left( {\sum\nolimits_{{i = 1}}^{N} {\left\| {{\mathbf{w}}_{2}^{*} } \right\|} \left\| {\tau _{i} } \right\|} \right) \\ & \le C^{2} \varepsilon /2 + CM\Delta _{1} \sqrt {C\varepsilon } . \\ \end{aligned}$$

Since$({\mathbf{w}}_{1}^{*},\;b_{1}^{*})$is the solution of (7), ${F_{L{\text{2-1}}}}{\text{(}}{\mathbf{w}}_{1}^{*},{\text{ }}b_{1}^{*}{\text{)}} \leq {F_{L2 - 1}}{\text{(}}{\mathbf{w}}_{2}^{*},{\text{ }}b_{2}^{*}{\text{)}}$.

Using Theorem1, we obtain

$$\begin{aligned} F_{{L2 - 1}} ({\mathbf{w}}_{1}^{*} ,\;b_{1}^{*} ) - F_{{L2 - 2}} ({\mathbf{w}}_{2}^{*} ,\;b_{2}^{*} ) & \le F_{{L2 - 1}} ({\mathbf{w}}_{1}^{*} ,\;b_{1}^{*} ) - F_{{L2 - 3}} ({\mathbf{w}}_{2}^{*} ,\;b_{2}^{*} ) \\ & \le F_{{L2 - 1}} ({\mathbf{w}}_{2}^{*} ,\;b_{2}^{*} ) - F_{{L2 - 3}} ({\mathbf{w}}_{2}^{*} ,\;b_{2}^{*} ) \\ & \le C^{2} \varepsilon /2 + CM\Delta _{1} \sqrt {C\varepsilon } . \\ \end{aligned}$$

• 2. For the LS-EVM, ${F_{LS - 1}}({\mathbf{w}}_{1}^{*},\;b_{1}^{*}) - {F_{LS - 2}}({\mathbf{w}}_{2}^{*},\;b_{2}^{*}) \leq CM{\Delta _2}\Omega \sqrt \varepsilon +C{\Omega ^2}\varepsilon /2$.

Since ${F_{LS - 1}}{\text{(}}{\mathbf{w}}_{2}^{*},\;b_{2}^{*}) - {F_{LS - 3}}({\mathbf{w}}_{2}^{*},\;b_{2}^{*}) \leq (CM/N){\mathbf{w}}{_{2}^{*T}}{\Delta _2}\sum\nolimits_{{i=1}}^{N} {{\tau _i}} +(C/2N)\sum\nolimits_{{i=1}}^{N} {{{{\text{(}}{\mathbf{w}}{{_{2}^{*T}}}{\tau _i}{\text{)}}}^2}} \;\; \leq CM{\Delta _2}\Omega \sqrt \varepsilon +C{\Omega ^2}\varepsilon /2,$ we have

$$\begin{gathered} {F_{LS - 1}}({\mathbf{w}}_{1}^{*},\;b_{1}^{*}) - {F_{LS - 2}}({\mathbf{w}}_{2}^{*},\;b_{2}^{*}) \leq {F_{LS - 1}}({\mathbf{w}}_{1}^{*},\;b_{1}^{*}) - {F_{LS - 3}}({\mathbf{w}}_{2}^{*},\;b_{2}^{*}) \hfill \\ \leq {F_{LS - 1}}({\mathbf{w}}_{2}^{*},\;b_{2}^{*}) - {F_{LS - 3}}({\mathbf{w}}_{2}^{*},\;b_{2}^{*}) \leq CM{\Delta _2}\Omega \sqrt \varepsilon +C{\Omega ^2}\varepsilon /2. \hfill \\ \end{gathered}$$

• 3. For the Hub-SVM,

  $$\begin{gathered} - ({C^2}/4N)\sqrt {C\varepsilon (1+h)/2} - ({\Delta _3}MC\sqrt {CN} /2N){(C\varepsilon (1+h)/2)^{1/4}} \leq {F_{Hub - 1app}}({\mathbf{w}}_{1}^{*}) - {F_{Hub - 2app}}({\mathbf{w}}_{2}^{*}), \hfill \\ \leq ({C^2}/4N)\sqrt {C\varepsilon (1+h)/2} +({\Delta _3}MC\sqrt {CN} /2N){(C\varepsilon (1+h)/2)^{1/4}}, \hfill \\ \end{gathered}$$

  where ${\Delta _3}= {\hbox{max} }_{{1 \leq t \leq M}} \sqrt {\left| {1+h - (C/2N){y_t}{{\mathbf{w}}^T}{{\mathbf{z}}_t}} \right|}$.

According to Theorems1 and 2, we get

$$\begin{aligned} {F_{Hub - 1app}}({\mathbf{w}}_{1}^{*}) - {F_{Hub - 2app}}({\mathbf{w}}_{2}^{*})&={F_{Hub - 1app}}({\mathbf{w}}_{1}^{*}) - {F_{Hub - 3app}}({\mathbf{w}}_{2}^{*}) \hfill \\ &\leq {F_{Hub - 1app}}({\mathbf{w}}_{2}^{*}) - {F_{Hub - 3app}}({\mathbf{w}}_{2}^{*}) \hfill \\ & \leq {(C/2N)^2}\sum\nolimits_{{i=1}}^{N} {\left| {{\mathbf{w}}{{_{2}^{*}}^T}{\tau _i}} \right|} {\text{+}}(C/2N)\sum\nolimits_{{t=1}}^{M} {\sqrt {(C/N)\left| {1+h - (C/2N){y_t}{\mathbf{w}}{{_{2}^{*}}^T}{{\mathbf{z}}_t}} \right|} } \sum\nolimits_{{i=1}}^{N} {{r_{i,t}}} \sqrt {\left| {{\mathbf{w}}{{_{2}^{*}}^T}{\tau _i}} \right|} . \hfill \\ \end{aligned}$$

Let us define ${\Delta _3}= {\hbox{max} }_{{1 \leq t \leq M}} \sqrt {\left| {1+h - (C/2N){y_t}{\mathbf{w}}{{_{2}^{*}}^T}{{\mathbf{z}}_t}} \right|}$, in terms of $\left\| {{{\mathbf{w}}^*}} \right\| \leq \sqrt {C(1+h)/2}$, we immediately have

$$\begin{gathered} {F_{Hub - 1app}}({\mathbf{w}}_{1}^{*}) - {F_{Hub - 2app}}({\mathbf{w}}_{2}^{*}) \leq ({C^2}/4N)\sqrt {C\varepsilon (1+h)/2} +({\Delta _3}MC\sqrt {CN} /2N){(C\varepsilon (1+h)/2)^{1/4}}, \hfill \\ {F_{Hub - 1app}}({\mathbf{w}}_{1}^{*}) - {F_{Hub - 2app}}({\mathbf{w}}_{2}^{*}) \geq - ({C^2}/4N)\sqrt {C\varepsilon (1+h)/2} - ({\Delta _3}MC\sqrt {CN} /2N){(C\varepsilon (1+h)/2)^{1/4}}. \hfill \\ \end{gathered}$$

Appendix 4

4.1 Proof of Corollary 2

Based on Theorem1, we know that ${F_{Hub - 3app}}({\mathbf{w}}_{3}^{*}) \leq {F_{Hub - 3app}}({\mathbf{w}}_{2}^{*})={F_{Hub - 2app}}({\mathbf{w}}_{2}^{*})$. Meanwhile, ${F_{Hub - 3app}}({\mathbf{w}}_{3}^{*})={F_{Hub - 2app}}({\mathbf{w}}_{3}^{*}) \geq {F_{Hub - 2app}}({\mathbf{w}}_{2}^{*})$. Hence, we have ${F_{Hub - 3app}}({\mathbf{w}}_{3}^{*})={F_{Hub - 3app}}({\mathbf{w}}_{2}^{*})$.

From these results, we get ${F_{Hub - 3app}}({\mathbf{w}}_{2}^{*}) - {F_{Hub - 3app}}({\mathbf{w}}_{1}^{*}) \leq 0$. From Theorem2, we have the following inequalities

$$\begin{aligned}& - {{\text{(}}C/N{\text{)}}^2}\sum\nolimits_{{i=1}}^{N} {\left| {{\mathbf{w}}{{_{2}^{*}}^T}{\tau _i}} \right|} - {\text{(}}C/N{\text{)}}\sum\nolimits_{{t=1}}^{M} {\sqrt {{\text{(}}2C/N{\text{)}}\left| {1+h - {\text{(}}C/N{\text{)}}{y_t}{\mathbf{w}}{{_{2}^{*}}^T}{{\mathbf{z}}_t}} \right|} }\\ & \quad\sum\nolimits_{{i=1}}^{N} {{r_{i,t}}} \sqrt {\left| {{\mathbf{w}}{{_{2}^{*}}^T}{\tau _i}} \right|} \leq {F_{Hub - {\text{1}}app}}{\text{(}}{\mathbf{w}}_{1}^{*}{\text{)}} - {F_{Hub - 3app}}{\text{(}}{\mathbf{w}}_{1}^{*}{\text{),}}\end{aligned}$$

and

$$\begin{aligned}& {F_{Hub - {\text{1}}app}}{\text{(}}{\mathbf{w}}_{2}^{*}{\text{)}} - {F_{Hub - 3app}}{\text{(}}{\mathbf{w}}_{2}^{*}{\text{)}} \leq {{\text{(}}C/N{\text{)}}^2}\sum\nolimits_{{i=1}}^{N} {\left| {{\mathbf{w}}{{_{2}^{*}}^T}{\tau _i}} \right|} {\text{+(}}C/N{\text{)}}\\ & \quad \sum\nolimits_{{t=1}}^{M} {\sqrt {{\text{(}}2C/N{\text{)}}\left| {1+h - {\text{(}}C/N{\text{)}}{y_t}{\mathbf{w}}{{_{2}^{*}}^T}{{\mathbf{z}}_t}} \right|} } \sum\nolimits_{{i=1}}^{N} {{r_{i,t}}} \sqrt {\left| {{\mathbf{w}}{{_{2}^{*}}^T}{\tau _i}} \right|}. \end{aligned}$$

Adding these two inequalities and using the properties of $\left\| {{{\mathbf{w}}^*}} \right\| \leq \sqrt {C(1+h)/2}$, we get

$$\begin{gathered} {F_{Hub - 1app}}({\mathbf{w}}_{2}^{*}) - {F_{Hub - 1app}}({\mathbf{w}}_{1}^{*}) \leq {\text{2(}}C/N{{\text{)}}^2}\sum\nolimits_{{i=1}}^{N} {\left| {{\mathbf{w}}{{_{2}^{*}}^T}{\tau _i}} \right|} \hfill \\ \quad +2 (C/N{{\text{)}}^{\frac{3}{2}}}\sum\nolimits_{{t=1}}^{M} {\sqrt {2\left| {1+h - {\text{(}}C/N{\text{)}}{y_t}{\mathbf{w}}{{_{2}^{*}}^T}{{\mathbf{z}}_t}} \right|} } \sum\nolimits_{{i=1}}^{N} {{r_{i,t}}} \sqrt {\left| {{\mathbf{w}}{{_{2}^{*}}^T}{\tau _i}} \right|} \hfill \\ \leq 2{C^2}\sqrt {C\varepsilon (1+h)} {\text{/}}N{\text{+2}}{\Delta _3}MC\sqrt {2CN} {(C\varepsilon (1+h))^{1/4}}/N. \hfill \\ \end{gathered}$$