Robust large-scale online kernel learning

Abstract

The control-based approach has been proved to be effective for developing robust online learning methods. However, the existing control-based kernel methods are infeasible for large-scale modeling due to their high computational complexity. This paper aims to propose a computationally efficient control-based framework for robust large-scale kernel learning problems. By random feature approximation and robust loss function, the learning problems are first transformed into a group of linear feedback control problems with sparse discrete large-scale algebraic Riccati equations (DARE). Then, with the solutions of the DAREs, two promising algorithms are developed to address large-scale binary classification and regression problems, respectively. Thanks to the sparseness, explicit solutions rather than numerical solutions of the DAREs are derived by utilizing matrix computation techniques developed in our study. This substantially reduces the complexity, and makes the proposed algorithms computationally efficient for large-scale complex datasets. Compared with the existing benchmarks, the proposed algorithms can achieve faster convergent, more robust and accurate modeling results. Theoretical analysis and encouraging numerical results on synthetic and realistic datasets are also provided to illustrate the effectiveness and efficiency of our algorithms.

Appendices

Appendix A

Proof

From standard optimal control theory [60], for any given n, V(E(n)) is quadratic, i.e., there exists a symmetric positive definite matrix $\mathcal {P}_n$ such that $V(E(n))=E(n)^T\mathcal {P}_nE(n)=E_n(1)^T\mathcal {P}_nE_n(1)$. A Hamilton-Jacobi equation is given as

$$\begin{aligned} V(E(n))= & \min _{U_n(1),\ldots ,U_n(N)}\sum \limits _{t=1}^{\infty }E_n(t)^TE_n(t)+\gamma U_n(t)^TU_n(t)\\= & \min \limits _{U_n(1)}(E_n(1)^TE_n(1)+\gamma U_n(1)^TU_n(1)+V(E_n(1)+\mathcal {B}_nU_n(1))). \end{aligned}$$

It follows that

$$\begin{aligned} V(E(n))= & \min \limits _{U_n(1)}(E_n(1)^TE_n(1)+\gamma U_n(1)^TU_n(1) \\&+(E_n(1)+\mathcal {B}_nU_n(1))^T\mathcal {P}_n(E_n(1)+\mathcal {B}_nU_n(1))). \end{aligned}$$

To minimize V(E(n)), we set the partial derivative with respect to $U_n(1)$ to zero:

$$\begin{aligned} 2U_n(1)^T\gamma I+2(E_n(1)+\mathcal {B}_nU_n(1))^T\mathcal {P}_n\mathcal {B}_n=0. \end{aligned}$$

This leads to

$$\begin{aligned} U_n^{\star }(1)=-(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)^{-1}\mathcal {B}_n^T\mathcal {P}_nE_n(1). \end{aligned}$$

V(E(n)) can be rewritten as

$$\begin{aligned} V(E(n))= & E_n(1)^T\mathcal {P}_nE_n(1)\\= & (E_n(1)+\mathcal {B}_nU_n^{\star }(1))^T\mathcal {P}_n(E_n(1)+\mathcal {B}_nU_n^{\star }(1))\\&+E_n(1)^TE_n(1)+\gamma U_n^{\star }(1)^TU_n^{\star }(1). \end{aligned}$$

With $U_n^\star (1)$, it follows

$$\begin{aligned}&E_n(1)^T\mathcal {P}_nE_n(1)\\= & E_n(1)^TE_n(1)+\gamma U_n^{\star }(1)^TU_n^{\star }(1)+(E_n(1)+\mathcal {B}_nU_n^{\star }(1))^T\mathcal {P}_n(E_n(1)+\mathcal {B}_nU_n^{\star }(1))\\= & E_n(1)^TE_n(1)+E_n(1)^T\mathcal {P}_nE_n(1)\\&+\gamma E_n(1)^T\mathcal {P}_n\mathcal {B}_n(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)^{-1}(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)^{-1}\mathcal {B}_n^T\mathcal {P}_nE_n(1)\\&+E_n(1)^T\mathcal {P}_n\mathcal {B}_n(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)^{-1}\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)^{-1}\mathcal {B}_n^T\mathcal {P}_nE_n(1)\\&-2E_n(1)^T\mathcal {P}_n\mathcal {B}_n(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)^{-1}\mathcal {B}_n^T\mathcal {P}_nE_n(1)\\= & E_n(1)^TE_n(1)+E_n(1)^T\mathcal {P}_nE_n(1)\\&+E_n(1)^T\mathcal {P}_n\mathcal {B}_n(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)^{-1}(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)^{-1}\mathcal {B}_n^T\mathcal {P}_nE_n(1)\\&-2E_n(1)^T\mathcal {P}_n\mathcal {B}_n(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)^{-1}\mathcal {B}_n^T\mathcal {P}_nE_n(1)\\= & E_n(1)^TE_n(1)+E_n(1)^T\mathcal {P}_nE_n(1)-E_n(1)^T\mathcal {P}_n\mathcal {B}_n(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)^{-1}\mathcal {B}_n^T\mathcal {P}_nE_n(1). \end{aligned}$$

Since this must hold for all $E_n(1)$, we have the following discrete-time algebraic Riccati equation:

$$\begin{aligned} \mathcal {P}_n=I+\mathcal {P}_n-\mathcal {P}_n\mathcal {B}_n(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)^{-1}\mathcal {B}_n^T\mathcal {P}_n. \end{aligned}$$

The solution $\mathcal {P}_n$ is symmetric positive definite and stabilizing, and the update law of the online learning is given by the optimal input $\Delta \theta (n)=U_n^{\star }(1)=\mathcal {F}_nE_n(1)=\mathcal {F}_nE(n)$, where $\mathcal {F}_n=-(\gamma I+\mathcal {B}_n^T\mathcal {P}_n\mathcal {B}_n)^{-1}\mathcal {B}_n^T\mathcal {P}_n$. The proof is completed. $\square$

Appendix B

Proof

In the following, for simplicity, $\mathcal {L}(f_n,x(n),y(n))$, $\mathcal {L}(f^{\star },x(n),y(n))$, $\mathcal {L}(\theta (n),x(n),y(n))$ and $\mathcal {L}(\theta ^{\star },x(n),y(n))$ are denoted as $\mathcal {L}_n(f_n)$, $\mathcal {L}_n(f^{\star })$, $\mathcal {L}_n(\theta (n))$ and $\mathcal {L}_n(\theta ^{\star })$, respectively. Apparently, the loss functions $\mathcal {L}_n$ are convex, the convexity of the loss function implies

$$\begin{aligned} \mathcal {L}_n(\theta (n))-\mathcal {L}_n(\theta ^{\star })\le \nabla \mathcal {L}_n(\theta (n))^T(\theta (n)-\theta ^{\star }). \end{aligned}$$

Meanwhile, for any fixed $\theta ^{\star }$, we have

$$\begin{aligned}&\Vert \theta (n+1)-\theta ^{\star }\Vert ^2\\&=\quad \Vert \theta (n)-\mathcal {B}_n^T\mathcal {G}_n^{-1}\mathcal {P}_n^{-1}E(n)-\theta ^{\star }\Vert ^2\\&=\quad \Vert \theta (n)-\theta ^{\star }\Vert ^2+\Vert \mathcal {B}_n^T\mathcal {G}_n^{-1}\mathcal {P}_n^{-1}E(n)\Vert ^2-2(\mathcal {B}_n^T\mathcal {G}_n^{-1}\mathcal {P}_n^{-1}E(n))^T(\theta (n)-\theta ^{\star })\\&=\quad \Vert \theta (n)-\theta ^{\star }\Vert ^2+(\mathcal {G}_n^{-1}\mathcal {P}_n^{-1}E(n))^T\mathcal {B}_n\mathcal {B}_n^T\mathcal {G}_n^{-1}\mathcal {P}_n^{-1}E(n)\\&\quad -2(\mathcal {G}_n^{-1}\mathcal {P}_n^{-1}E(n))^T\mathcal {B}_n(\theta (n)-\theta ^{\star }). \end{aligned}$$

In the settings of our proposed algorithms, $\mathcal {G}_n=\mathcal {B}_n\mathcal {B}_n^T$, $\mathcal {P}_n$ and E(n) are scalars. Noticing that

$$\begin{aligned} \mathcal {B}_n^T=\frac{\partial \mathcal {L}(\theta (n),x(n),y(n))}{\partial \theta (n)}=\nabla \mathcal {L}(\theta (n),x(n),y(n))=\nabla \mathcal {L}_n(\theta (n)), \end{aligned}$$

we have

$$\begin{gathered} \left\| {\theta (n + 1) - \theta ^{{ \star }} } \right\|^{2} \hfill \\ \quad = \left\| {\theta (n) - \theta ^{{ \star }} } \right\|^{2} + ({\mathcal{G}}_{n}^{{ - 1}} {\mathcal{P}}_{n}^{{ - 1}} E(n))^{T} {\mathcal{P}}_{n}^{{ - 1}} E(n) \hfill \\ \quad - 2{\mathcal{G}}_{n}^{{ - 1}} {\mathcal{P}}_{n}^{{ - 1}} E(n)\nabla {\mathcal{L}}_{n} (\theta (n))^{T} (\theta (n) - \theta ^{{ \star }} ) \hfill \\ \quad = \left\| {\theta (n) - \theta ^{{ \star }} } \right\|^{2} + {\mathcal{G}}_{n}^{{ - 1}} ({\mathcal{P}}_{n}^{{ - 1}} )^{2} E(n)^{2} \hfill \\ \quad - 2{\mathcal{G}}_{n}^{{ - 1}} {\mathcal{P}}_{n}^{{ - 1}} E(n)\nabla {\mathcal{L}}_{n} (\theta (n))^{T} (\theta (n) - \theta ^{{ \star }} ). \hfill \\ \end{gathered}$$

Then,

$$\begin{aligned}&\mathcal {G}_n^{-1}\mathcal {P}_n^{-1}E(n)\nabla \mathcal {L}_n(\theta (n))^T(\theta (n)-\theta ^{\star })\\= & \Vert \theta (n)-\theta ^{\star }\Vert ^2-\Vert \theta (n+1)-\theta ^{\star }\Vert ^2+\mathcal {G}_n^{-1}(\mathcal {P}_n^{-1})^2E(n)^2. \end{aligned}$$

For $\mathcal {L}_n(\theta (n))-\mathcal {L}_n(\theta ^{\star })\le \nabla \mathcal {L}_n(\theta (n))^T(\theta (n)-\theta ^{\star })$, it follows

$$\begin{gathered} {\mathcal{L}}_{n} (\theta (n)) - {\mathcal{L}}_{n} (\theta ^{{ \star }} ) \hfill \\ \quad \le \frac{1}{{2{\mathcal{G}}_{n}^{{ - 1}} {\mathcal{P}}_{n}^{{ - 1}} E(n)}}\left( {\left\| {\theta (n) - \theta ^{{ \star }} } \right\|^{2} - \left\| {\theta (n + 1) - \theta ^{{ \star }} {}} \right\|^{2} } \right) \hfill \\ \quad + \frac{1}{2}{\mathcal{P}}_{n}^{{ - 1}} E(n), \hfill \\ \end{gathered}$$

which yields

$$\begin{gathered} \sum\limits_{{n = 1}}^{N} {\left( {{\mathcal{L}}_{n} (\theta (n)) - {\mathcal{L}}_{n} (\theta ^{{ \star }} )} \right)} \hfill \\ \quad \le \sum\limits_{{n = 1}}^{N} {\left( {\frac{1}{{2{\mathcal{G}}_{n}^{{ - 1}} {\mathcal{P}}_{n}^{{ - 1}} E(n)}}\left( {\left\| {\theta (n) - \theta ^{{ \star }} } \right\|^{2} - \left\| {\theta (n + 1) - \theta ^{{ \star }} } \right\|^{2} } \right) \quad + \frac{1}{2}{\mathcal{P}}_{n}^{{ - 1}} E(n)} \right)} . \hfill \\ \end{gathered}$$

Since $\mathcal {P}_n=\frac{1}{2} \left( 1+\sqrt{1+4 \gamma \mathcal {G}_n^{-1}} \right)$, $\theta (n)$ and $\theta ^{\star }$ are assumed to lie in a compact set, it is trivial to verify that $\mathcal {G}_n$ and E(n) are positive and bounded, $\forall n$. Thus, there exit positive constants $c_1$ and $c_2$, such that $1/\mathcal {G}_n^{-1}\mathcal {P}_n^{-1}E(n)\le c_1\sqrt{N}$ and $\frac{1}{2}\mathcal {P}_n^{-1}E(n)\le c_2/\sqrt{N}$, for $\forall n$. On the other hand, since by the proposed learning law, $\theta (n)$ always converges to the optimal vector $\theta ^{\star }$, $\Vert \theta (n)-\theta ^{\star }\Vert ^2-\Vert \theta (n+1)-\theta ^{\star }\Vert ^2>0$. It follows

$$\begin{aligned}&\sum \limits _{n=1}^{N}\bigg {(}\mathcal {L}_n(\theta (n))-\mathcal {L}_n(\theta ^{\star })\bigg {)}\\\le & \sum \limits _{n=1}^{N}c_1\sqrt{N}(\Vert \theta (n)-\theta ^{\star }\Vert ^2-\Vert \theta (n+1)-\theta ^{\star }\Vert ^2)+\sum \limits _{n=1}^{N}\frac{1}{2}\mathcal {P}_n^{-1}E(n)\\= & c_1\sqrt{N}(\Vert \theta (1)-\theta ^{\star }\Vert ^2-\Vert \theta (n+1)-\theta ^{\star }\Vert ^2)+\sum \limits _{n=1}^{N}\frac{1}{2}\mathcal {P}_n^{-1}E(n)\\\le & c_1\sqrt{N}(\Vert \theta (1)-\theta ^{\star }\Vert ^2-\Vert \theta (n+1)-\theta ^{\star }\Vert ^2)+c_2\sqrt{N}\\\le & c_1\sqrt{N}\Vert \theta (1)-\theta ^{\star }\Vert ^2+c_2\sqrt{N}. \end{aligned}$$

Let $\theta (1)=0$, and we have

$$\begin{aligned} \sum \limits _{n=1}^{N}\mathcal {L}_n(\theta (n))-\sum \limits _{n=1}^{N}\mathcal {L}_n(\theta ^{\star })\le & (c_1||\theta ^{\star }||^2+c_2)\sqrt{N}. \end{aligned}$$

Based on the Claim 1 in [58], there exists a constant $\delta _0$ ($\delta _0$ is arbitrary small as D increases), such that for $\forall x_1,x_2$,

$$\begin{aligned} \mid \mathcal {Z}_{\varvec{u}}(x_1)^T\mathcal {Z}_{\varvec{u}}(x_2)-K_{\sigma }(x_1,x_2)\mid< & \delta _0. \end{aligned}$$

Following the method given in [54, 56], when D is sufficiently large, it is trivial to verify that there exists there exists a constant $\delta$ ($\delta$ is also arbitrary small as D increases) and a positive constant $c_3$, such that

$$\begin{gathered} \left| {\sum\limits_{{t = n}}^{N} {{\mathcal{L}}_{t} } (\theta ^{{ \star }} ) - \sum\limits_{{n = 1}}^{N} {{\mathcal{L}}_{n} } (f^{{ \star }} ){\mid } \le \sum\limits_{{n = 1}}^{N} {\mid } {\mathcal{L}}_{n} (\theta ^{{ \star }} ) - {\mathcal{L}}_{n} (f^{{ \star }} )} \right| \hfill \\ \quad \le c_{3} \delta N. \hfill \\ \end{gathered}$$

Let $\delta =1/\sqrt{N}$,

$$\begin{aligned} \sum \limits _{n=1}^{N}\mathcal {L}_n(\theta (n))-\sum \limits _{n=1}^{N}\mathcal {L}_n(f^{\star })\le & (c_1||\theta ^{\star }||^2+c_2+c_3)\sqrt{N}. \end{aligned}$$

The proof is completed. $\square$