A Sparse Online Approach for Streaming Data Classification via Prototype-Based Kernel Models

Abstract

Processing big data streams through machine learning algorithms has various challenges, such as little time to train the models, hardware memory constraints, and concept drift. In this paper, we show that prototype-based kernel classifiers designed by sparsification procedures, such as the approximate linear dependence (ALD) method, provides an adequate tradeoff between accuracy and size complexity of kernelized nearest neighbor classifiers. The proposed approach automatically selects relevant samples from the training data stream to form a sparse dictionary of prototypes, which are then used in kernelized distance metrics to classify arriving samples on the fly. Additionally, the proposed method is fully adaptive, in the sense that it updates and removes prototypes from the dictionary, enabling it to learn continuously in nonstationary environments. The results obtained from a comprehensive set of computer simulations involving artificial and real streaming data sets indicate that the proposed algorithm can build models with low complexity and competitive classification error rates compared to state of the art.

Appendix - Kernelized Distance Calculation

This section is devoted to demonstrate that kernel functions based on the squared Euclidean distance $\left\| \mathbf{x} - \mathbf{w} _i \right\| _{2}^{2}$, such as the widely used Gaussian radial basis function (RBF), lead to kernelized distances that reduces to the standard Euclidean distance function. Also, the derivative of these functions are proportional to either the difference between the argument vectors $(\mathbf{x} - \mathbf{w} _i)$ or a normalized version of this difference.

Firstly, considering the linear kernel $k(\mathbf {u},\mathbf {v})=\mathbf {u}^T\mathbf {v}$ and Eqs. (9) and (11), the cost function and its derivative are

$$\begin{aligned} J_i(\mathbf{x} )= & {} \mathbf{x} ^{T}{} \mathbf{x} -2\mathbf{w} _{i}^{T}{} \mathbf{x} + \mathbf{w} _i^{T}{} \mathbf{w} _i, \end{aligned}$$

(33)

$$\begin{aligned}= & {} (\mathbf{x} - \mathbf{w} _i)^T(\mathbf{x} - \mathbf{w} _i), \end{aligned}$$

(34)

$$\begin{aligned}= & {} \left\| \mathbf{x} - \mathbf{w} _i \right\| _{2}^{2}, \end{aligned}$$

(35)

whose gradient vector is given by

$$\begin{aligned} \nabla J_i(\mathbf {x}) = 2(\mathbf{w} _{i} - \mathbf{x} ). \end{aligned}$$

(36)

Now, considering the Gaussian kernel, the cost function and its gradient vector are given by

$$\begin{aligned} J_i \left( \mathbf {x} \right)= & {} \exp \left( - \frac{\left\| \mathbf{x} -\mathbf{x} \right\| _{2}^{2}}{2\gamma ^2} \right) -2 \exp \left( - \frac{\left\| \mathbf{w} _i-\mathbf{x} \right\| _{2}^{2}}{2\gamma ^2} \right) + \exp \left( - \frac{\left\| \mathbf{w} _i-\mathbf{w} _i \right\| _{2}^{2}}{2\gamma ^2} \right) \end{aligned}$$

(37)

$$\begin{aligned} J_i \left( \mathbf {x} \right)= & {} 2 - 2 \exp \left( - \frac{\left\| \mathbf{w} _i-\mathbf{x} \right\| _{2}^{2}}{2\gamma ^2} \right) . \end{aligned}$$

(38)

$$\begin{aligned} \nabla J_{i}(\mathbf{x} )= & {} - 2 \exp \left( - \frac{\left\| \mathbf{w} _i-\mathbf{x} \right\| _{2}^{2}}{2\gamma ^2} \right) \left( - \frac{1}{2\gamma ^2} \right) 2(\mathbf{w} _{i} - \mathbf{x} )\nonumber \\ \nabla J_{i}(\mathbf{x} )= & {} \left( \frac{2}{\gamma ^2} \right) \exp \left( - \frac{\left\| \mathbf{w} _i-\mathbf{x} \right\| _{2}^{2}}{2\gamma ^2} \right) (\mathbf{w} _{i} - \mathbf{x} ). \end{aligned}$$

(39)

Without loss of generality, we can set $\gamma =\frac{1}{\sqrt{2}}$. Thus, Eq. (39) can be rewritten as

$$\begin{aligned} \nabla J_{i}(\mathbf{x} ) = \left[ \frac{4}{\exp \left( \left\| \mathbf{w} _i-\mathbf{x} \right\| _{2}^{2} \right) }\right] (\mathbf{w} _{i} - \mathbf{x} ). \end{aligned}$$

(40)

Finally, the Cauchy kernel function leads, respectively, to the following cost function and gradient vector:

$$\begin{aligned} J_i(\mathbf{x} )= & {} \left( 1 + \frac{\left\| \mathbf{x} - \mathbf{x} \right\| ^2}{\gamma ^2} \right) ^{-1} -2\left( 1 + \frac{\left\| \mathbf{w} _i - \mathbf{x} \right\| ^2}{\gamma ^2} \right) ^{-1} + \left( 1 + \frac{\left\| \mathbf{w} _i - \mathbf{w} _i \right\| ^2}{\gamma ^2} \right) ^{-1} \end{aligned}$$

(41)

$$\begin{aligned}= & {} 2 - 2\left( 1 + \frac{\left\| \mathbf{w} _i - \mathbf{x} \right\| ^2}{\gamma ^2} \right) ^{-1} = 2 - 2\left( \frac{\gamma ^2}{\gamma ^2 + \left\| \mathbf{w} _i - \mathbf{x} \right\| ^2} \right) , \end{aligned}$$

(42)

and

$$\begin{aligned}&\nabla J_{i}(\mathbf{x} ) = - 2\gamma ^2\left[ -\frac{1}{\left( \gamma ^2 + \left\| \mathbf{w} _i - \mathbf{x} \right\| ^2 \right) ^2} \right] 2\left( \mathbf{w} _i - \mathbf{x} \right) \nonumber \\&\quad = \left[ \frac{4\gamma ^2}{\left( \gamma ^2 + \left\| \mathbf{w} _i - \mathbf{x} \right\| ^2 \right) ^2} \right] \left( \mathbf{w} _i - \mathbf{x} \right) \end{aligned}$$

(43)

As one can inferred from the Eqs. (35), (38) and (42), the minimum value of these cost functions $J_i \left( \mathbf {x} \right)$ occurs when the squared Euclidean distance $\left\| \mathbf{w} _i-\mathbf{x} \right\| _{2}^{2}$ is minimum. Furthermore, for fixed hyperparameters, the gradient vectors resulting from these cost functions are proportional to $\left( \mathbf{w} _i - \mathbf{x} \right)$, differing only by factors that depend on $\left\| \mathbf{w} _i-\mathbf{x} \right\| _{2}^{2}$. For a given iteration, these factors are constant. This feature then motivated us to use a common learning rule in Eq. (20) to update the prototypes whenever we use one these kernel functions.