Incremental Weighted Naive Bays Classifiers for Data Stream

Abstract

A naive Bayes classifier is a simple probabilistic classifier based on applying Bayes’ theorem with naive independence assumption. The explanatory variables (X _i ) are assumed to be independent from the target variable (Y ). Despite this strong assumption this classifier has proved to be very effective on many real applications and is often used on data stream for supervised classification. The naive Bayes classifier simply relies on the estimation of the univariate conditional probabilities P(X _i  | C). This estimation can be provided on a data stream using a “supervised quantiles summary.” The literature shows that the naive Bayes classifier can be improved (1) using a variable selection method (2) weighting the explanatory variables. Most of these methods are related to batch (off-line) learning and need to store all the data in memory and/or require reading more than once each example. Therefore they cannot be used on data stream. This paper presents a new method based on a graphical model which computes the weights on the input variables using a stochastic estimation. The method is incremental and produces a Weighted Naive Bayes Classifier for data stream. This method will be compared to classical naive Bayes classifier on the Large Scale Learning challenge datasets.

Appendix: Derivative of the Cost Function

The graphical model is built to have directly the values of the P(C _k  | X) at the output. The goal is to maximize the likelihood and therefore to minimize the negative log likelihood. The first step in the calculation is to decompose the softmax considering that each output could be seen as the succession of two steps: an activation followed by a function of this activation.

Here the activation function could be seen as: $O_{k} = f(H_{k}) =\mathrm{ exp}(H_{k})$ and the output of the softmax part of our graphical model is: $P_{k} = \frac{O_{k}} {\sum _{j=1}^{K}O_{j}}$. The derivative of the activation function is:

$$\displaystyle{ \frac{\partial O_{k}} {\partial H_{k}} = f'(H_{k}) =\mathrm{ exp}(H_{k}) = O_{k} }$$

(5)

The cost function being the −log likelihood, we have to consider two cases: (1) the desired output is equal to 1 or (2) the desired output is equal to 0. For the following we note:

$$\displaystyle{ \frac{\partial \mathrm{Cost}} {\partial H_{k}} = \frac{\partial C} {\partial P_{k}} \frac{\partial P_{k}} {\partial O_{k}} \frac{\partial O_{k}} {\partial H_{k}} }$$

(6)

In the case where the desired output of the output k is equal to 1 by replacing (5) in (6):

$$\displaystyle{ \frac{\partial \mathrm{Cost}} {\partial H_{k}} = \frac{\partial C} {\partial P_{k}} \frac{\partial P_{k}} {\partial O_{k}} \frac{\partial O_{k}} {\partial H_{k}} = \frac{-1} {P_{k}} \frac{\partial P_{k}} {\partial O_{k}}O_{k} }$$

(7)

$$\displaystyle\begin{array}{rcl} \frac{\partial \mathrm{Cost}} {\partial H_{k}} & =& \frac{-1} {P_{k}}\left [\sum _{l=1,l\neq k}^{K}\left ( \frac{O_{l}} {\big(\sum _{j=1}^{K}O_{j}\big)^{2}}\right )\right ]O_{k} \\ & =& \frac{-1} {P_{k}}\left [\frac{\big(\sum _{j=1}^{K}O_{j}\big) - O_{k}} {\big(\sum _{j=1}^{K}O_{j}\big)^{2}} \right ]O_{k} {}\end{array}$$

(8)

$$\displaystyle\begin{array}{rcl} \frac{\partial \mathrm{Cost}} {\partial H_{k}} & =& \frac{-1} {P_{k}}\left [\frac{\big(\sum _{j=1}^{K}O_{j}\big) - O_{k}} {\big(\sum _{j=1}^{K}O_{j}\big)} \right ] \frac{O_{k}} {\big(\sum _{j=1}^{K}O_{j}\big)} \\ & =& \frac{-1} {P_{k}}\left [1 - \frac{O_{k}} {\big(\sum _{j=1}^{K}O_{j}\big)}\right ] \frac{O_{k}} {\big(\sum _{j=1}^{K}O_{j}\big)} {}\end{array}$$

(9)

Therefore

$$\displaystyle{ \frac{\partial \mathrm{Cost}} {\partial H_{k}} = \frac{-1} {P_{k}}[1 - P_{k}]P_{k} = P_{k} - 1 }$$

(10)

In the case where the desired output of the output k is equal to 0 the error is only transmitted by the normalization part of the softmax function since the derivative for an output where the desired value is 0 is equal to 0. Therefore with similar steps we have: $\frac{\partial \mathrm{Cost}} {\partial H_{k}} = P_{k}$

Finally we conclude: $\frac{\partial \mathrm{Cost}} {\partial H_{k}} = P_{k} - T_{k},\forall k$ where T _k is the desired probability and P _k the estimated probability. Then the rest of the calculation of $\frac{\partial \mathrm{Cost}} {\partial w_{\mathit{ik}}}$ is straightforward.