Multi-label incremental learning applied to web page categorization

Abstract

Multi-label problems are challenging because each instance may be associated with an unknown number of categories, and the relationship among the categories is not always known. A large amount of data is necessary to infer the required information regarding the categories, but these data are normally available only in small batches and distributed over a period of time. In this work, multi-label problems are tackled using an incremental neural network known as the evolving Probabilistic Neural Network (ePNN). This neural network is capable of continuous learning while maintaining a reduced architecture, so that it can always receive training data when available with no drastic growth of its structure. We carried out a series of experiments on web page data sets and compared the performance of ePNN to that of other multi-label categorizers. On average, ePNN outperformed the other categorizers in four out of five metrics used for evaluation, and the structure of ePNN was less complex than that of the other algorithms evaluated.

Appendix

Suppose that a training instance x is presented to the neural network and the outputs of all components of each GMM in the neural network are calculated.

If the component s is the most activated, and it is not in the GMM assigned to the class of x, then it is desirable to reduce the output value of s and increase the output value of the component r, which is the most activated in the GMM assigned to the class of x. In other words, if

$$ f_r(x,\mu_{r},\Upsigma_{r},\varphi_{r}) < f_s(x,\mu_{s},\Upsigma_{s},\varphi_{s}), $$

then it is desired to find new values of the receptive field sizes of the components s ($\varphi_{s_{\rm new}}$) and r ($\varphi_{r_{\rm new}}$), such that

$$ f_r(x,\mu_{r},\Upsigma_{r},\varphi_{r_{\rm new}}) \geq f_s(x,\mu_{s},\Upsigma_{s},\varphi_{s_{\rm new}}). $$

(17)

The value of $\varphi_{r_{\rm new}}$ is computed using Eq. (6). To obtain the value of $\varphi_{s_{\rm new}}$, a constant η, 0 < η ≤ 1, should be added to Eq. (17) to achieve equality

$$ \eta f_r(x,\mu_{r},\Upsigma_{r},\varphi_{r_{\rm new}}) = f_s(x,\mu_{s},\Upsigma_{s},\varphi_{s_{\rm new}}). $$

Therefore, from Eq. (2) one obtains

$$ \eta f_r(x,\mu_{r},\Upsigma_{r},\varphi_{r_{\rm new}}) = \frac{1}{\sqrt{2\pi}\varphi_{s_{\rm new}}}\exp\left(\frac{D_s}{\varphi_{s_{\rm new}}^{2}}\right), $$

where D _s  = x′^T μ′ _s  − 1.

To solve this equation, a Taylor expansion is applied to the exponential function to linearize it. Therefore,

$$ \eta f_r(x,\mu_{r},\Upsigma_{r},\varphi_{r_{\rm new}}) = \frac{1}{\sqrt{2\pi}{\varphi_{s_{\rm new}}}} \left[\exp\left(\frac{D_s}{\varphi_{s}^2}\right) - 2\frac{D_s}{\varphi_{s}^{3}} \exp\left(\frac{D_s}{\varphi_{s}^2}\right)(\varphi_{s_{\rm new}} - \varphi_{s}) \right]. $$

The Taylor expansion is valid for small values of $\epsilon = \varphi_{s_{\rm new}} - \varphi_{s}$. After further manipulation, the value of $\varphi_{s_{\rm new}}$ can be obtained using Eq. (18),

$$ \varphi_{s_{\rm new}} = \frac{\varphi_{s} f_s(x,\mu_{s},\Upsigma_{s},\varphi_{s})(\varphi_{s}^{2} + 2D_s)} {\eta\varphi_{s}^{2}f_r(x,\mu_{r},\Upsigma_{r},\varphi_{r_{\rm new}}) + 2D_s f_s(x,\mu_{s},\Upsigma_{s},\varphi_{s})}. $$

(18)

To prevent the value of the receptive field size from being significantly altered when a single training instance is presented, two thresholds are employed. The first is used over the saturated linear function to limit the value and render the Taylor expansion applicable [parameter α in Eq. (7)]. The second threshold is used when the receptive field size is updated [parameter ρ in Eqs. (6)–(7)].