Concept-evolution detection in non-stationary data streams: a fuzzy clustering approach

Abstract

We have entered the era of networked communications where concepts such as big data and social networks are emerging. The explosion and profusion of available data in a broad range of application domains cause data streams to become an inevitable part of the most real-world applications. In the classification of data streams, there are four major challenges: infinite length, concept drift, recurring and evolving concepts. This paper proposes a novel method to address the mentioned challenges with a focus on the last one. Unlike the existing methods for detection of evolving concepts, we cast joint classification and detection of evolving concepts into optimizing an objective function by extending a fuzzy agglomerative clustering method. Moreover, rather than keeping instances or hyper-sphere summaries of previously seen classes, we just maintain boundaries in the kernel space and generate instances of each class on demand. This approach enhances the accuracy and reduces the memory usage of the proposed method. We empirically evaluated and showed the effectiveness of the proposed approach on several synthetic and real datasets. Experimental results on synthetic and real datasets show the superiority of the proposed method over the related state-of-the-art methods in this area.

Appendix

In this appendix, we give the proof of Theorem 1. The goal of the optimization procedure is to simultaneously find fuzzy memberships U and cluster centers Z such that the objective function given in Eq. (11) is minimized. Onclad adopts an alternating optimization approach to minimize $J_{\textsc {Onclad}}$. Minimizing $J_{\textsc {Onclad}}$ with the constraints is a kind of constrained nonlinear optimization problem. We use Lagrange multipliers, and then the following Lagrange function is obtained.

$$\begin{aligned} J_{\textsc {Onclad}}(U,Z)= & {} \sum \limits _{j=1}^{K}\sum \limits _{i=1}^{N}u_{ij}d_{ij} + \gamma \sum \limits _{j=1}^{K}\sum \limits _{i=1}^{N}u_{ij}\log u_{ij}\nonumber \\&+\, \alpha \sum \limits _{m|P_m \in C^i}\sum \limits _{\begin{array}{c} n| P_n \in C^i \\ m\ne n \end{array}}\sum \limits _{k=1}^{K}\sum \limits _{\begin{array}{c} l=1 \\ l\ne k \end{array}}^{K}u_{mk}u_{nl}\nonumber \\&+\, \beta \sum \limits _{m|P_m \in C^i}\sum \limits _{\begin{array}{c} n| P_n \in C^j \\ i\ne j \end{array}}\sum \limits _{k=1}^{K}u_{mk}u_{nk}\nonumber \\&+\, \sum \limits _{i=1}^{n}{\lambda }_i \left( \sum \limits _{j=1}^{K}u_{ij}-1\right) \end{aligned}$$

(11)

This objective function is minimized using alternating optimization approach. First, we fix fuzzy memberships U and minimize the objective function with respect to Z and then we fix cluster centers Z and minimize the objective function with respect to U. Optimizing the cluster centers $Z \equiv [z_{jl}]_{K \times m}$ and the fuzzy memberships $U \equiv [u_{ij}]_{K \times m}$ obtained by the following lemmas.

Lemma 1

Given the fuzzy memberships U are fixed, the optimal values for the cluster centers $Z \equiv [z_{jl}]_{K \times m}$ are obtained using following equation:

$$\begin{aligned} z_{jl} = \dfrac{\sum \nolimits _{i=1}^{N} u_{ij} x_{il}}{\sum \nolimits _{i=1}^{N} u_{ij}}. \end{aligned}$$

(12)

Proof

By taking derivative of Eq. (11) with respect to each cluster center and setting it to zero, we obtain:

$$\begin{aligned} \dfrac{\partial J(U,\mathbf Z )}{\partial z_{jl}} = \sum \limits _{i=1}^{N}2u_{ij}(z_{jl}-x_{il})=0 \end{aligned}$$

(13)

Thus, the solution for $z_{jl}$ equals to

$$\begin{aligned} z_{jl} = \dfrac{\sum \nolimits _{i=1}^{N} u_{ij} x_{il}}{\sum \nolimits _{i=1}^{N} u_{ij}}, \end{aligned}$$

(14)

which completes the proof of lemma. $\square$

Lemma 2

Given the cluster centers Z are fixed, the optimal value of fuzzy memberships are equal to:

$$\begin{aligned} u_{ij}= \dfrac{\exp \left( \dfrac{-d_{ij}}{\gamma }\right) \exp \left( \dfrac{-\alpha A_{ij}}{\gamma }\right) \exp \left( \dfrac{-\beta B_{ij}}{\gamma }\right) }{\sum \nolimits _{l=1}^{K} \exp \left( \dfrac{-d_{il}}{\gamma }\right) \exp \left( \dfrac{-\alpha A_{il}}{\gamma }\right) \exp \left( \dfrac{-\beta B_{il}}{\gamma }\right) } \end{aligned}$$

(15)

Proof

Taking derivative of Eq. (11) with respect to each fuzzy membership and setting it to zero, we obtain:

$$\begin{aligned} \dfrac{\partial J(\mathbf U ,Z)}{\partial u_{ij}}= & {} d_{ij} + \gamma (1 + \log u_{ij}) + \alpha \left( \underbrace{\sum \limits _{\begin{array}{c} n|P_i \in C^m, P_n \in C^l \\ m\ne l \end{array}} \sum \limits _{\begin{array}{c} k=1 \\ k\ne j \end{array}}^{K} u_{nk}}_{A_{ij}}\right) \nonumber \\&+\, \beta \left( \underbrace{\sum \limits _{n|P_i,P_n \in C^m}u_{nj}}_{B_{ij}}\right) + \lambda _i=0 \end{aligned}$$

(16)

Solving the above equation for $u_{ij}$, we obtain:

$$\begin{aligned} u_{ij}= \exp (-1)\exp \left( \dfrac{-d_{ij}}{\gamma }\right) \exp \left( \dfrac{-\alpha A_{ij}}{\gamma }\right) \exp \left( \dfrac{-\beta B_{ij}}{\gamma }\right) \exp \left( \dfrac{-\lambda _i}{\gamma }\right) \end{aligned}$$

(17)

Because of the constraint $\sum \nolimits _{j=1}^{K} u_{ij}=1$, the Lagrange multipliers are equal to

$$\begin{aligned} \sum \limits _{j=1}^{K} u_{ij}= & {} \sum \limits _{j=1}^{K} \exp (-1)\exp \left( \dfrac{-d_{ij}}{\gamma }\right) \exp \left( \dfrac{-\alpha A_{ij}}{\gamma }\right) \exp \left( \dfrac{-\beta B_{ij}}{\gamma }\right) \exp \left( \dfrac{-\lambda _i}{\gamma }\right) \nonumber \\= & {} \exp (-1)\exp \left( \dfrac{-\lambda _i}{\gamma }\right) \sum \limits _{j=1}^{K} \exp \left( \dfrac{-d_{ij}}{\gamma }\right) \exp \left( \dfrac{-\alpha A_{ij}}{\gamma }\right) \exp \left( \dfrac{-\beta B_{ij}}{\gamma }\right) = 1\nonumber \\ \end{aligned}$$

(18)

By some algebraic simplification, we obtain:

$$\begin{aligned} \exp \left( \dfrac{-\lambda _i}{\gamma }\right) = \dfrac{1}{\exp (-1) \sum \nolimits _{j=1}^{K} \exp \left( \dfrac{-d_{ij}}{\gamma }\right) \exp \left( \dfrac{-\alpha A_{ij}}{\gamma }\right) \exp \left( \dfrac{-\beta B_{ij}}{\gamma }\right) } \end{aligned}$$

(19)

By substituting Eq. (19) in Eq. (17), we obtain the closed form solution for the optimal memberships as

$$\begin{aligned} u_{ij}= \dfrac{\exp \left( \dfrac{-d_{ij}}{\gamma }\right) \exp \left( \dfrac{-\alpha A_{ij}}{\gamma }\right) \exp \left( \dfrac{-\beta B_{ij}}{\gamma }\right) }{\sum \nolimits _{l=1}^{K} \exp \left( \dfrac{-d_{il}}{\gamma }\right) \exp \left( \dfrac{-\alpha A_{il}}{\gamma }\right) \exp \left( \dfrac{-\beta B_{il}}{\gamma }\right) }, \end{aligned}$$

(20)

which completes the proof of lemma. $\square$

The following lemma shows the convergence of the alternating minimization procedure by updating Z and U using Eqs. (12) and (15), respectively.

Lemma 3

Let J(Z) be $J_{\textsc {Onclad}}$ where fuzzy memberships are fixed, let J(U) be $J_{\textsc {Onclad}}$ where cluster centers are fixed and $\alpha , \beta , \gamma >0$. Z and U are local optimum of $J_{\textsc {Onclad}}$ if $z_{ij}$ and $u_{ij}$ are calculated using Eqs. (12) and (15), respectively.

Proof

The necessity has been proven in Lemmas 1 and 2. In order to prove their sufficiency, the Hessian matrices H(J(Z)) of J(Z) and H(J(U)) of J(U) are obtained as follows.

$$\begin{aligned} h_{fg,il}(J(Z))= & {} \dfrac{\partial }{\partial _{fg}}\left[ \dfrac{\partial J(Z)}{\partial z_{il}}\right] = {\left\{ \begin{array}{ll} \sum \nolimits _{j=1}^{K} 2u_{ij}, &{}\quad \textit{if}\, f=i, g=l\\ 0, &{} \quad \textit{otherwise}\\ \end{array}\right. } \end{aligned}$$

(21)

$$\begin{aligned} h_{fg,ij}(J(U))= & {} \dfrac{\partial }{\partial _{fg}}\left[ \dfrac{\partial J(U)}{\partial u_{ij}}\right] = {\left\{ \begin{array}{ll} \dfrac{\gamma }{u_{ij}}, &{} \quad \textit{if}\, f=i, g=j\\ 0, &{}\quad \textit{otherwise}\\ \end{array}\right. } \end{aligned}$$

(22)

According to these equations, H(J(Z)) and H(J(U)) are diagonal matrices. Also, it is mentioned that $u_{ij}\in (0,1]$ and $\gamma >0$. Hence, the Hessian matrices are positive definite, and Eqs. (12) and (15) are the sufficient conditions to minimize J(Z) and J(U), respectively. $\square$

Proof of Theorem 1

The necessary conditions for $J_{\textsc {Onclad}}$ to attain its local minimum were proven in Lemmas 1 and 2. According to Lemma 3, $J_{\textsc {Onclad}}(U^{(t+1)},Z^{(t+1)}) \le J_{\textsc {Onclad}}(U^{(t)},Z^{(t)})$, and the convergence to the local minima is proved.$\square$