Summarizing numeric spatial data streams by trend cluster discovery

Abstract

Advances in pervasive computing and sensor technologies have paved the way for the explosive living ubiquity of geo-physical data streams. The management of the massive and unbounded streams of sensor data produced poses several challenges, including the real-time application of summarization techniques, which should allow the storage and query of this amount of georeferenced and timestamped data in a server with limited memory. In order to face this issue, we have designed a summarization technique, called SUMATRA, which segments the stream into windows, computes summaries window-by-window and stores these summaries in a database. Trend clusters are discovered as summaries of each window. They are clusters of georeferenced data which vary according to a similar trend along the window time horizon. Several compression techniques are also investigated to derive a compact, but accurate representation of these trends for storage in the database. A learning strategy to automatically choose the best trend compression technique is designed. Finally, an in-network modality for tree-based trend cluster discovery is investigated in order to achieve an efficacious aggregation schema which drastically reduces the number of bytes transmitted across the network and maintains a longer network lifespan. This schema is mapped onto the routing structure of a tree-based WSN topology. Experiments performed with several data streams of real sensor networks assess the summarization capability, the accuracy and the efficiency of the proposed summarization schema.

Appendices

Appendix 1 [Proposition 1]

The size of the window $W_i$ is $\sigma _I(n_i+1)+\sigma _F n_i w (bytes)$.

Proof

This proposition is proved by considering that to store $W_i$, it is essential to store the (integer) enumerative window code $i$, the (integer) identifiers of nodes which are active along the window and, for each node, the series of $w$ (float) measurements. Then,

$$\begin{aligned} \begin{array}{ccl} size(W_i)&{}=&{}size(i)+\displaystyle \sum _{p\in N_i}{size(p)}+ \displaystyle \sum _{p\in N_i}\displaystyle \sum _{t=1}^w{size(H_i[p][t])}\\ &{}=&{}{\sigma _I}+{\sigma _I n_i} + \sigma _F n_i w= \sigma _I(n_i+1) + \sigma _F n_i w \hbox { (bytes).} \end{array} \end{aligned}$$

(31)

Appendix 2 [Proposition 4]

Let $P_i$ be the trend cluster set computed from $W_i$, then the size of $P_i$ is $\sigma _{I}(n_i+p_i+1)+\sigma _F p_iw (bytes)$, where $size$(integer)= $\sigma _{I}$ and $size$(float)= $\sigma _F$.

Proof

As $P_i=\{[i,C_k,T_k]\ | \ k=1,2,\ldots ,p_i ]\}$, then:

$$\begin{aligned} \begin{array}{ccl} size(P_i)&{}=&{}size(i)+\displaystyle \sum _{k=1}^{p_i}{\left( size(k)+size([C_k,T_k])\right) }\\ &{}=&{}{\sigma _I}+\displaystyle \sum _{k=1}^{p_i}{\left( \sigma _I+size(C_k) +size( T_k )\right) .} \end{array} \end{aligned}$$

(32)

By replacing $size(C_k)$ as it is reported in Proposition 2 and $size(T_k)$ as it is reported in Proposition 3 in Eq. 32, $size(P_i)$ is equal to:

$$\begin{aligned} \begin{array}{ccl} size(P_i)&{}=&{}\sigma _I+\sum _{k=1}^{p_i}{(\sigma _I\sharp C_k+\sigma _F w)}\\ &{}=&{} \sigma _I+ \sigma _I p_i + \sigma _I n_i+ \sigma _F p_i w =\sigma _I(n_i+p_i+1)+\sigma _F p_i w \hbox { (bytes),} \end{array} \end{aligned}$$

(33)

where $\displaystyle \sum _{k=1}^{p_i}{\sharp C_k}=n_i$, as $C_1,\ldots C_{p_i}$ is a partitioning of $N_i$.

Appendix 3 [Proposition 5]

$P_i$ is a summarization of $W_i$ under the condition that ${p_i}/{n_i}<={w}/({{\sigma _I}/{\sigma _F}+w})$.

Proof

Proving Proposition 5 is equivalent to proving that, under the hypothesis that ${p_i}/{n_i}<{w}/({{\sigma _I}/{\sigma _F}+w})$, then $size(P_i)\le size(W_i)$. According to Propositions 1–4 we have that:

$$\begin{aligned} \begin{array}{ccl} size(P_i)&{}=&{}\sigma _I(n_i+p_i+1)+\sigma _F p_i w \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \hbox {[Proposition~1].}\\ &{}=&{} \sigma _I(n_i+1) +\sigma _I p_i + \sigma _F p_i w = \sigma _I(n_i+1) + p_i (\sigma _I+\sigma _F w) \\ &{}\mathbf \le &{} \sigma _I (n_i+1) + {(\sigma _F w n_i)}/{(\sigma _I+\sigma _F w)} (\sigma _I+w\sigma _F) \quad [p_i\le \frac{(\sigma _I w n_i)}{(\sigma _I+\sigma _F w)}]\\ &{}=&{} \sigma _I(n_i+1)+ w\sigma _F n_i = {size(W_i)} \quad \quad \quad \quad \quad \quad \quad \hbox {[Proposition~4].} \end{array}\nonumber \\ \end{aligned}$$

(34)

Appendix 4 [Proposition 6]

Let $P_i$ be the set of trend clusters computed in $W_i$. $W_i$ can be reconstructed from $P_i$ with an absolute error upper bound that is $\delta$.

Proof

Let $N_i$ be the set of nodes partitioned into the trend clusters of $P_i$, that is, $N_i=\displaystyle \bigcup _{[i,C_k,T_k]\in P_i} {C_k}$. Let $\pi _i: N_i \mapsto \mathbb R ^w$ be the stream reconstruction function which is associated to $P_i$ and is defined such that $\pi _i(u)= [v_{k_1}, \ldots , v_{k_w}]$, where $u\in C_k$, $T_k=[(1,v_{k_1}),\ldots ,(w,v_{k_w})]$ and $[i,C_k,T_k]\in P_i$. According to Definition 5, the triple $[i,C_k,T_k]\in P_i$ satisfies the property of polyline purity, hence:

$$\begin{aligned} \underbrace{\forall [i,C_k,T_k] \in P_i \ \forall u\in C_k}_{\forall u \in N_i} \ \forall t=1,\ldots ,w\ :|value(u,t)-v_{k_t}|\le \delta , \end{aligned}$$

(35)

where $value(u,t)$ is the measurement transmitted by $u$ at the $t$-th snapshot of the window. By considering that $\pi _i(u)[t]=v_{k_t}$, we can reformulate Eq. 35 in the same way, as follows:

$$\begin{aligned} \forall u\in N_i \ \forall t=1,\ldots ,w\ :\ |value(u,t)-\pi _i(u)[t]|\le \delta . \end{aligned}$$

(36)

Thus, we conclude that the similarity domain threshold $\delta$ represents an upper bound of the absolute error performed by $P_i$ to summarize $W_i$.