Elite: an elastic infrastructure for big spatiotemporal trajectories

Abstract

As the volumes of spatiotemporal trajectory data continue to grow at a rapid pace; a new generation of data management techniques is needed in order to be able to utilize these data to provide a range of data-driven services, including geographic-type services. Key challenges posed by spatiotemporal data include the massive data volumes, the high velocity with which the data are captured, the need for interactive response times, and the inherent inaccuracy of the data. We propose an infrastructure, Elite, that leverages peer-to-peer and parallel computing techniques to address these challenges. The infrastructure offers efficient, parallel update and query processing by organizing the data into a layered index structure that is logically centralized, but physically distributed among computing nodes. The infrastructure is elastic with respect to storage, meaning that it adapts to fluctuations in the storage volume, and with respect to computation, meaning that the degree of parallelism can be adapted to best match the computational requirements. Further, the infrastructure offers advanced functionality, including probabilistic simulations, for contending with the inaccuracy of the underlying data in query processing. Extensive empirical studies offer insight into properties of the infrastructure and indicate that it meets its design goals, thus enabling the effective management of big spatiotemporal data.

Appendices

Appendix 1: Algorithms

Appendix 2: Routing cost estimation

Lemma 1

The maximum routing cost of a three-dimensional torus of N nodes is approximately equal to $0.91 N^{\frac{1}{3}}$.

Proof

For any torus node n, it takes one hop to reach its six neighbors (first-order neighbors) and two hops to reach its 18 second order neighbors. The number of i-th order neighbors $a^i$ can be represented by [40]:

$$\begin{aligned} a_i = 2+ 4i^2 \end{aligned}$$

Suppose the furthest node on the torus takes m hops from node n. The total number of nodes N visited equals the summation of the number of 1-th to m-th order neighbors. We have:

$$\begin{aligned} \sum _{i=1}^{m} a_i = N \Rightarrow \sum _{i=1}^m 2+ 4i^2 = N \Rightarrow m \approx \left( \frac{3}{4}N\right) ^{\frac{1}{3}} \end{aligned}$$

Thus, the maximum routing cost equals to the distance to n’s m-th order neighbor, $0.91N^{\frac{1}{3}}$. $\square$

Lemma 2

The average routing cost of a three-dimensional torus of N nodes is approximately equal to $0.69 N^{\frac{1}{3}}$ hops.

Proof

Based on Lemma 1, the furthest node from n requires m hops. Then, the average number of hops is:

$$\begin{aligned} { avg\_number\_of\_hops }= & {} \frac{1}{N} \sum _{i=1}^{m} a_i \cdot i = \frac{1}{N} \sum _{i=1}^{m} 2i + 4 i^3\nonumber \\= & {} \frac{1}{N} \left( m^4+\frac{8}{3}m^3+2m^2+\frac{1}{3}m\right) \nonumber \\\approx & {} 0.69 N^{\frac{1}{3}} \end{aligned}$$

(6)

$\square$

Appendix 3: Estimation of h

Let us consider the cost estimation on torus node $T_i$. After the range search $Q_i \oplus d_{{ max }} \oplus U_{{ max }}$ (Step 6 in Algorithm 2), we get a set $C_i$ of candidate trajectories with the average length $\overline{{\mathcal {T}}_c\cdot \varDelta t}=\frac{1}{|C_i|} \sum _{{\mathcal {T}} \in C_i} {\mathcal {T}}\cdot \varDelta t$. According to Definition 5, the cost of STNNQ depends on the number of trajectories at each snapshot. To estimate that, we first assume the trajectories are uniformly distributed in the spatiotemporal region $Q_i \oplus d_{{ max }} \oplus U_{{ max }}$.

$$\begin{aligned} { \#~of~objects~per~snapshot } = \frac{ \overline{{\mathcal {T}}_c\cdot \varDelta t} \cdot |C_i| }{|Q_i\cdot \varDelta t|} \end{aligned}$$

(7)

We define the density $\rho$, as the number of objects per snapshot divided by the area of the filtering bound $\pi (d_{{ max }}+U_{{ max }})^2$.

Lemma 3

Assume a two-dimensional region S in the spatial domain ${\mathfrak {S}}$, where the points are uniformly distributed, and let $N(S) = m$ represent the fact that there are m points inside region S. The probability of $N(S) = m$ is given by:

$$\begin{aligned} P(N(S)=m) = \frac{\rho |S| e^{-\rho |S|m}}{m!} \end{aligned}$$

(8)

Proof

The probability that m points out of n objects are in S is:

$$\begin{aligned} P(N(S)=m)= \left( {\begin{array}{c}n\\ m\end{array}}\right) \left( \frac{|S|}{|{\mathfrak {S}}|}\right) ^m \left( 1-\frac{|S|}{|{\mathfrak {S}}|}\right) ^{n-m} \end{aligned}$$

The extreme form of the binomial distribution is a Poisson distribution. Let $\rho = \frac{n}{|{\mathfrak {S}}|}$. The above equation becomes:

$$\begin{aligned} P(N(S)=m) = \frac{(\rho |S|)^me^{-\rho |S|}}{m!} \end{aligned}$$

Then, the probability that there is at least one point in S is:

$$\begin{aligned} P(N \ge 1)= & {} \sum _{i=1}^{\infty } P(N = i) = \sum _{i=1}^{\infty } \frac{(\rho |S|)^ie^{-\rho |S|}}{i!}\\= & {} 1 - e^{-\rho |S|} \end{aligned}$$

$\square$

Then, we can infer that there is a nearest neighbor within the circular region S to the query point with a probability higher than $P^*$. In our implementation, we set $P^*$ to 0.9, which is reasonably large for S to contain the nearest neighbor.

$$\begin{aligned} |S| = -\frac{ln\left( 1-P^*\right) }{\rho } \end{aligned}$$

(9)

The number of candidate objects per snapshot is estimated as:

$$\begin{aligned} h = \rho (|S \oplus U_{{ max }}|). \end{aligned}$$

(10)

Appendix 4: Obtaining $d_{\mathrm{max}}$

In our system, we try a series of range queries $Q_i \oplus d \oplus U_{{ max }}$ to incrementally obtain $d_{{ max }}$, where $d=5, 10, 20\,\%, \cdots$ of torus node $T_i$’s spatial domain size. Upon collecting the candidate trajectory set $C_i$ by the range search parameterized with d, we test whether the union of these trajectories’ time spans can cover $Q_i$’s $\varDelta t$, i.e., to decide whether $\cup _{UT \in C}UT\cdot \varDelta t \supseteq Q_i\cdot \varDelta t$ is true. If true, it means that there always exists at least an object for each timestamp in $Q_i\cdot \varDelta t$. Therefore, current d is taken as $d_{{ max }}$, which is sufficiently large for not missing any possible candidate trajectories. Otherwise, we need to increase d incrementally and repeat the aforementioned process.