Stream sampling over windows with worst-case optimality and $\ell$-overlap independence

Abstract

Sampling provides fundamental support to numerous applications that cannot afford to materialize all the objects arriving at a rapid speed. Existing stream sampling algorithms guarantee small space and query overhead, but all require worst-case update time proportional to the number of samples. This creates a performance issue when a large sample set is required. In this paper, we propose a new sampling algorithm that is optimal simultaneously in all the three aspects: space, query time, and update time. In particular, the algorithm handles an update in O(1) worst-case time with a very small hidden constant. Our algorithm also ensures a strong independence guarantee: the sample sets of all the queries are mutually independent as long as the overlap between two query windows is small.

Appendices

Appendix 1: Space lower bound for our problem of Sect. 2 under disjoint independence

We will need the following mathematical fact:

Lemma 11

Let x, y be any positive real values satisfying $x \ge y$ and $x \ge 1$. Then $1 - (1 - 1/x)^y = \varOmega (y/x)$.

Proof

It is fundamental to verify that, for any real value $z, 1 + z \le e^z$, and for any real value $z \in [0, 1], e^{-z} \le 1 - (1-1/e) z$. Therefore:

$$\begin{aligned} 1 - (1 - 1/x)^y\ge & {} 1 - e^{-y/x} \\\ge & {} 1 - \Big (1 - \Big (1 - \frac{1}{e}\Big ) \frac{y}{x}\Big ) = \Big (1 - \frac{1}{e}\Big ) \frac{y}{x}. \end{aligned}$$

$\square$

Let $\mathcal {A}_3$ be an algorithm solving our problem under $\ell = 0$. Suppose that $n \ge r$ stream elements have been received. Consider the i-th element $e_i$ where $i \in [1, n - r]$. Define a random variable $E_i$ to be 1 if $e_i$ is retained by $\mathcal {A}_3$ at this moment, or 0 otherwise. Motivated by Gemulla and Lehner, we look at the query with parameter $w = n - i + 1$. As each WR sample of the query picks $e_i$ with probability $1/w, e_i$ is picked by at least one of its r samples with probability $1 - (1 - 1/w)^r$. It thus follows that

$$\begin{aligned} \mathbf {Pr}[E_i = 1]\ge & {} 1 - \left( 1 - \frac{1}{n - i +1} \right) ^r \\ \text {(by Lemma 11)}= & {} \varOmega \left( \frac{r}{n-i}\right) . \end{aligned}$$

Hence, the expected space used by $\mathcal {A}_3$ is at least

$$\begin{aligned} \sum _{i=1}^{n-r} \mathbf {E}[E_i] = \sum _{i=1}^{n-r} \varOmega \left( \frac{r}{n-i}\right) = \varOmega (r \log (n/r)). \end{aligned}$$

The worst-case space of $\mathcal {A}_3$ cannot be smaller, and thus, must also be $\varOmega (r \log (n/r))$.

Appendix 2: Proof of Lemma 10

The lemma is trivial if $Z + 1 > y$; next, we assume $Z + 1 \le y$.

Consider first $i = 2$. Let $b_1, b_2$ be the level-1 buckets covered by b. All the elements in $b_1, b_2$ are directly retained. The lemma holds on b because Lemma 2 obtains $R_b[1]$ with a single random number generated after the entire $b_2$ has been received, i.e., at or after $n = y \ge Z + 1$.

Consider $i = j \ge 3$. Redefine $b_1, b_2$ as the level-($i-1$) buckets covered by b, whose size-1 sample sets are $R_{b_1}, R_{b_2}$, respectively. Inductively assume that the lemma holds on $R_{b_1}(Z)$ and $R_{b_2}(Z)$. Lemma 2 generates a random number, at or after $n = y$, to decide whether $R_b[1]$ equals $R_{b_1}[1]$ or $R_{b_2}[1]$. Hence, given the number, $R_b(Z)$ is fully determined by $R_{b_1}(Z)$ and $R_{b_2}(Z)$.