A Tight Lower Bound for High Frequency Moment Estimation with Small Error

Abstract

We show an Ω((n ^{1 − 2/p} logM)/ε ²) bits of space lower bound for (1 + ε)-approximating the p-th frequency moment $F_p = \|x\|_p^p = \sum_{i=1}^n |x_i|^p$ of a vector x ∈ { − M, − M + 1, …, M}ⁿ with constant probability in the turnstile model for data streams, for any p > 2 and ε ≥ 1/n ^1/p (we require ε ≥ 1/n ^1/p since there is a trivial O(n logM) upper bound). This lower bound matches the space complexity of an upper bound of Ganguly for any ε < 1/log^O(1) n, and is the first of any bound in the long sequence of work on estimating F _p to be shown to be optimal up to a constant factor for any setting of parameters. Moreover, our technique improves the dependence on ε in known lower bounds for cascaded moments, also known as mixed norms. We also continue the study of tight bounds on the dimension of linear sketches (drawn from some distribution) required for estimating F _p over the reals. We show a dimension lower bound of Ω(n ^{1 − 2/p}/ε ²) for sketches providing a (1 + ε)-approximation to $\|x\|_p^p$ with constant probability, for any p > 2 and ε ≥ 1/n ^1/p. This is again optimal for ε < 1/log^O(1) n.