Fingerprints for highly similar streams

Abstract

We propose an approach for approximating the Jaccard similarity of two streams, $J(A,B)=\frac{|A\cap B|}{|A\cup B|}$, for domains where this similarity is known to be high. Our method is based on a reduction from Jaccard similarity to $F_2$ norm estimation, for which there exists a sketch that is efficient in terms of both size and compute time, which we augment by a sampling technique. Our approach offers an improvement in the fingerprint size that is quadratic in the degree of similarity between the streams. More precisely, to approximate the Jaccard similarity up to a multiplicative factor of ϵ with confidence δ, it suffices to take a fingerprint of size $O(\ln (\frac{1}{\delta })\frac{{(1-t)}^2}{{ϵ}^2}\log \frac{1}{1-t})$ where t is the known minimal Jaccard similarity between the streams. Further, computing our fingerprint can be done in time $O(1)$ per element in the stream.