Skip to main content
SpringerLink
Log in

Tracking set-expression cardinalities over continuous update streams

  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract.

There is growing interest in algorithms for processing and querying continuous data streams (i.e., data seen only once in a fixed order) with limited memory resources. In its most general form, a data stream is actually an update stream, i.e., comprising data-item deletions as well as insertions. Such massive update streams arise naturally in several application domains (e.g., monitoring of large IP network installations or processing of retail-chain transactions). Estimating the cardinality of set expressions defined over several (possibly distributed) update streams is perhaps one of the most fundamental query classes of interest; as an example, such a query may ask “what is the number of distinct IP source addresses seen in passing packets from both router R 1 and R 2 but not router R 3?”. Earlier work only addressed very restricted forms of this problem, focusing solely on the special case of insert-only streams and specific operators (e.g., union). In this paper, we propose the first space-efficient algorithmic solution for estimating the cardinality of full-fledged set expressions over general update streams. Our estimation algorithms are probabilistic in nature and rely on a novel, hash-based synopsis data structure, termed ”2-level hash sketch”. We demonstrate how our 2-level hash sketch synopses can be used to provide low-error, high-confidence estimates for the cardinality of set expressions (including operators such as set union, intersection, and difference) over continuous update streams, using only space that is significantly sublinear in the sizes of the streaming input (multi-)sets. Furthermore, our estimators never require rescanning or resampling of past stream items, regardless of the number of deletions in the stream. We also present lower bounds for the problem, demonstrating that the space usage of our estimation algorithms is within small factors of the optimal. Finally, we propose an optimized, time-efficient stream synopsis (based on 2-level hash sketches) that provides similar, strong accuracy-space guarantees while requiring only guaranteed logarithmic maintenance time per update, thus making our methods applicable for truly rapid-rate data streams. Our results from an empirical study of our synopsis and estimation techniques verify the effectiveness of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Alon N, Gibbons PB, Matias Y, Szegedy M (1999) Tracking join and self-join sizes in limited storage. In: Proceedings of the 18th ACM SIGACT-SIGMOD-SIGART symposium on principles of database systems, Philadeplphia

  2. Alon N, Matias Y, Szegedy M (1996) The space complexity of approximating the frequency moments. In: Proceedings of the 28th annual ACM symposium on the theory of computing, Philadelphia, pp 20-29

  3. Alon N, Spencer JH (1992) The probabilistic method. Wiley, New York

  4. Bar-Yossef Z, Jayram TS, Kumar R, Sivakumar D, Trevisan L (2002) Counting distinct elements in a data stream. In: Proceedings of the 6th international workshop on randomization and approximation techniques in computer science (RANDOM’02), Cambridge, MA

  5. Broder AZ, Charikar M, Frieze AM, Mitzenmacher M (1998) Min-wise independent permutations. In: Proceedings of the 30th ACM symposium on the theory of computing, pp 327-336

  6. Charikar M, Chaudhuri S, Motwani R, Narasayya V (2000) Towards estimation error guarantees for distinct values. In: Proceedings of the 19th ACM SIGACT-SIGMOD-SIGART symposium on principles of database systems, Dallas, TX

  7. Charikar M, Chen K, Farach-Colton M (2002) Finding frequent items in data streams. In: Proceedings of the international colloquium on automata, languages, and programming, Malaga, Spain

  8. Chen Z, Korn F, Koudas N, Muthukrishnan S (2000) Selectivity estimation for Boolean queries. In: Proceedings of the 19th ACM SIGACT-SIGMOD-SIGART symposium on principles of database systems, Dallas, TX

  9. Cohen E (1997) Size-estimation framework with applications to transitive closure and reachability. J Comput Sys Sci 55(3):441-453

    Article  MathSciNet  MATH  Google Scholar 

  10. Cormen TH, Leiserson CE, Rivest RL (1990) Introduction to algorithms. MIT Press, Cambridge, MA

  11. Dobra A, Garofalakis M, Gehrke J, Rastogi R (2002) Processing complex aggregate queries over data streams. In: Proceedings of the 2002 ACM SIGMOD international conference on management of data, Madison, WI, pp 61-72

  12. Feigenbaum J, Kannan S, Strauss M, Viswanathan M (1999) An approximate L1-difference algorithm for massive data streams. In: Proceedings of the 40th IEEE symposium on foundations of computer science, pp 501-511

  13. Feigenbaum J, Kannan S, Strauss M, Viswanathan M (1999) An approximate L 1-difference algorithm for massive data streams. In: Proceedings of the 40th annual IEEE symposium on foundations of computer science, New York

  14. Flajolet P, Martin GN (1985) Probabilistic counting algorithms for data base applications. J Comput Sys Sci 31:182-209

    Article  MathSciNet  MATH  Google Scholar 

  15. Ganguly S, Garofalakis M, Rastogi R (2003) Processing set expressions over continuous update streams. In: Proceedings of the 2003 ACM SIGMOD international conference on management of data, San Diego, CA

  16. Garofalakis M, Gibbons PB (2001) Approximate query processing: taming the terabytes. Tutorial in: 27th international conference on very large data bases, Rome, Italy

    Google Scholar 

  17. Garofalakis M, Gehrke J, Rastogi R (2002) Querying and mining data streams: you only get one look. Tutorial in: 28th international conference on very large data bases, Hong Kong

    Google Scholar 

  18. Gibbons PB (2001) Distinct sampling for highly-accurate answers to distinct values queries and event reports. In: Proceedings of the 27th international conference on very large data bases, Rome, Italy

  19. Gibbons PB, Tirthapura S (2001) Estimating simple functions on the union of data streams. In: Proceedings of the 13th annual ACM symposium on parallel algorithms and architectures, Crete Island, Greece

  20. Gilbert AC, Kotidis Y, Muthukrishnan S, Strauss MJ (2001) Surfing wavelets on streams: one-pass summaries for approximate aggregate queries. In: Proceedings of the 27th international conference on very large data bases, Rome, Italy

  21. Gilbert AC, Kotidis Y, Muthukrishnan S, Strauss MJ (2002) How to summarize the universe: dynamic maintenance of quantiles. In: Proceedings of the 28th international conference on very large data bases, pp 454-465, Hong Kong

  22. Greenwald M, Khanna S (2001) Space-efficient online computation of quantile summaries. In: Proceedings of the 2001 ACM SIGMOD international conference on management of data, Santa Barbara, CA

  23. Haas PJ, Naughton JF, Seshadri S, Stokes L (1995) Sampling-based estimation of the number of distinct values of an attribute. In: Proceedings of the 21st international conference on very large data bases, pp 311-322

  24. Indyk P (1999) A small approximately min-wise independent family of hash functions. In: Proceedings of the 10th annual ACM-SIAM symposium on discrete algorithms, Baltimore, MD, pp 454-456

  25. Indyk P (2000) Stable distributions, pseudorandom generators, embeddings and data stream computation. In: Proceedings of the 41st annual IEEE symposium on foundations of computer science, Redondo Beach, CA, pp 189-197

  26. Kalyanasundaram B, Schnitger G (1992) The probabilistic communication complexity of set intersection. SIAM J Discrete Math 5(4):545-557

    MathSciNet  MATH  Google Scholar 

  27. Kushilevitz E, Nisan N (1997) Communication complexity. Cambridge University Press, Cambridge, UK

  28. Singh Manku G, Motwani R (2002) Approximate frequency counts over data streams. In: Proceedings of the 28th international conference on very large data bases, Hong Kong, pp 346-357

  29. Melton J, Simon AR (1993) Understanding the new SQL: a complete guide. Morgan Kaufmann, San Francisco

    Google Scholar 

  30. Motwani R, Raghavan P (1995) Randomized algorithms. Cambridge University Press, Cambridge, UK

  31. Thaper N, Guha S, Indyk P, Koudas N (2002) Dynamic multidimensional histograms. In: Proceedings of the 2002 ACM SIGMOD international conference on management of data, Madison, WI, pp 428-439

Download references

Author information

Authors and Affiliations

  1. Bell Laboratories, Lucent Technologies, 600 Mountain Avenue, NJ 07974, Murray Hill, USA

    Sumit Ganguly, Minos Garofalakis & Rajeev Rastogi

Authors
  1. Sumit Ganguly
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Minos Garofalakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rajeev Rastogi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sumit Ganguly.

Additional information

Received: 20 October 2003, Accepted: 16 April 2004, Published online: 14 September 2004

Edited by: J. Gehrke and J. Hellerstein.

Sumit Ganguly: sganguly@cse.iitk.ac.in Current affiliation: Department of Computer Science and Engineering, Indian Institute of Technology, Kanpur, India

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ganguly, S., Garofalakis, M. & Rastogi, R. Tracking set-expression cardinalities over continuous update streams. VLDB 13, 354–369 (2004). https://doi.org/10.1007/s00778-004-0135-3

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-004-0135-3

Keywords:

Search

Navigation