research-article

Structure-aware sampling on data streams

Authors:

Graham Cormode,

Nick DuffieldAuthors Info & Claims

SIGMETRICS '11: Proceedings of the ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems

Pages 197 - 208

https://doi.org/10.1145/1993744.1993763

Published: 07 June 2011 Publication History

Abstract

The massive data streams observed in network monitoring, data processing and scientific studies are typically too large to store. For many applications over such data, we must obtain compact summaries of the stream. These summaries should allow accurate answering of post hoc queries with estimates which approximate the true answers over the original stream. The data often has an underlying structure which makes certain subset queries, in particular range queries, more relevant than arbitrary subsets. Applications such as access control, change detection, and heavy hitters typically involve subsets that are ranges or unions thereof.

Random sampling is a natural summarization tool, being easy to implement and flexible to use. Known sampling methods are good for arbitrary queries but fail to optimize for the common case of range queries. Meanwhile, specialized summarization algorithms have been proposed for rangesum queries and related problems. These can outperform sampling giving fixed space resources, but lack its flexibility and simplicity. Particularly, their accuracy degrades when queries span multiple ranges.

We define new stream sampling algorithms with a smooth and tunable trade-off between accuracy on range-sum queries and arbitrary subset-sum queries. The technical key is to relax requirements on the variance over all subsets to enable better performance on the ranges of interest. This boosts the accuracy on range queries while retaining the prime benefits of sampling, in particular flexibility and accuracy, with tail bounds guarantees. Our experimental study indicates that structure-aware summaries can drastically improve range-sum accuracy with respect to state-of-the-art stream sampling algorithms and outperform deterministic methods on range-sum queries and hierarchical heavy hitter queries.

References

[1]

C. Buragohain and S. Suri. Quantiles on streams. In Encyclopedia of Database Systems. Springer, 2009.

[2]

M. T. Chao. A general purpose unequal probability sampling plan. Biometrika, 69(3):653--656, 1982.

[3]

H. Chernoff. A measure of the asymptotic efficiency for test of a hypothesis based on the sum of observations. Annals of Math. Statistics, 23:493--509, 1952.

[4]

E. Cohen, G. Cormode and N. Duffield. Structure-aware sampling: Flexible and accurate summarization. arXiv:1102.5146, 2011.

[5]

E. Cohen, N. Duffield, H. Kaplan, C. Lund, and M. Thorup. Stream sampling for variance-optimal estimation of subset sums. In ACM-SIAM SODA, 2009.

Digital Library

[6]

E. Cohen, N. Duffield, H. Kaplan, C. Lund, and M. Thorup. Composable, Scalable, and Accurate Weight Summarization of Unaggregated Data Sets In VLDB, 2009.

Digital Library

[7]

E. Cohen, N. Duffield, C. Lund, M. Thorup, and H. Kaplan. Variance optimal sampling based estimation of subset sums. Tech. report arXiv:0803.0473v1 {cs.DS}, 2008.

[8]

G. Cormode and M. Hadjieleftheriou. Finding frequent items in data streams. In VLDB, 2008.

Digital Library

[9]

G. Cormode, F. Korn, S. Muthukrishnan, and D. Srivastava. Space- and time-efficient deterministic algorithms for biased quantiles over data streams. In ACM PODS, 2006.

Digital Library

[10]

G. Cormode, F. Korn, S. Muthukrishnan, and D. Srivastava. Finding hierarchical heavy hitters in streaming data. ACM Trans. Knowl. Discov. Data, 1(4):1--48, 2008.

Digital Library

[11]

R. Gandhi, S. Khuller, S. Parthasarathy, and A. Srinivasan. Dependent rounding and its applications to approximation algorithms. J. Assoc. Comput. Mach., 53(3):324--360, 2006.

Digital Library

[12]

J. Hájek. Sampling from a finite population. Marcel Dekker, 1981.

[13]

D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from a finite universe. J. Amer. Stat. Assoc., 47(260):663--685, 1952.

[14]

A. Panconesi and A. Srinivasan. Randomized distributed edge coloring via an extension of the Chernoff-Hoeffding bounds. SIAM J. Comput., 26(2):350--368, 1997.

Digital Library

[15]

N. Shrivastava, C. Buragohain, D. Agrawal, and S. Suri. Medians and beyond: new aggregation techniques for sensor networks. In ACM SenSys, 2004.

Digital Library

[16]

A. Srinivasan. Distributions on level-sets with applications to approximation algorithms. In IEEE FOCS. 2001.

Digital Library

[17]

M. Szegedy and M. Thorup. On the variance of subset sum estimation. In Proc. ESA, 2007.

Digital Library

[18]

Y. Tillé. Sampling Algorithms, Springer, 2006.

[19]

J. Vitter. Random sampling with a reservoir. ACM Trans. Math. Softw., 11(1):37--57, 1985.

Digital Library

[20]

Y. Zhang, S. Singh, S. Sen, N. Duffield, and C. Lund. Online identification of hierarchical heavy hitters. In SIGMETRICS, 2004.

Digital Library

Cited By

Patil MMagdy A(2021)LATEST: Learning-Assisted Selectivity Estimation Over Spatio-Textual Streams2021 IEEE 37th International Conference on Data Engineering (ICDE)10.1109/ICDE51399.2021.00142(1607-1618)Online publication date: Apr-2021
https://doi.org/10.1109/ICDE51399.2021.00142
Siddique AEldawy AHristidis V(2019)Comparing synopsis techniques for approximate spatial data analysisProceedings of the VLDB Endowment10.14778/3342263.334263512:11(1583-1596)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.14778/3342263.3342635
Wang PWang XTao JZhang PGuan X(2019)Continuously Distinct Sampling over Centralized and Distributed High Speed Data StreamsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2018.286545230:2(300-314)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1109/TPDS.2018.2865452
Show More Cited By

Index Terms

Structure-aware sampling on data streams
1. Mathematics of computing
  1. Probability and statistics

Recommendations

Structure-aware sampling on data streams
Performance evaluation review

The massive data streams observed in network monitoring, data processing and scientific studies are typically too large to store. For many applications over such data, we must obtain compact summaries of the stream. These summaries should allow accurate ...
Adaptive stratified reservoir sampling over heterogeneous data streams

Reservoir sampling is a known technique for maintaining a random sample of a fixed size over a data stream of an unknown size. While reservoir sampling is suitable for applications demanding a sample over the whole data stream, it is not designed for ...
Tracking set-expression cardinalities over continuous update streams

There is growing interest in algorithms for processing and querying continuous data streams (i.e., data seen only once in a fixed order) with limited memory resources. In its most general form, a data stream is actually an <i>update</i> stream, i.e., ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMETRICS '11: Proceedings of the ACM SIGMETRICS joint international conference on Measurement and modeling of computer systems

June 2011

376 pages

ISBN:9781450308144

DOI:10.1145/1993744

General Chair:
Arif Merchant
Google, USA
,
Program Chairs:
Kimberly Keeton
HP Labs, USA
,
Dan Rubenstein
Columbia University, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMETRICS: ACM Special Interest Group on Measurement and Evaluation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 June 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMETRICS '11

Sponsor:

SIGMETRICS

SIGMETRICS '11: ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems

June 7 - 11, 2011

California, San Jose, USA

Acceptance Rates

Overall Acceptance Rate 459 of 2,691 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
288
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)1

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Patil MMagdy A(2021)LATEST: Learning-Assisted Selectivity Estimation Over Spatio-Textual Streams2021 IEEE 37th International Conference on Data Engineering (ICDE)10.1109/ICDE51399.2021.00142(1607-1618)Online publication date: Apr-2021
https://doi.org/10.1109/ICDE51399.2021.00142
Siddique AEldawy AHristidis V(2019)Comparing synopsis techniques for approximate spatial data analysisProceedings of the VLDB Endowment10.14778/3342263.334263512:11(1583-1596)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.14778/3342263.3342635
Wang PWang XTao JZhang PGuan X(2019)Continuously Distinct Sampling over Centralized and Distributed High Speed Data StreamsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2018.286545230:2(300-314)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1109/TPDS.2018.2865452
Wei XLiu YWang XSun BGao SRokne J(2019)A survey on quality-assurance approximate stream processing and applicationsFuture Generation Computer Systems10.1016/j.future.2019.07.047Online publication date: Jul-2019
https://doi.org/10.1016/j.future.2019.07.047
Yang YZhang WZhang YLin XWang L(2019)Selectivity Estimation on Set Containment SearchData Science and Engineering10.1007/s41019-019-00104-14:3(254-268)Online publication date: 23-Sep-2019
https://doi.org/10.1007/s41019-019-00104-1
Yang YZhang WZhang YLin XWang L(2019)Selectivity Estimation on Set Containment SearchDatabase Systems for Advanced Applications10.1007/978-3-030-18576-3_20(330-349)Online publication date: 24-Apr-2019
https://doi.org/10.1007/978-3-030-18576-3_20
Wu GYun XWang SFu GLi CLiu YLi BWang YZhao Z(2019)Accelerating Real-Time Tracking Applications over Big Data Stream with Constrained SpaceDatabase Systems for Advanced Applications10.1007/978-3-030-18576-3_1(3-18)Online publication date: 22-Apr-2019
https://dl.acm.org/doi/10.1007/978-3-030-18576-3_1
Aufaure MChiky RCuré OKhrouf HKepeklian G(2016)From Business Intelligence to semantic data stream managementFuture Generation Computer Systems10.1016/j.future.2015.11.01563:C(100-107)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.1016/j.future.2015.11.015
Wang XZhang YZhang WLin XWang W(2014)Selectivity estimation on streaming spatio-textual data using local correlationsProceedings of the VLDB Endowment10.14778/2735471.27354728:2(101-112)Online publication date: 1-Oct-2014
https://dl.acm.org/doi/10.14778/2735471.2735472
Al-Kateb MLee B(2014)Adaptive stratified reservoir sampling over heterogeneous data streamsInformation Systems10.1016/j.is.2012.03.00539(199-216)Online publication date: 1-Jan-2014
https://dl.acm.org/doi/10.1016/j.is.2012.03.005
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten