skip to main content
10.1145/2745754.2745757acmconferencesArticle/Chapter ViewAbstractPublication PagespodsConference Proceedingsconference-collections
research-article

External Memory Stream Sampling

Published: 20 May 2015 Publication History

Abstract

This paper aims to understand the I/O-complexity of maintaining a big sample set---whose size exceeds the internal memory's capacity---on a data stream. We study this topic in a new computation model, named the external memory stream (EMS) model, that naturally extends the standard external memory model to stream environments. A suite of EMS-indigenous techniques are presented to prove matching lower and upper bounds for with-replacement (WR) and without-replacement (WoR) sampling on append-only and time-based sliding window streams, respectively. Our results imply that, compared to RAM, the EMS model is perhaps a more suitable computation model for studying stream sampling, because the new model separates different problems by their hardness in ways that could not be observed in RAM.

References

[1]
A. Aggarwal and J. S. Vitter. The input/output complexity of sorting and related problems. CACM, 31(9):1116--1127, 1988.
[2]
L. Arge. The buffer tree: A technique for designing batched external data structures. Algorithmica, 37(1):1--24, 2003.
[3]
B. Babcock, M. Datar, and R. Motwani. Sampling from a moving window over streaming data. In SODA, pages 633--634, 2002.
[4]
V. Braverman, R. Ostrovsky, and C. Zaniolo. Optimal sampling from sliding windows. JCSS, 78(1):260--272, 2012.
[5]
M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows. SIAM J. of Comp., 31(6):1794--1813, 2002.
[6]
C. Fan, M. Muller, and I. Rezucha. Development of sampling plans by using sequential (item-by-item) selection techniques and digital computers. Am. Stat. Assn. J., 57:387--402, 1962.
[7]
R. Gemulla and W. Lehner. Deferred maintenance of disk-based random samples. In EDBT, pages 423--441, 2006.
[8]
R. Gemulla and W. Lehner. Sampling time-based sliding windows in bounded space. In SIGMOD, pages 379--392, 2008.
[9]
J. Gustedt. Efficient sampling of random permutations. Journal of Discrete Algorithms, 6(1):125--139, 2008.
[10]
X. Hu, M. Qiao, and Y. Tao. Independent range sampling. In PODS, pages 246--255, 2014.
[11]
D. A. Hutchinson, P. Sanders, and J. S. Vitter. Duality between prefetching and queued writing with parallel disks. SIAM J. of Comp., 34(6):1443--1463, 2005.
[12]
C. Jermaine, A. Pol, and S. Arumugam. Online maintenance of very large random samples. In SIGMOD, pages 299--310, 2004.
[13]
T. Jones. A note on sampling a tape-file. CACM, 5(6):343, 1962.
[14]
Y. Matias, E. Segal, and J. S. Vitter. Efficient bundle sorting. SIAM J. of Comp., 36(2):394--410, 2006.
[15]
S. Nath and P. B. Gibbons. Online maintenance of very large random samples on flash storage. PVLDB, 1(1):970--983, 2008.
[16]
S. Nath and P. B. Gibbons. Online maintenance of very large random samples on flash storage. VLDB J., 19(1):67--90, 2010.
[17]
A. Pavan, K. Tangwongsan, S. Tirthapura, and K. Wu. Counting and sampling triangles from a graph stream. PVLDB, 6(14):1870--1881, 2013.
[18]
A. Pol, C. M. Jermaine, and S. Arumugam. Maintaining very large random samples using the geometric file. VLDB J., 17(5):997--1018, 2008.
[19]
J. S. Vitter. Random sampling with a reservoir. ACM Trans. Math. Softw., 11(1):37--57, 1985.
[20]
K. Yi. Dynamic indexability and the optimality of B-trees. JACM, 59(4), 2012.

Cited By

View all
  • (2023)Practical Dynamic Extension for Sampling IndexesProceedings of the ACM on Management of Data10.1145/36267441:4(1-26)Online publication date: 12-Dec-2023
  • (2022)Algorithmic Techniques for Independent Query SamplingProceedings of the 41st ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems10.1145/3517804.3526068(129-138)Online publication date: 12-Jun-2022
  • (2017)Stream sampling over windows with worst-case optimality and $$\ell $$ℓ-overlap independenceThe VLDB Journal — The International Journal on Very Large Data Bases10.1007/s00778-017-0461-x26:4(493-510)Online publication date: 1-Aug-2017

Index Terms

  1. External Memory Stream Sampling

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    PODS '15: Proceedings of the 34th ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems
    May 2015
    358 pages
    ISBN:9781450327572
    DOI:10.1145/2745754
    • General Chair:
    • Tova Milo,
    • Program Chair:
    • Diego Calvanese
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 20 May 2015

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. i/o-efficient algorithms
    2. lower bound
    3. sampling
    4. stream

    Qualifiers

    • Research-article

    Funding Sources

    • HKRGC

    Conference

    SIGMOD/PODS'15
    Sponsor:
    SIGMOD/PODS'15: International Conference on Management of Data
    May 31 - June 4, 2015
    Victoria, Melbourne, Australia

    Acceptance Rates

    PODS '15 Paper Acceptance Rate 25 of 80 submissions, 31%;
    Overall Acceptance Rate 642 of 2,707 submissions, 24%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)3
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 06 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Practical Dynamic Extension for Sampling IndexesProceedings of the ACM on Management of Data10.1145/36267441:4(1-26)Online publication date: 12-Dec-2023
    • (2022)Algorithmic Techniques for Independent Query SamplingProceedings of the 41st ACM SIGMOD-SIGACT-SIGAI Symposium on Principles of Database Systems10.1145/3517804.3526068(129-138)Online publication date: 12-Jun-2022
    • (2017)Stream sampling over windows with worst-case optimality and $$\ell $$ℓ-overlap independenceThe VLDB Journal — The International Journal on Very Large Data Bases10.1007/s00778-017-0461-x26:4(493-510)Online publication date: 1-Aug-2017

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media