Article

Variance estimation over sliding windows

Authors:

Linfeng Zhang,

Yong GuanAuthors Info & Claims

PODS '07: Proceedings of the twenty-sixth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

Pages 225 - 232

https://doi.org/10.1145/1265530.1265562

Published: 11 June 2007 Publication History

Get Access

Abstract

Capturing characteristics of large data streams has received considerable attention. The constraints in space and time restrict the data stream processing to only one pass (or a small number of passes). Processing data streams over sliding windows make the problem more difficult and challenging. In this paper, we address the problem of maintaining ∈-approximate variance of data streams over sliding windows. To our knowledge, the best existing algorithm requires O(1/∈2 log N) space, though the lower bound for this problem is Ω(1/∈ log N). We propose the first ∈-approximation algorithm to this problem that is optimal in both space and worst case time. Our algorithm requires O(1/∈ log N) space. Furthermore, its running time is O(1) in worst case.

References

[1]

A. Arasu and G. S. Manku. Approximate counts and quantiles over sliding windows. In Proceedings of the 23rd ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems (PODS 2004), Paris, France, June 2004.

Digital Library

Google Scholar

[2]

B. Babcock, S. Babu, M. Datar, R. Motwani, and J. Widom. Models and issues in data stream systems. In Proceedings of the 21st ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems (PODS 2002), pages 1--16, Madison, USA, June 2002.

Digital Library

Google Scholar

[3]

B. Babcock, M. Datar, R. Motwani, and L. O'Callaghan. Maintaining variance and k-medians over data stream windows. In Proceedings of the 22nd ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems (PODS 2003), pages 234--243, San Diego, USA, June 2003.

Digital Library

Google Scholar

[4]

M. Datar, A. Gionis, P. Indyk, and R. Motwani. Maintaining stream statistics over sliding windows. In Proceedings of the 13th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA 2002), pages 635--644, San Francisco, USA, Jan. 2002.

Digital Library

Google Scholar

[5]

J. Feigenbaum, S. Kannan, and J. Zhang. Computing diameter in the streaming and sliding-window models. Algorithmica, 41(1):25--41, 2004.

Digital Library

Google Scholar

[6]

P. B. Gibbons and S. Tirthapura. Distributed streams algorithms for sliding windows. In Proceedings of the 14th ACM Symposium on Parallel Algorithms and Architectures (SPAA 2002), Winnipeg, Manitoba, Canada, Aug. 2002.

Digital Library

Google Scholar

[7]

L. Golab, D. DeHaan, E. D. Demaine, A. López-Ortiz, and J. I. Munro. Identifying frequent items in sliding windows over on-line packet streams. In Proceedings of the Internet Measurement Conference (IMC 2003), Miami, USA, Oct. 2003.

Digital Library

Google Scholar

[8]

X. Lin, H. Lu, J. Xu, and J. X. Yu. Continuously maintaining quantile summaries of the most recent N elements over a data stream. In Proceedings of the 20th International Conference on Data Engineering (ICDE 2004), Boston, USA, Mar. 2004.

Digital Library

Google Scholar

[9]

S. Muthukrishnan. Data streams: Algorithms and applications. Technical report, Rutgers University, Piscataway, USA, 2003.

Google Scholar

[10]

S. Tirthapura, B. Xu, and C. Busch. Sketching asynchronous streams over sliding windows. In Proceedings of the 25th ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing (PODC 2006), Denver, USA, July 2006.

Digital Library

Google Scholar

[11]

Y. Zhu and D. Shasha. Statstream: Statistical monitoring of thousands of data streams in real time. In Proceedings of 28th International Conference on Very Large Data Bases (VLDB 2002), Hong Kong, China, Aug. 2002.

Digital Library

Google Scholar

Cited By

View all

Darányi AAbonyi J(2024)Fault Diagnostics Based on the Analysis of Probability Distributions Estimated Using a Particle FilterSensors10.3390/s2403071924:3(719)Online publication date: 23-Jan-2024
https://doi.org/10.3390/s24030719
Hardel VHervieux PManfredi G(2023)Relaxation to Quantum Equilibrium and the Born Rule in Nelson’s Stochastic DynamicsFoundations of Physics10.1007/s10701-023-00730-w53:6Online publication date: 6-Nov-2023
https://doi.org/10.1007/s10701-023-00730-w
Nguyen TShih MSrivastava DTirthapura SXu B(2020)Stratified random sampling from streaming and stored dataDistributed and Parallel Databases10.1007/s10619-020-07315-w39:3(665-710)Online publication date: 23-Oct-2020
https://dl.acm.org/doi/10.1007/s10619-020-07315-w
Show More Cited By

Index Terms

Variance estimation over sliding windows
1. Information systems
  1. Information systems applications
    1. Data mining
2. Theory of computation
  1. Design and analysis of algorithms

Recommendations

Optimal sampling from sliding windows
PODS '09: Proceedings of the twenty-eighth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

A sliding windows model is an important case of the streaming model, where only the most "recent" elements remain active and the rest are discarded in a stream. The sliding windows model is important for many applications (see, e.g., Babcock, Babu, ...
Optimal sampling from sliding windows

A sliding windows model is an important case of the streaming model, where only the most ''recent'' elements remain active and the rest are discarded. The sliding windows model is important for many applications (see, e.g., Babcock, Babu, Datar, Motwani ...
Distributed streams algorithms for sliding windows
SPAA '02: Proceedings of the fourteenth annual ACM symposium on Parallel algorithms and architectures
This paper presents algorithms for estimating aggregate functions over a "sliding window" of the N most recent data items in one or more streams. Our results include
- For a single stream, we present the first ε-approximation scheme for the number of 1's ...

Comments

Information & Contributors

Information

Published In

PODS '07: Proceedings of the twenty-sixth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

June 2007

328 pages

ISBN:9781595936851

DOI:10.1145/1265530

General Chair:
Phokion Kolaitis
IBM Almaden
,
Program Chair:
Leonid Libkin
University of Edinburgh

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 June 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGMOD/PODS07

Sponsor:

SIGMOD/PODS07: International Conference on Management of Data

June 11 - 13, 2007

Beijing, China

Acceptance Rates

PODS '07 Paper Acceptance Rate 28 of 187 submissions, 15%;

Overall Acceptance Rate 642 of 2,707 submissions, 24%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
781
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Darányi AAbonyi J(2024)Fault Diagnostics Based on the Analysis of Probability Distributions Estimated Using a Particle FilterSensors10.3390/s2403071924:3(719)Online publication date: 23-Jan-2024
https://doi.org/10.3390/s24030719
Hardel VHervieux PManfredi G(2023)Relaxation to Quantum Equilibrium and the Born Rule in Nelson’s Stochastic DynamicsFoundations of Physics10.1007/s10701-023-00730-w53:6Online publication date: 6-Nov-2023
https://doi.org/10.1007/s10701-023-00730-w
Nguyen TShih MSrivastava DTirthapura SXu B(2020)Stratified random sampling from streaming and stored dataDistributed and Parallel Databases10.1007/s10619-020-07315-w39:3(665-710)Online publication date: 23-Oct-2020
https://dl.acm.org/doi/10.1007/s10619-020-07315-w
Bobek SNalepa GŚlażyński M(2018)HEARTDROID—Rule engine for mobile and context‐aware expert systemsExpert Systems10.1111/exsy.1232836:1Online publication date: 23-Aug-2018
https://doi.org/10.1111/exsy.12328
Nalepa GNalepa G(2017)Rules in Mobile Context-Aware SystemsModeling with Rules Using Semantic Knowledge Engineering10.1007/978-3-319-66655-6_17(403-430)Online publication date: 5-Oct-2017
https://doi.org/10.1007/978-3-319-66655-6_17
Lahiri BMukherjee ATirthapura S(2016)Identifying correlated heavy-hitters in a two-dimensional data streamData Mining and Knowledge Discovery10.1007/s10618-015-0438-630:4(797-818)Online publication date: 1-Jul-2016
https://dl.acm.org/doi/10.1007/s10618-015-0438-6
Rivetti NBusnel YMostefaoui A(2015)Efficiently Summarizing Data Streams over Sliding WindowsProceedings of the 2015 IEEE 14th International Symposium on Network Computing and Applications (NCA)10.1109/NCA.2015.46(151-158)Online publication date: 28-Sep-2015
https://dl.acm.org/doi/10.1109/NCA.2015.46
Chen LMei Q(2014)Mining frequent items in data stream using time fading modelInformation Sciences: an International Journal10.1016/j.ins.2013.09.007257(54-69)Online publication date: 1-Feb-2014
https://dl.acm.org/doi/10.1016/j.ins.2013.09.007
Liu YZhang LGuan Y(2010)Sketch-Based Streaming PCA Algorithm for Network-Wide Traffic Anomaly DetectionProceedings of the 2010 IEEE 30th International Conference on Distributed Computing Systems10.1109/ICDCS.2010.45(807-816)Online publication date: 21-Jun-2010
https://dl.acm.org/doi/10.1109/ICDCS.2010.45
Feigenblat GItzhaki OPorat E(2010)The frequent items problem, under polynomial decay, in the streaming modelTheoretical Computer Science10.1016/j.tcs.2010.04.029411:34-36(3048-3054)Online publication date: 1-Jul-2010
https://dl.acm.org/doi/10.1016/j.tcs.2010.04.029
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Optimal sampling from sliding windows

Optimal sampling from sliding windows

Distributed streams algorithms for sliding windows

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations