skip to main content
10.1145/3321408.3323084acmotherconferencesArticle/Chapter ViewAbstractPublication Pagesacm-turcConference Proceedingsconference-collections
research-article

A memory-compact and fast sketch for online tracking heavy hitters in a data stream

Published: 17 May 2019 Publication History

Abstract

Network traffic measurement is important for network management, including bandwidth management to mitigate network congestion, and security management to detect DDOS attacks and worm spreading. However, with the explosive volume of network data and the fast transmission speed of network packets (in giga or even tera bps), it is a challenging task to measure the size of each network flow both accurately and memory-efficiently, using the size-limited SRAM memory of line card. Therefore, many sublinear space algorithms for processing data streams have been proposed, such as CountMin (CM), Count Sketch (CS) and Virtual Active Counters (VAC), which achieve extreme memory compactness by providing probabilistic guarantees on flow size measurement accuracy. However, these existing algorithms can still be greatly improved as to the performance of both online recording and querying the per-flow size, which is needed for online tracking heavy hitters. Our paper proposes a highly compact and efficient counter architecture, called CountMin virtual active counter (CM-VAC), which provides more accurate measurement results than CM and CS under a very tight memory space. We also achieve higher query speed than VAC by modifying its query policy. We demonstrate the superior performance of our algorithm by both experimental results and theoretical analysis based on CAIDA network traces.

References

[1]
Z. Li, Y. Huang, G. Liu, F. Wang, Y. Liu, Z. Zhang, and Y. Dai, "Challenges, designs, and performances of large-scale open-p2sp content distribution," IEEE TPDS, vol. 24, no. 11, pp. 2181--2191, 2013.
[2]
N. Bandi, A. Metwally, D. Agrawal, and A. El Abbadi, "Fast data stream algorithms using associative memories," Proc. of ACM SIGMOD, 2007.
[3]
R. T. Schweller, A. Gupta, E. Parsons, and C. Yan, "Reversible sketches for efficient and accurate change detection over network data streams," Proc. of ACM SIGCOMM, 2004.
[4]
A. Kumar, M. Sung, J. Xu, and W. Jia, "Data streaming algorithms for efficient and accurate estimation of flow size distribution." Proc. of ACM SIGMETRICS, 2004.
[5]
A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang, "Data streaming algorithms for estimating entropy of network traffic," Proc. of ACM SIGMETRICS, pp. 145--156, 2006.
[6]
G. Cormode and S. Muthukrishnan, "An improved data stream summary: The count-min sketch and its applications," J. Algorithms, vol. 55, no. 1, pp. 58--75, 2005.
[7]
R. Ben-Basat, G. Einziger, R. Friedman, M. C. Luizelli, and E. Waisbard, "Constant time updates in hierarchical heavy hitters," Proc. of ACM SIGCOMM, 2017.
[8]
Z. Liu, A. Manousis, G. Vorsanger, V. Sekar, and V. Braverman, "One sketch to rule them all: Rethinking network flow monitoring with univmon," Proc. of ACM SIGCOMM, 2016.
[9]
M. Yu, L. Jose, and R. Miao, "Software defined traffic measurement with opensketch," Proc. of NSDI, 2013.
[10]
Y. Zhou, Y. Zhou, S. Chen, and Y. Zhang, "Highly compact virtual active counters for per-flow traffic measurement," IEEE INFOCOM, 2018.
[11]
Y. Lu, A. Montanari, B. Prabhakar, S. Dharmapurikar, and A. Kabbani, "Counter braids: a novel counter architecture for per-flow measurement," Proc. of ACM SIGMETRICS, 2008.
[12]
CAIDA, "The caida anonymized internet traces," 2016.
[13]
C. Estan and G. Varghese, "New directions in traffic measurement and accounting," Computer Communication Review, vol. 32, no. 4, 2002.
[14]
M. Charikar, K. C. Chen, and M. Farach-Colton, "Finding frequent items in data streams," Theor. Comput. Sci., vol. 312, pp. 3--15, 2004.
[15]
S. Ganguly, P. B. Gibbons, Y. Matias, and A. Silberschatz, "Bifocal sampling for skew-resistant join size estimation." ACM SIGMOD Record, vol. 25, no. 2, pp. 271--281, 1996.
[16]
Y. Qiao, T. Li, and S. Chen, "One memory access bloom filters and their generalization," Proc of IEEE INFOCOM, pp. 1745--1753, 2011.
[17]
R. Stanojevic, "Small active counters," Proc. of IEEE INFOCOM, 2007.
[18]
C. Estan, K. Keys, D. Moore, and G. Varghese, "Building a better netflow," Proc. of ACM SIGCOMM, pp. 245--256, 2004.
[19]
A. Metwally, D. Agrawal, and A. E. Abbadi, "Efficient computation of frequent and top-k elements in data streams," Proc. of ICDT, 2005.
[20]
P. K. Agarwal, G. Cormode, Z. Huang, J. M. Phillips, Z. Wei, and K. Yi, "Mergeable summaries," ACM TODS, vol. 38, no. 4, 2013.
[21]
L. Tao, S. Chen, and Y. Ling, "Per-flow traffic measurement through randomized counter sharing," IEEE/ACM TON (Trans. on Networking), vol. 20, no. 5, pp. 1622--1634, 2012.
[22]
M. Chen, S. Chen, and Z. Cai, "Counter tree: A scalable counter architecture for per-flow traffic measurement," IEEE/ACM TON (Trans. on Networking), vol. 25, no. 2, pp. 1249--1262, 2017.

Cited By

View all
  • (2023)LossDetection: Real-Time Packet Loss Monitoring System for Sampled Traffic DataIEEE Transactions on Network and Service Management10.1109/TNSM.2022.320338920:1(30-45)Online publication date: Mar-2023
  1. A memory-compact and fast sketch for online tracking heavy hitters in a data stream

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ACM TURC '19: Proceedings of the ACM Turing Celebration Conference - China
    May 2019
    963 pages
    ISBN:9781450371582
    DOI:10.1145/3321408
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 17 May 2019

    Permissions

    Request permissions for this article.

    Check for updates

    Qualifiers

    • Research-article

    Funding Sources

    • Collaborative innovation center of novel software technology & industrialization
    • National Natural Science Foundation of China

    Conference

    ACM TURC 2019

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)9
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 19 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)LossDetection: Real-Time Packet Loss Monitoring System for Sampled Traffic DataIEEE Transactions on Network and Service Management10.1109/TNSM.2022.320338920:1(30-45)Online publication date: Mar-2023

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media