skip to main content
10.1145/3487552.3487856acmconferencesArticle/Chapter ViewAbstractPublication PagesimcConference Proceedingsconference-collections
research-article
Public Access

Precise error estimation for sketch-based flow measurement

Published: 02 November 2021 Publication History

Abstract

As a class of approximate measurement approaches, sketching algorithms have significantly improved the estimation of network flow information using limited resources. While these algorithms enjoy sound error-bound analysis under worst-case scenarios, their actual errors can vary significantly with the incoming flow distribution, making their traditional error bounds too "loose" to be useful in practice. In this paper, we propose a simple yet rigorous error estimation method to more precisely analyze the errors for posterior sketch queries by leveraging the knowledge from the sketch counters. This approach will enable network operators to understand how accurate the current measurements are and make appropriate decisions accordingly (e.g., identify potential heavy users or answer "what-if" questions to better provision resources). Theoretical analysis and trace-driven experiments show that our estimated bounds on sketch errors are much tighter than previous ones and match the actual error bounds in most cases.

References

[1]
Mohammad Alizadeh, Tom Edsall, Sarang Dharmapurikar, Ramanan Vaidyanathan, Kevin Chu, Andy Fingerhut, Vinh The Lam, Francis Matus, Rong Pan, Navindra Yadav, et al. 2014. CONGA: Distributed congestion-aware load balancing for datacenters. In Proc. of ACM SIGCOMM.
[2]
Mohammad Alizadeh, Shuang Yang, Milad Sharif, Sachin Katti, Nick McKeown, Balaji Prabhakar, and Scott Shenker. 2013. PFabric: Minimal near-Optimal Datacenter Transport. In Proc. of ACM SIGCOMM.
[3]
Ran Ben Basat, Gil Einziger, Roy Friedman, Marcelo Caggiani Luizelli, and Erez Waisbard. 2017. Constant Time Updates in Hierarchical Heavy Hitters. In Proc. of ACM SIGCOMM and CoRR/1707.06778.
[4]
Theophilus Benson, Aditya Akella, and David A Maltz. 2010. Network traffic characteristics of data centers in the wild. In Proc. of SIGCOMM IMC.
[5]
CAIDA. 2018. The CAIDA UCSD Anonymized Internet Traces equinix-chicago. http://www.caida.org/data/passive/passive_dataset.xml
[6]
Moses Charikar, Kevin Chen, and Martin Farach-Colton. 2002. Finding Frequent Items in Data Streams. In Proc. of ICALP.
[7]
Graham Cormode, Flip Korn, S. Muthukrishnan, and Divesh Srivastava. 2008. Finding Hierarchical Heavy Hitters in Streaming Data. ACM Trans. Knowl. Discov. Data (2008).
[8]
Graham Cormode and S. Muthukrishnan. 2005. An Improved Data Stream Summary: The Count-Min Sketch and Its Applications. J. Algorithms (2005).
[9]
Cristian Estan and George Varghese. 2002. New directions in traffic measurement and accounting. In Proc. of ACM SIGCOMM.
[10]
Cristian Estan and George Varghese. 2003. New directions in traffic measurement and accounting: Focusing on the elephants, ignoring the mice. ACM Transactions on Computer Systems (2003).
[11]
Bin Fan, Hyeontaek Lim, David G Andersen, and Michael Kaminsky. 2011. Small cache, big effect: Provable load balancing for randomly partitioned cluster services. In Proc. of SoCC.
[12]
Qun Huang, Patrick PC Lee, and Yungang Bao. 2018. SketchLearn: Relieving User Burdens in Approximate Measurement with Automated Statistical Inference. In Proc. of ACM SIGCOMM.
[13]
Intel. [n. d.]. High Performance Layer-4 Load Balancer based on DPDK. https://github.com/iqiyi/dpvs.
[14]
Xin Jin, Xiaozhou Li, Haoyu Zhang, Robert Soulé, Jeongkeun Lee, Nate Foster, Changhoon Kim, and Ion Stoica. 2017. NetCache: Balancing Key-Value Stores with Fast In-Network Caching. In Proc. of ACM SOSP.
[15]
Balachander Krishnamurthy, Subhabrata Sen, Yin Zhang, and Yan Chen. 2003. Sketch-based Change Detection: Methods, Evaluation, and Applications. In Proc. of ACM IMC.
[16]
Hongqiang Harry Liu, Srikanth Kandula, Ratul Mahajan, Ming Zhang, and David Gelernter. 2014. Traffic engineering with forward fault correction. In Proc. of ACM SIGCOMM.
[17]
Zaoxing Liu, Zhihao Bai, Zhenming Liu, Xiaozhou Li, Changhoon Kim, Vladimir Braverman, Xin Jin, and Ion Stoica. 2019. DistCache: Provable Load Balancing for Large-Scale Storage Systems with Distributed Caching. In Proc. of USENIX FAST.
[18]
Zaoxing Liu, Antonis Manousis, Gregory Vorsanger, Vyas Sekar, and Vladimir Braverman. 2016. One Sketch to Rule Them All: Rethinking Network Flow Monitoring with UnivMon. In Proc. of ACM SIGCOMM.
[19]
Zaoxing Liu, Samson Zhou, Ori Rottenstreich, Vladimir Braverman, and Jennifer Rexford. 2019. Memory-Efficient Performance Monitoring on Programmable Switches with Lean Algorithms. Proc. of SIAM/ACM APoCS (2019).
[20]
Yi Lu, Andrea Montanari, Balaji Prabhakar, Sarang Dharmapurikar, and Abdul Kabbani. 2008. Counter Braids: A Novel Counter Architecture for PerFlowMeasurement. In Proc. of ACM SIGMETRICS.
[21]
MACCDC. 2012. Capture Traces from Mid-Atlantic CCDC. http://www.netresec.com/?page=MACCDC
[22]
Rui Miao, Hongyi Zeng, Changhoon Kim, Jeongkeun Lee, and Minlan Yu. 2017. SilkRoad: Making Stateful Layer-4 Load Balancing Fast and Cheap Using Switching ASICs. In Proc. of ACM SIGCOMM.
[23]
Masoud Moshref, Minlan Yu, Ramesh Govindan, and Amin Vahdat. 2015. Scream: Sketch resource allocation for software-defined measurement. In Proc. of ACM CoNEXT.
[24]
Chen Peiqing, Chen Dong, Zheng Lingxiao, Li Jizhou, and Yang Tong. 2021. Out of Many We are One: Measuring Item Batch with Clock-Sketch. In Proceedings of the 2021 International Conference on Management of Data (Virtual Event, China) (SIGMOD '21). Association for Computing Machinery, New York, NY, USA.
[25]
David MW Powers. 1998. Applications and explanations of Zipf's law. In New methods in language processing and computational natural language learning.
[26]
Alex Rousskov and Duane Wessels. 2004. High-performance benchmarking with Web Polygraph. Software: Practice and Experience (2004).
[27]
Rachee Singh, Manya Ghobadi, Klaus-Tycho Foerster, Mark Filer, and Phillipa Gill. 2018. RADWAN: rate adaptive wide area network. In Proc. of ACM SIGCOMM.
[28]
Cha Hwan Song, Pravein Govindan Kannan, Bryan Kian Hsiang Low, and Mun Choon Chan. 2020. FCM-sketch: generic network measurements with data plane support. In Proc. of CoNEXT.
[29]
Daniel Ting. 2018. Count-min: Optimal estimation and tight error bounds using empirical error distributions. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2319--2328.
[30]
Tong Yang, Jie Jiang, Peng Liu, Qun Huang, Junzhi Gong, Yang Zhou, Rui Miao, Xiaoming Li, and Steve Uhlig. 2018. Elastic Sketch: Adaptive and Fast Network-wide Measurements. In Proc. of ACM SIGCOMM.
[31]
Yinda Zhang, Zaoxing Liu, Ruixin Wang, Tong Yang, Jizhou Li, Ruijie Miao, Peng Liu, Ruwen Zhang, and Junchen Jiang. 2021. CocoSketch: High-Performance Sketch-based Measurement over Arbitrary Partial Key Query. In Proc. of ACM SIGCOMM.

Cited By

View all
  • (2024)OctoSketchProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691914(1621-1639)Online publication date: 16-Apr-2024
  • (2024)Raising the Level of Abstraction for Sketch-Based Network Telemetry with SketchPlanProceedings of the 2024 ACM on Internet Measurement Conference10.1145/3646547.3689016(651-658)Online publication date: 4-Nov-2024
  • (2024)Unbiased Real-Time Traffic SketchingIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.328400411:3(2371-2383)Online publication date: May-2024
  • Show More Cited By

Index Terms

  1. Precise error estimation for sketch-based flow measurement

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      IMC '21: Proceedings of the 21st ACM Internet Measurement Conference
      November 2021
      768 pages
      ISBN:9781450391290
      DOI:10.1145/3487552
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      In-Cooperation

      • USENIX Assoc: USENIX Assoc

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 02 November 2021

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. error estimation
      2. network algorithm
      3. sketch

      Qualifiers

      • Research-article

      Funding Sources

      Conference

      IMC '21
      IMC '21: ACM Internet Measurement Conference
      November 2 - 4, 2021
      Virtual Event

      Acceptance Rates

      Overall Acceptance Rate 277 of 1,083 submissions, 26%

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)197
      • Downloads (Last 6 weeks)21
      Reflects downloads up to 15 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)OctoSketchProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691914(1621-1639)Online publication date: 16-Apr-2024
      • (2024)Raising the Level of Abstraction for Sketch-Based Network Telemetry with SketchPlanProceedings of the 2024 ACM on Internet Measurement Conference10.1145/3646547.3689016(651-658)Online publication date: 4-Nov-2024
      • (2024)Unbiased Real-Time Traffic SketchingIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.328400411:3(2371-2383)Online publication date: May-2024
      • (2024)NeoMem: Hardware/Software Co-Design for CXL-Native Memory Tiering2024 57th IEEE/ACM International Symposium on Microarchitecture (MICRO)10.1109/MICRO61859.2024.00111(1518-1531)Online publication date: 2-Nov-2024
      • (2024)DISCO: A Dynamically Configurable Sketch Framework in Skewed Data Streams2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00365(4801-4814)Online publication date: 13-May-2024
      • (2023)EasyQuantile: Efficient Quantile Tracking in the Data PlaneProceedings of the 7th Asia-Pacific Workshop on Networking10.1145/3600061.3600084(123-129)Online publication date: 29-Jun-2023
      • (2023)JoinSketch: A Sketch Algorithm for Accurate and Unbiased Inner-Product EstimationProceedings of the ACM on Management of Data10.1145/35889351:1(1-26)Online publication date: 30-May-2023
      • (2023)Double-Anonymous Sketch: Achieving Top-K-fairness for Finding Global Top-K Frequent ItemsProceedings of the ACM on Management of Data10.1145/35889331:1(1-26)Online publication date: 30-May-2023
      • (2023)LadderFilter: Filtering Infrequent Items with Small Memory and Time OverheadProceedings of the ACM on Management of Data10.1145/35886901:1(1-21)Online publication date: 30-May-2023
      • (2023)SketchINT: Empowering INT With TowerSketch for Per-Flow Per-Switch MeasurementIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2023.330392434:11(2876-2894)Online publication date: 1-Nov-2023
      • Show More Cited By

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Login options

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media