Abstract
Accurately finding heavy flows in data streams is challenging owing to limited memory availability. Prior algorithms have focused on accuracy in heavy flow detection but cannot provide the frequency of a heavy flow exactly. In this paper, we designed a two-mode counter, called Matthew Counter, for the efficient use of memory and an accurate record flow frequency. The key ideas in Matthew Counter are the use of idle high-bits in the counter and the adoption of a power-weakening method. Matthew Counter allows sufficient competition during the early stages of identifying heavy flows and amplifying the relative advantage when the counter is sufficiently large to ensure the level of accuracy. We also present an invertible sketch, called MCSketch, for supporting heavy-flow detection with small and static memory based on Matthew Counter. The experiment results show that MCSketch achieves a higher accuracy than existing algorithms for heavy flow detection. Moreover, MCSketch reduces the average relative error by approximately 1 to 3 orders of magnitude in comparison to other state-of-art approaches.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Goyal, A., Daume, H., Cormode, G., et al.: Sketch algorithms for estimating point queries in NLP. In: Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning, pp. 1093–1103 (2012)
Wang, J., Wen, R., Li, J., et al.: Detecting and mitigating target link-flooding attacks using SDN. IEEE Trans Dependable Secur Comput 16, 944–956 (2018). https://doi.org/10.1109/TDSC.2018.2822275
Zheng, J., Li, Q., Gu, G., et al.: Realtime DDoS defense using COTS SDN switches via adaptive correlation analysis. IEEE Trans. Inf. Forensics Secur. 13, 1838–1853 (2018). https://doi.org/10.1109/TIFS.2018.2805600
Tang, L., Huang, Q., Lee, P.P.C.: MV-Sketch: a fast and compact invertible sketch for heavy flow detection in network data streams. In: Proceedings - IEEE INFOCOM 2019-April, pp. 2026–2034 (2019). https://doi.org/10.1109/INFOCOM.2019.8737499
Gong, J., Yang, T., Zhang, H., et al.: HeavyKeeper: An accurate algorithm for finding top-k elephant flows. In: Proc 2018 USENIX Annual Tech Conference USENIX ATC 2018, pp. 909–921 (2018). https://doi.org/10.1109/tnet.2019.2933868
Li, J., Li, Z., Xu, Y., et al.: WavingSketch: An Unbiased and Generic Sketch for Finding Top-k Items in Data Streams. In: Proceedings ACM SIGKDD International Conference Knowledge Discovery Data Mining, pp. 1574–1584 (2020). https://doi.org/10.1145/3394486.3403208
Source code related to MCSketch. https://github.com/Paper-commits/MCSketch
Cormode, G., Muthukrishnan, S.: An improved data stream summary: the count-min sketch and its applications. J. Algorithms 55, 58–75 (2005). https://doi.org/10.1016/j.jalgor.2003.12.001
Estan, C., Varghese, G.: New directions in traffic measurement and accounting. Comput. Commun. Rev. 32, 75 (2002). https://doi.org/10.1145/510726.510749
Cormode, G., Hadjieleftheriou, M.: Finding frequent items in data streams. Proc. VLDB Endow. 1(2), 1530–1541 (2008). https://doi.org/10.14778/1454159.1454225
Yang, T., Huang, Q., Miao, R., et al.: Elastic sketch: adaptive and fast network-wide measurements. In: SIGCOMM 2018 - Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. Association for Computing Machinery, Inc, pp. 561–575 (2018)
Zhou, Y., Yang, T., Jiang, J., et al.: Cold filter: a meta-framework for faster and more accurate stream processing. In: Proceedings of the ACM SIGMOD Int. Conf. Manage. Data, pp 741–756 (2018)
Ting, D.: Count-min: optimal estimation and tight error bounds using empirical error distributions. In: Proc ACM SIGKDD Int. Conf. Knowl. Discov. Data Min., pp. 2319–2328 (2018). https://doi.org/10.1145/3219819.3219975
Huang, Q., Lee, P.P.C., Bao, Y.: SketChlearn: Relieving user burdens in approximate measurement with automated statistical inference. In: SIGCOMM 2018 - Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. Association for Computing Machinery, Inc, pp. 576–590 (2018)
Jie, L., Hongchang, C., Penghao, S., et al.: OrderSketch: an unbiased and fast sketch for frequency estimation of data streams. Comput. Networks 201, 108563 (2021)
Huang, Q., Lee, P.P.C.: A hybrid local and distributed sketching design for accurate and scalable heavy key detection in network data streams. Comput. Networks 91, 298–315 (2015). https://doi.org/10.1016/j.comnet.2015.08.025
Wu, M., Huang, H., Sun, Y., et al.: ActiveKeeper : an accurate and efficient algorithm for finding top- k elephant flows. IEEE Commun. Lett. 7798, 1–5 (2021). https://doi.org/10.1109/LCOMM.2021.3077902
The caida anonymized internet traces 2016. http://www.caida.org/data/overview/
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lu, J., Chen, H., Zhang, Z. (2023). MCSketch: An Accurate Sketch for Heavy Flow Detection and Heavy Flow Frequency Estimation. In: Li, B., Yue, L., Tao, C., Han, X., Calvanese, D., Amagasa, T. (eds) Web and Big Data. APWeb-WAIM 2022. Lecture Notes in Computer Science, vol 13421. Springer, Cham. https://doi.org/10.1007/978-3-031-25158-0_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-25158-0_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25157-3
Online ISBN: 978-3-031-25158-0
eBook Packages: Computer ScienceComputer Science (R0)