Abstract
Per-flow spread measurement in high-speed networks, which aims to estimate the number of distinct elements of each flow, plays an important role in many practical applications. Most existing solutions adopt compact data structures (i.e., sketches) to share memory units among flows so that they can fit in limited on-chip memory, resulting in low estimation accuracy for small flows. Unlike sketch-based solutions, non-duplicate sampling measures per-flow spreads by sampling each distinct element with the same sampling probability. However, it ignores that, compared to small flows, large flows only need lower sampling probabilities to achieve the same relative estimation error, wasting significant on-chip memory for large flows. This paper presents multi-layer adaptive sampling to complement the prior work by assigning lower probabilities to larger flows. The proposed framework employs a multi-layer model to sample distinct elements, ensuring that most small flows will stay in lower layers and large flows will get to higher layers. Besides, higher layers are designed with smaller overall probabilities to ensure that larger flows have lower sampling probabilities. Experimental results based on real Internet traces show that, compared to the state-of-the-art method, our solution can reduce up to \(86\%\) average relative errors for per-flow spread estimation and reduce the FPRs and FNRs of flow misclassification by around one to two magnitudes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
CAIDA: The CAIDA UCSD Anonymized Internet Traces 2016. http://www.caida.org/data/passive/passive_2016_dataset.xml. Accessed 28 Jul 2019
Chen, M., Chen, S., Cai, Z.: Counter tree: a scalable counter architecture for per-flow traffic measurement. IEEE/ACM Trans. Networking 25(2), 1249–1262 (2017). https://doi.org/10.1109/TNET.2016.2621159
Choi, B.Y., Park, J., Zhang, Z.L.: Adaptive random sampling for traffic load measurement. In: IEEE International Conference on Communications, 2003, ICC 2003, vol. 3, pp. 1552–1556. IEEE (2003)
Dai, H., Shahzad, M., Liu, A.X., Li, M., Zhong, Y., Chen, G.: Identifying and estimating persistent items in data streams. IEEE/ACM Trans. Networking 26(6), 2429–2442 (2018)
Dimitropoulos, X., Hurley, P., Kind, A.: Probabilistic lossy counting: an efficient algorithm for finding heavy hitters. ACM SIGCOMM Comput. Commun. Rev. 38(1), 7–16 (2008)
Du, Y., Huang, H., Sun, Y.E., Chen, S., Gao, G.: Self-adaptive sampling for network traffic measurement. In: IEEE INFOCOM 2021-IEEE Conference on Computer Communications, pp. 1–10. IEEE (2021)
Duffield, N., Lund, C., Thorup, M.: Learn more, sample less: control of volume and variance in network measurement. IEEE Trans. Inf. Theory 51(5), 1756–1775 (2005)
Duffield, N., Lund, C., Thorup, M., Thorup, M.: Flow sampling under hard resource constraints. In: ACM SIGMETRICS Performance Evaluation Review, vol. 32, pp. 85–96 (2004)
Estan, C., Varghese, G.: New directions in traffic measurement and accounting: focusing on the elephants, ignoring the mice. ACM Trans. Comput. Syst. (TOCS) 21(3), 270–313 (2003)
Hao, F., Kodialam, M., Lakshman, T.: ACCEL-RATE: a faster mechanism for memory efficient per-flow traffic estimation. In: ACM SIGMETRICS Performance Evaluation Review, vol. 32, pp. 155–166 (2004)
Heule, S., Nunkesser, M., Hall, A.: HyperLogLog in practice: algorithmic engineering of a state of the art cardinality estimation algorithm. In: Proceedings of of the 16th International Conference on Extending Database Technology (EDBT 2013), pp. 683–692 (2013)
Hu, C., Wang, S., Tian, J., Liu, B., Cheng, Y., Chen, Y.: Accurate and efficient traffic monitoring using adaptive non-linear sampling method. In: IEEE INFOCOM 2008-The 27th Conference on Computer Communications, pp. 26–30. IEEE (2008)
Huang, H., et al.: You can drop but you can’t hide: \(k\)-persistent spread estimation in high-speed networks. In: Proceedings of the IEEE Conference on Computer Communications (INFOCOM 2018). pp. 1889–1897 (2018)
Kumar, A., Xu, J., Wang, J.: Space-code bloom filter for efficient per-flow traffic measurement. IEEE J. Sel. Areas Commun. 24(12), 2327–2339 (2006)
Li, T., Chen, S., Luo, W., Zhang, M.: Scan detection in high-speed networks based on optimal dynamic bit sharing. In: Proceedings of the IEEE Conference on Computer Communications (INFOCOM 2011), pp. 3200–3208 (2011)
Lieven, P., Scheuermann, B.: High-speed per-flow traffic measurement with probabilistic multiplicity counting. In: Proceedings of the IEEE Conference on Computer Communications (INFOCOM 2010), pp. 1–9 (2010)
Lu, Y., Montanari, A., Prabhakar, B., Dharmapurikar, S., Kabbani, A.: Counter braids: a novel counter architecture for per-flow measurement. ACM SIGMETRICS Perform. Eval. Rev. 36(1), 121–132 (2008)
Sun, Y.E., Huang, H., Ma, C., Chen, S., Du, Y., Xiao, Q.: Online spread estimation with non-duplicate sampling. In: Proceedings of IEEE INFOCOM 2020, pp. 2440–2448 (2020)
Xiao, Q., Qiao, Y., Zhen, M., Chen, S.: Estimating the persistent spreads in high-speed networks. In: Proceedings of the IEEE 22nd International Conference on Network Protocols (ICNP 2014), pp. 131–142 (2014)
Yang, T., Xu, J., Liu, X., Liu, P., Wang, L., Bi, J., Li, X.: A generic technique for sketches to adapt to different counting ranges. In: Proceedings of the IEEE Conference on Computer Communications (INFOCOM 2019), pp. 2017–2025 (2019)
Yoon, M., Li, T., Chen, S., Peir, J.K.: Fit a compact spread estimator in small high-speed memory. IEEE/ACM Trans. Networking (TON) 19(5), 1253–1264 (2011)
Zhou, Y., Zhang, Y., Ma, C., Chen, S., Odegbile, O.O.: Generalized sketch families for network traffic measurement. In: Proceedings of the ACM on Measurement and Analysis of Computing Systems (POMACS), vol. 3, no. 3, p. 51 (2019)
Zhou, Y., Zhou, Y., Chen, M., Chen, S.: Persistent spread measurement for big network data based on register intersection. In: Proceedings of the ACM on Measurement and Analysis of Computing Systems, vol. 1, p. 15 (2017)
Zhou, Y., Zhou, Y., Chen, M., Xiao, Q., Chen, S.: Highly compact virtual counters for per-flow traffic measurement through register sharing. In: Proceedings of the IEEE GLOBECOM 2016, pp. 1–6 (2016)
Zhou, Y., Zhou, Y., Chen, S., Zhang, Y.: Per-flow counting for big network data stream over sliding windows. In: Proceedings of the IEEE/ACM IWQoS 2017, pp. 1–10 (2017)
Zhou, Y., Zhou, Y., Chen, S., Zhang, Y.: Highly compact virtual active counters for per-flow traffic measurement. In: Proceedings of the IEEE Conference on Computer Communications (INFOCOM 2018) (2018)
Acknowledgments
The corresponding authors of this paper are Yang Du and Shiping Chen. The research of authors was supported by National Natural Science Foundation of China under Grant No. 62072322, No. 61873177, and No. U20A20182, Natural Science Foundation of Jiangsu Province under Grant No. BK20210706, and Jiangsu Planned Projects for Postdoctoral Research Funds under Grant No. 2021K165B. The research of Guoju Gao was partially supported by the National Natural Science Foundation of China (NSFC) under Grant No. 62102275, the NSF of Jiangsu in China under Grant No. BK20210704, and the NSF of the Jiangsu Higher Education Institutions of China under Grant No. 21KJB520025.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, B. et al. (2022). Multi-layer Adaptive Sampling for Per-Flow Spread Measurement. In: Lai, Y., Wang, T., Jiang, M., Xu, G., Liang, W., Castiglione, A. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2021. Lecture Notes in Computer Science(), vol 13155. Springer, Cham. https://doi.org/10.1007/978-3-030-95384-3_46
Download citation
DOI: https://doi.org/10.1007/978-3-030-95384-3_46
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-95383-6
Online ISBN: 978-3-030-95384-3
eBook Packages: Computer ScienceComputer Science (R0)