Yang Zhou
Skip Abstract Section
Abstract
Network measurement plays an important role for many network functions such as detecting network anomalies and identifying big flows. However, most existing measurement solutions fail to achieve high performance in software as they often incorporate heavy computations and a large number of random memory accesses. We present Agg-Evict, a generic framework for accelerating network measurement in software. Agg-Evict aggregates the incoming packets on the same flows and sends them as a batch, reducing the number of computations and random memory accesses in the subsequent measurement solutions. We perform extensive experiments on top of DPDK with 10G NIC and observe that almost all the tested measurement solutions under Agg-Evict can achieve 14.88 Mpps throughput and see up to 5.7X lower average processing latency per packet.
- Source code and experiment details related to agg-evict. https://github.com/zhouyangpkuer/Agg-Evict.Google Scholar
- Amin Vahdat. 2014. Enter the Andromeda zone - Google Cloud Platform's Latest Networking Stack. http://goo.gl/smN6W0.Google Scholar
- Data Plane Development Kit (DPDK). http://dpdk.org/.Google Scholar
- Intel SSE2 Documentation. https://software.intel.com/en-us/node/683883.Google Scholar
- The CAIDA anonymized 2016 internet traces. http://www.caida.org/data/passive/passive_2016_dataset.xml.Google Scholar
- Zero packet loss tests. https://www.ietf.org/rfc/rfc2544.txt.Google Scholar
- O. Alipourfard, M. Moshref, and M. Yu. Re-evaluating measurement algorithms in software. In HotNets, page 20. ACM, 2015. Google ScholarDigital Library
- T. Benson, A. Akella, and D. A. Maltz. Network traffic characteristics of data centers in the wild. In IMC, pages 267--280. ACM, 2010. Google ScholarDigital Library
- B. H. Bloom. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7):422--426, 1970. Google ScholarDigital Library
- G. Cormode. Sketch techniques for approximate query processing. Foundations and Trends in Databases. NOW publishers, 2011.Google Scholar
- G. Cormode and S. Muthukrishnan. An improved data stream summary: the count-min sketch and its applications. Journal of Algorithms, 55(1):58--75, 2005. Google ScholarDigital Library
- H. Dai, L. Meng, and A. X. Liu. Finding persistent items in distributed, datasets. In Proc. IEEE INFOCOM, 2018.Google ScholarCross Ref
- H. Dai, M. Shahzad, A. X. Liu, and Y. Zhong. Finding persistent items in data streams. Proceedings of the VLDB Endowment, 10(4):289--300, 2016. Google ScholarDigital Library
- H. Dai, Y. Zhong, A. X. Liu, W. Wang, and M. Li. Noisy bloom filters for multi-set membership testing. In Proc. ACM SIGMETRICS, pages 139--151, 2016. Google ScholarDigital Library
- X. Dimitropoulos, P. Hurley, and A. Kind. Probabilistic lossy counting: an efficient algorithm for finding heavy hitters. ACM SIGCOMM CCR, 38(1):5--5, 2008. Google ScholarDigital Library
- P. Flajolet and G. N. Martin. Probabilistic counting algorithms for data base applications. Journal of computer and system sciences, 31(2):182--209, 1985. Google ScholarDigital Library
- M. T. Goodrich and M. Mitzenmacher. Invertible bloom lookup tables. In Proceedings of the 49th Annual Allerton Conference on Communication, Control, and Computing, pages 792--799. IEEE, 2011.Google ScholarCross Ref
- S. Heule, M. Nunkesser, and A. Hall. Hyperloglog in practice: algorithmic engineering of a state of the art cardinality estimation algorithm. In Proceedings of the 16th International Conference on Extending Database Technology, pages 683--692. ACM, 2013. Google ScholarDigital Library
- Q. Huang, X. Jin, P. P. Lee, R. Li, L. Tang, Y.-C. Chen, and G. Zhang. Sketchvisor: Robust network measurement for software packet processing. In SIGCOMM, pages 113--126. ACM, 2017. Google ScholarDigital Library
- B. Krishnamurthy, S. Sen, Y. Zhang, and Y. Chen. Sketch-based change detection: methods, evaluation, and applications. In Proc. ACM IMC, pages 234--247. ACM, 2003. Google ScholarDigital Library
- A. Kumar, M. Sung, J. J. Xu, and J. Wang. Data streaming algorithms for efficient and accurate estimation of flow size distribution. In ACM SIGMETRICS Performance Evaluation Review, volume 32, pages 177--188. ACM, 2004. Google ScholarDigital Library
- A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang. Data streaming algorithms for estimating entropy of network traffic. In ACM SIGMETRICS Performance Evaluation Review, volume 34, pages 145--156. ACM, 2006. Google ScholarDigital Library
- B. Li, E. Mazur, Y. Diao, A. McGregor, and P. Shenoy. A platform for scalable one-pass analytics using mapreduce. In Proc. ACM SIGMOD, pages 985--996. ACM, 2011. Google ScholarDigital Library
- Y. Li, R. Miao, C. Kim, and M. Yu. Flowradar: a better netflow for data centers. In NSDI, pages 311--324. USENIX Association, 2016. Google ScholarDigital Library
- Z. Liu, A. Manousis, G. Vorsanger, V. Sekar, and V. Braverman. One sketch to rule them all: Rethinking network flow monitoring with univmon. In SIGCOMM, pages 101--114. ACM, 2016. Google ScholarDigital Library
- A. Metwally, D. Agrawal, and A. El Abbadi. Efficient computation of frequent and top-k elements in data streams. In International Conference on Database Theory, pages 398--412. Springer, 2005. Google ScholarDigital Library
- S. Meyers. Cpu caches and why you care, 2013. (Cited on slice 20).Google Scholar
- M. Moshref, M. Yu, R. Govindan, and A. Vahdat. Trumpet: Timely and precise triggers in data centers. In SIGCOMM, pages 129--143. ACM, 2016. Google ScholarDigital Library
- S. Narayana, A. Sivaraman, V. Nathan, P. Goyal, V. Arun, M. Alizadeh, V. Jeyakumar, and C. Kim. Language-directed hardware design for network performance monitoring. In Proc. ACM SIGCOMM, pages 85--98. ACM, 2017. Google ScholarDigital Library
- P. Patel, D. Bansal, L. Yuan, A. Murthy, A. Greenberg, D. A. Maltz, R. Kern, H. Kumar, M. Zikos, H. Wu, et al. Ananta: Cloud scale load balancing. In ACM SIGCOMM CCR, volume 43, pages 207--218. ACM, 2013. Google ScholarDigital Library
- D. M. Powers. Applications and explanations of Zipf's law. In Proc. EMNLP-CoNLL. Association for Computational Linguistics, 1998. Google ScholarDigital Library
- R. Schweller, A. Gupta, E. Parsons, and Y. Chen. Reversible sketches for efficient and accurate change detection over network data streams. In IMC, pages 207--212. ACM, 2004. Google ScholarDigital Library
- R. Schweller, Z. Li, Y. Chen, et al. Reversible sketches: enabling monitoring and analysis over high-speed data streams. IEEE/ACM Transactions on Networking (ToN), 15(5):1059--1072, 2007. Google ScholarDigital Library
- V. Sekar, N. G. Duffield, O. Spatscheck, J. E. van der Merwe, and H. Zhang. Lads: Large-scale automated ddos detection system. In ATC, pages 171--184. USENIX Association, 2006. Google ScholarDigital Library
- K.-Y. Whang, B. T. Vander-Zanden, and H. M. Taylor. A linear-time probabilistic counting algorithm for database applications. ACM Transactions on Database Systems (TODS), 15(2):208--229, 1990. Google ScholarDigital Library
- K. Xie, X. Li, X. Wang, and et al. On-line anomaly detection with high accuracy. IEEE/ACM Transactions on Networking, 2018. Google ScholarDigital Library
- T. Yang, J. Gong, H. Zhang, S. L. Zou, Lei, and X. Li. Heavyguardian: Separate and guard hot items in data streams. In Proc. ACM SIGKDD, 2018. Google ScholarDigital Library
- T. Yang, J. Jiang, P. Liu, Q. Huang, J. Gong, Y. Zhou, R. Miao, X. Li, and S. Uhlig. Elastic sketch: Adaptive and fast network-wide measurements. In Proc. ACM SIGCOMM, 2018. Google ScholarDigital Library
- T. Yang, A. X. Liu, M. Shahzad, D. Yang, Q. Fu, G. Xie, and X. Li. A shifting framework for set queries. IEEE/ACM Transactions on Networking, 25(5):3116--3131, 2017. Google ScholarDigital Library
- T. Yang, L. Wang, Y. Shen, M. Shahzad, Q. Huang, X. Jiang, K. Tan, and X. Li. Empowering sketches with machine learning for network measurements. In Proc. ACM SIGCOMM workshop on NetAI, 2018. Google ScholarDigital Library
- M. Yu, L. Jose, and R. Miao. Software defined traffic measurement with opensketch. In NSDI, pages 29--42, 2013. Google ScholarDigital Library
- Y. Zhou, T. Yang, J. Jiang, B. Cui, M. Yu, X. Li, and S. Uhlig. Cold filter: A meta-framework for faster and more accurate stream processing. In Proc. ACM SIGMOD, 2018. Google ScholarDigital Library
Index Terms
- Accelerating network measurement in software
Recommendations
SketchVisor: Robust Network Measurement for Software Packet Processing
SIGCOMM '17: Proceedings of the Conference of the ACM Special Interest Group on Data CommunicationNetwork measurement remains a missing piece in today's software packet processing platforms. Sketches provide a promising building block for filling this void by monitoring every packet with fixed-size memory and bounded errors. However, our analysis ...
Network measurement for 100 GbE network links using multicore processors
Network measurement has been playing a crucial role in network operations, since it cannot only detect the anomalies, but also facilitate traffic engineering. With the fast development of high speed network of 100Gbps and beyond, how to efficiently ...
Accuracy Improvement for Spatial Composition-Based End-to-End Network Measurement
ITNG '15: Proceedings of the 2015 12th International Conference on Information Technology - New GenerationsSpatial composition method for end-to-end network measurement avoids lengthy measurements of a path and the path is divided into some sub-paths when multiple measured paths share the underlay network route. The performance of the overall path is ...
Comments