Abstract
With the explosive growth of IoT and other interactive network services, billions of devices are now connected, leading to highly fluctuating traffic and diverse QoS requirements for servers. This, coupled with the C10M problem, means benchmarks for interactive services should be able to handle millions of concurrency, bursty load and multiple QoS evaluation. However, existing general benchmarks for network services cannot fully meet these requirements.
To address this issue, we propose MCCBench as a benchmark for high concurrent interactive network services. MCCBench includes a methodology for load generation, service framework, and service performance evaluation, allowing for the measurement of over 10 million concurrent connections, bursty loads, and labeling of requests with different service qualities. The performance evaluation metrics include tail latency measured on the server side, and long-lived concurrent connections. To implement MCCBench, we have developed an open-source toolset called MCCBench-IoT, which includes a load generator, an IoT service system based on a user-space network stack, and an accurate monitor for measuring tail latency.
We verified MCCBench by building a testbed with MCCBench-IoT to emulate a typical IoT service, successfully testing tail latency under a concurrency of 10.2 million on a single server node. The testbed was scaled to 300 million concurrency with cluster configuration. By providing a comprehensive benchmark for high-concurrent interactive network services, MCCBench can help improve the quality of service for such services and enable better decision-making for network infrastructure design and optimization.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Smarter Planet. https://www.ibm.com/ibm/history/ibm100/us/en/icons/smarterplanet/. Accessed 4 Jul 2022
Global Smart Transportation Market Size Report (2030). https://www.grandviewresearch.com/industry-analysis/smart-transportation-market. Accessed 4 Jul 2022
Rzadca, K., et al: Autopilot: workload autoscaling at Google. In: Proceedings of the Fifteenth European Conference on Computer Systems (EuroSys 2020). Association for Computing Machinery, pp. 1–16. 2020. https://doi.org/10.1145/3342195.3387524
Botta, A., De Donato, W., Persico, V., Pescapé, A.: Integration of cloud computing and internet of things: a survey. Future Gener. Comput. Syst. 56, 684–700 (2016). https://doi.org/10.1016/j.future.2015.09.021
Roy, A., Zeng, H., Bagga, J., Porter, G., Snoeren, A.C.: Inside the social network’s (datacenter) network. In: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication (SIGCOMM 2015), pp. 123–137. Association for Computing Machinery (2015). https://doi.org/10.1145/2785956.2787472
Huang, D.Y., Apthorpe, N., Li, F., Acar, G., Feamster, N.: IoT inspector: crowdsourcing labeled network traffic from smart home devices at scale. In: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, pp 1–21. ACM (2020).https://doi.org/10.1145/3397333
Lu, S., Yao, Y., Shi, W.: Collaborative learning on the edges: a case study on connected vehicles. In: 2nd USENIX Workshop on Hot Topics in Edge Computing (HotEdge 19) (2019)
De Domenico, M., Altmann, E.G.: Unraveling the origin of social bursts in collective attention. Sci. Rep. 10, 1–9 (2020)
Tadakamalla, U., Menascé, D.A.: Characterization of IoT workloads. In: Zhang, T., Wei, J., Zhang, L.-J. (eds.) EDGE 2019. LNCS, vol. 11520, pp. 1–15. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23374-7_1
Abdel-Basset, M., Ding, W., Abdel-Fatah, L.: The fusion of internet of intelligent things (IoIT) in remote diagnosis of obstructive Sleep Apnea: a survey and a new model. Inf. Fusion. 61, 84–100 (2020). https://doi.org/10.1016/j.inffus.2020.03.010
Jianfeng, Z.: Call for establishing benchmark science and engineering. arXiv preprint arXiv:2112.09514 (2021)
I. BIPM, I. IFCC, I. IUPAC, O. ISO, The international vocabulary of metrology-basic and general concepts and associated terms (VIM), 3rd edn. JCGM 200: 2012, in: JCGM (Joint Committee for Guides in Metrology) (2012)
Worldwide Smart Home Devices Market Grew 11.7% in 2021 with Double-Digit Growth Forecast Through 2026, According to IDC. https://www.idc.com/getdoc.jsp?containerId=prUS49051622. Accessed 4 Jul 2022
TPC-H. https://www.tpc.org/tpch/. Accessed 15 Jul 2022
TPC-C. https://www.tpc.org/tpcc/. Accessed 15 Jul 2022
TeraSort benchmark. https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/examples/terasort/. Accessed 15 Jul 2022
The C10K problem. http://www.kegel.com/c10k.html#related. Accessed 15 Jul 2022
C10M. http://c10m.robertgraham.com/p/blog-page.html. Accessed 15 Jul 2022
Wrk. https://github.com/wg/wrk.git. Accessed 15 Jul 2022
Wu, W., Feng, X., Zhang, W., Chen, M.: MCC: a predictable and scalable massive client load generator. In: Gao, W., Zhan, J., Fox, G., Lu, X., Stanzione, D. (eds.) Bench 2019. LNCS, vol. 12093, pp. 319–331. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49556-5_29
Netperf. http://www.cs.kent.edu/~farrell/dist/ref/Netperf.html. Accessed 15 Jul 2022
Redis-5.0.4. https://download.redis.io/releases/redis-5.0.14.tar.gz. Accessed 15 Jul 2022
Nginx. http://nginx.org/en/index.html. Accessed 15 Jul 2022
MCC. https://github.com/acs-network/mcc. Accessed 15 Jul 2022
Migratorydata server. http://migratorydata.com/. Accessed 15 Jul 2022
Zheng, C., Tang, Q., Lu, Q., Li, J., Zhou, Z., Liu, Q.: Janus: a user-level TCP stack for processing 40 million concurrent TCP connections. In: IEEE International Conference on Communications (ICC), pp. 1–7. IEEE (2018). https://doi.org/10.1109/ICC.2018.8422993
Yilmaz, Y.S., Aydin, B.I., Demirbas, M.: Google Cloud Messaging (GCM): an evaluation. In: 2014 IEEE Global Communications Conference, pp. 2807–2812 (2014). https://doi.org/10.1109/GLOCOM.2014.7037233
Ixia breakingpoint. https://www.ixiacom.com/products/breakingpoint. Accessed 4 Jul 2022
Spirent TestCenter Benchmarking. https://www.spirent.cn/assets/u/datasheet-spirent-testcenter-benchmarking-bundle. Accessed 4 Jul 2022
ab-Apache HTTP server benchmarking tool. https://httpd.apache.org/docs/2.0/programs/ab.html. Accessed 4 Jul 2022
Chen, S., Delimitrou, C., Martínez, J.F.: PARTIES: QoS-aware resource partitioning for multiple interactive services. In: Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2019), pp. 107–120. Association for Computing Machinery, New York, NY, USA (2019)
Kasture, H., Sanchez, D.: Tailbench: a benchmark suite and evaluation methodology for latency-critical applications. In: 2016 IEEE International Symposium on Workload Characterization (IISWC), pp. 1–10. IEEE (2016). https://doi.org/10.1109/IISWC.2016.7581261
Zhang, Y., Meisner, D., Mars, J., Tang, L.: Treadmill: attributing the source of tail latency through precise load testing and statistical inference. In: Proceedings of the 43rd International Symposium on Computer Architecture (ISCA 2016), pp. 456–468. IEEE Press (2016). https://doi.org/10.1109/ISCA.2016.47
Dean, J., Barroso, L.A.: The tail at scale. Commun. ACM 56, 74–80 (2013). https://doi.org/10.1145/2408776.2408794
Vulimiri, A., Godfrey, P.B., Mittal, R., Sherry, J., Ratnasamy, S., Shenker, S.: Low latency via redundancy. In: Proceedings of the ninth ACM conference on Emerging networking experiments and technologies, pp. 283–294 (2013)
Lindgaard, G., Fernandes, G., Dudek, C., Brown, J.: Attention web designers: you have 50 milliseconds to make a good first impression! Behav. Inf. Technol. 25(2), 115–126 (2006)
Wu, W., Feng, X., Zhang, W., Chen, M.: MCC: a predictable and scalable massive client load generator. In: Gao, W., Zhan, J., Fox, G., Lu, X., Stanzione, D. (eds.) Bench 2019. LNCS, vol. 12093, pp. 319–331. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49556-5_29
Qstack. https://github.com/acs-network/Qstack. Accessed 4 Jul 2022
QStack: Re-architecting User-space Network Stack to Optimize CPU Efficiency and Service Quality. https://arxiv.org/abs/2210.08432. Accessed 19 Oct 2022
HCMonitor. https://github.com/acs-network/hcmonitor. Accessed 4 Jul 2022
Song, H., Zhang, W., Liu, K., Shen, Y., Chen, M.: HCMonitor: an accurate measurement system for high concurrent network services. Concurrency Comput. Pract. Experience, 34(12), e6081. https://doi.org/10.1002/cpe.6081
Acknowledgment
Thanks to Ms. Xiaohong Wang for her valuable support and suggestions on the use of OneITLab. The work was supported by the National Key Research and Development Plan of China under Grant No. 2022YFB4500403, the Strategic Priority Research Program of the Chinese Academy of Sciences under Grant No. XDA0320000 and XDA0320300.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Song, H., Zhang, W., Chen, M. (2023). MCCBench: A C10M Benchmark Oriented to Interactive Network Services. In: Gainaru, A., Zhang, C., Luo, C. (eds) Benchmarking, Measuring, and Optimizing. Bench 2022. Lecture Notes in Computer Science, vol 13852. Springer, Cham. https://doi.org/10.1007/978-3-031-31180-2_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-31180-2_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-31179-6
Online ISBN: 978-3-031-31180-2
eBook Packages: Computer ScienceComputer Science (R0)