Abstract
Elastic scaling in/out of operator parallelism degree is needed for processing real time dynamic data streams under low latency and high stability requirements. Usually the operator parallelism degree is set when a streaming application is submitted to a stream computing system and kept intact during runtime. This may substantially affect the performance of the system due to the fluctuation of input streams and availability of system resources. To address the problems brought by the static parallelism setting, we propose and implement a machine learning based elastic strategy for operator parallelism (named Me-Stream) in big data stream computing systems. The architecture of Me-Stream and its key models are introduced, including parallel bottleneck identification, parameter plan generation, parameter migration and conversion, and instances scheduling. Metrics of execution latency and process latency of the proposed scheduling strategy are evaluated on the widely used big data stream computing system Apache Storm. The experimental results demonstrate the efficiency and effectiveness of the proposed strategy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cao, H., Wu, C.E.Q., Bao, L., Hou, A., Shen, W.: Throughput optimization for Storm-based processing of stream data on clouds. Future Gener. Comput. Syst. 112, 567–579 (2020)
Paris, C., Stephan, E., Gyula, F., Seif, H., Stefan, R., Kostas, T.: State management in Apache Flink: consistent stateful distributed stream processing. Proc. VLDB Endow. 10(12), 1718–1729 (2017)
Apache, Storm. http://storm.apache.org
Flink. https://flink.apache.org/
Spark Streaming. https://spark.apache.org/streaming/
Samza. http://samza.apache.org/
Apex. https://apex.apache.org/
Google Cloud Dataflow. https://cloud.google.com/dataflow/
Deng, S., Wang, B., Huang, S., Yue, C., Zhou, J., Wang, G.: Self-adaptive framework for efficient stream data classification on storm. IEEE Trans. Syst. Man Cybern. Syst. 50(1), 123–136 (2020)
Li, C., Zhang, J., Luo, Y.: Real-time scheduling based on optimized topology and communication traffic in distributed real-time computation platform of storm. J. Netw. Comput. Appl. 87, 100–115 (2017)
Muhammad, A., Aleem, M., Islam, M.A.: TOP-Storm: a topology-based resource-aware scheduler for Stream Processing Engine. Cluster Comput. 24(1), 417–431 (2020). https://doi.org/10.1007/s10586-020-03117-y
Pathan, R., Voudouris, P., Stenstrom, P.: Scheduling parallel real-time recurrent tasks on multicore platforms. IEEE Trans. Parallel Distrib. Syst. 29(4), 915–928 (2018)
Li, H., Wu, J., Jiang, Z., Li, X., Wei, X.: Task allocation for stream processing with recovery latency guarantee. In: Proceedings of the 2017 IEEE International Conference on Cluster Computing, CLUSTER 2017, pp. 379–383. IEEE Press,September 2017
Zhang, J., Li, C., Zhu, L., Liu, Y.: The real-time scheduling strategy based on traffic and load balancing in storm. In: Proceedings of the 18th IEEE International Conference on High Performance Computing and Communications, HPCC 2016, pp. 372–379. IEEE Press, January 2017
Muhammad, A., Aleem, M.: A3-Storm: topology-, traffic-, and resource-aware storm scheduler for heterogeneous clusters. J. Supercomput. 77(2), 1059–1093 (2020). https://doi.org/10.1007/s11227-020-03289-9
You, Y., Demmel, J.: Runtime data layout scheduling for machine learning dataset. In: Proceedings of the 46th International Conference on Parallel Processing, ICPP 2017, pp. 452–461. IEEE Press,September 2017
Al-Sinayyid, A., Zhu, M.: Job scheduler for streaming applications in heterogeneous distributed processing systems. J. Supercomput. 76(12), 9609–9628 (2020). https://doi.org/10.1007/s11227-020-03223-z
Cheng, D., Wang, Y.: Adaptive scheduling parallel jobs with dynamic batching in spark streaming. IEEE Trans. Parallel Distrib. Syst. 29(12), 2672–2685 (2018)
Wei, X.: Pec: proactive elastic collaborative resource scheduling in data stream processing. IEEE Trans. Parallel Distrib. Syst. 30(7), 1628–1642 (2019)
Wang, W., Zhang, C.:An on-the-fly scheduling strategy for distributed stream processing platform. In: IEEE International Conference on Parallel & Distributed Processing with Applications, Ubiquitous Computing & Communications, Big Data & Cloud Computing, Social Computing & Networking, Sustainable Computing & Communications (2018)
TawfiqulIslam, M., Karunasekera, S., Buyya, R.: dSpark: deadline-based resource allocation for big data applicationsin apache spark. In: IEEE 13th International Conference on e-Science, 24–27 October 2017
Acknowledgements
This work is supported by the National Natural Science Foundation of China under Grant No. 61972364, the Fundamental Research Funds for the Central Universities under Grant No. 2652021001, and Melbourne-Chindia Cloud Computing (MC3) Research Network.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Li, W., Sun, D., Gao, S., Buyya, R. (2022). A Machine Learning-Based Elastic Strategy for Operator Parallelism in a Big Data Stream Computing System. In: Xiang, W., Han, F., Phan, T.K. (eds) Broadband Communications, Networks, and Systems. BROADNETS 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 413. Springer, Cham. https://doi.org/10.1007/978-3-030-93479-8_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-93479-8_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-93478-1
Online ISBN: 978-3-030-93479-8
eBook Packages: Computer ScienceComputer Science (R0)