Abstract
There are a large number of real-time streaming data task processing workflows in online anomaly detection and intelligent transportation systems. Real-time data transmission is required between these processing tasks. To ensure real-time performance, it is inevitable for real-time workflow scheduling. However, due to the connected sensor data being updated in real-time, the task load of the workflow will fluctuate. In this case, the original scheduling plan is no longer optimal and needs to be renewed. Adjustments in scheduling plan bring new challenges to dynamic scheduling. At the same time, when the scheduling plan changes, the deep learning models and data used in these workflow tasks need to be redeployed at a high cost. To address these challenges, we use the data generating rate to express the task load’s fluctuation. Besides, we also consider the migration cost caused by the adjustment of the scheme in the continuous scheduling process during modeling and optimization. To achieve a low-cost scheduling scheme in a short time, we propose the NSGA-II-Seq2Seq algorithm which employs the historical scheduling plan to generate a candidate scheduling plan. The experiments are carried out in the modified WorkflowSim. Through experiment, it is found that the method proposed in this paper can adapt to the change of task load and can produce an adaptively fine-tuning scheduling scheme. In multiple consecutive scheduling experiments, the population optimization process of NSGA is accelerated, which significantly reduces the time to obtain the scheduling plan.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Dayarathna M, Suzumura T (2013) Automatic optimization of stream programs via source program operator graph transformations. Distrib Parallel Databases 31(4):543–599. https://doi.org/10.1007/s10619-013-7130-x
Zeng X-Q, Li G-Z (2014) Incremental partial least squares analysis of big streaming data. Pattern Recogn. https://doi.org/10.1016/j.patcog.2014.05.022
Salinas S, Chen X, Ji J, Li P (2016) A tutorial on secure outsourcing of large-scale computations for big data. IEEE Access 4:1–1. https://doi.org/10.1109/ACCESS.2016.2549982
Xie Y et al (2019) A novel directional and non-local-convergent particle swarm optimization based workflow scheduling in cloud-edge environment. Futur Gener Comput Syst 97:361–378. https://doi.org/10.1016/j.future.2019.03.005
Barika M, Garg S, Ranjan R (2020) Cost effective stream workflow scheduling to handle application structural changes. Futur Gener Comput Syst 112:348–361. https://doi.org/10.1016/j.future.2020.05.036
Hou S et al (2017) A distributed deployment algorithm of process fragments with uncertain traffic matrix. IEEE Trans Netw Serv Manage PP:1–1. https://doi.org/10.1109/TNSM.2017.2728863
Li W, Xia Y, Zhou M, Sun X, Zhu Q (2018) Fluctuation-aware and predictive workflow scheduling in cost-effective infrastructure-as-a-service clouds. IEEE Access PP:1–1. https://doi.org/10.1109/ACCESS.2018.2869827
Pan Y et al (2020) A novel approach to scheduling workflows upon cloud resources with fluctuating performance. Mob Netw Appl 25(2):690–700. https://doi.org/10.1007/s11036-019-01450-0
Pan Y et al (2020) A stochastic-performance-distribution-based approach to cloud workflow scheduling with fluctuating performance. In: Ku W-S, Kanemasa Y, Serhani MA, Zhang L-J (eds) Web Services—ICWS 2020, Lecture Notes in Computer Science. Springer International Publishing, Cham, pp 33–48
Liu H et al (2020) Scheduling multi-workflows over edge computing resources with time-varying performance, a novel probability-mass function and DQN-based approach. In: Ku W-S, Kanemasa Y, Serhani MA, Zhang L-J (edsWeb Services—ICWS 2020, Lecture Notes in Computer Science. Springer International Publishing, Cham, pp 197–209
Ismayilov G, Topcuoglu HR (2020) Neural network based multi-objective evolutionary algorithm for dynamic workflow scheduling in cloud computing. Futur Gener Comput Syst 102:307–322. https://doi.org/10.1016/j.future.2019.08.012
Liu X-F, Zhan Z-H, Zhang J (2018) Neural network for change direction prediction in dynamic optimization. IEEE Access PP:1–1. https://doi.org/10.1109/ACCESS.2018.2881538
Barika M, Garg S, Ranjan R (2019) Adaptive scheduling for efficient execution of dynamic stream workflows. Tech. Rep. 2019 arXiv:1912.08397
Xu X, Fu S, Yuan Y, Qi L, Dou W (2018) Energy-efficient computation offloading in cloudlet-based mobile cloud using NSGA-II, pp 1–6
Cho K et al (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. https://doi.org/10.3115/v1/D14-1179
Xiao Y et al (2021) History-based attention in Seq2Seq model for multi-label text classification. Knowl-Based Syst 224:107094. https://doi.org/10.1016/j.knosys.2021.107094
Konak A, Coit DW, Smith AE (2006) Multi-objective optimization using genetic algorithms: a tutorial. Reliabil Eng Syst Saf 91(9):992–1007. https://doi.org/10.1016/j.ress.2005.11.018
Chen W, Deelman E (2012) WorkflowSim: a toolkit for simulating scientific workflows in distributed environments, pp 1–8
Juve G et al (2013) Characterizing and profiling scientific workflows. Futur Gener Comput Syst 29(3):682–692. https://doi.org/10.1016/j.future.2012.08.015
Acknowledgements
This work is supported by National Natural Science Foundation of China (Grant no. 61832004), and Projects of International Cooperation and Exchanges NSFC(Grant no. 62061136006).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhang, M., Yang, Z., Yan, J. et al. Task-load aware and predictive-based workflow scheduling in cloud-edge collaborative environment. J Reliable Intell Environ 8, 35–47 (2022). https://doi.org/10.1007/s40860-022-00173-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40860-022-00173-6