Abstract
As a key cloud management problem, Cost-aware Dynamic Multi-Workflow Scheduling (CDMWS) aims to assign virtual machine (VM) instances to execute tasks in workflows so as to minimize the total costs, including both the penalties for violating Service Level Agreement (SLA) and the VM rental fees. Powered by deep neural networks, Reinforcement Learning (RL) methods can construct effective scheduling policies for solving CDMWS problems. Traditional policy networks in RL often use basic feedforward architectures to separately determine the suitability of assigning any VM instances, without considering all VMs simultaneously to learn their global information. This paper proposes a novel self-attention policy network for cloud workflow scheduling (SPN-CWS) that captures global information from all VMs. We also develop an Evolution Strategy-based RL (ERL) system to train SPN-CWS reliably and effectively. The trained SPN-CWS can effectively process all candidate VM instances simultaneously to identify the most suitable VM instance to execute every workflow task. Comprehensive experiments show that our method can noticeably outperform several state-of-the-art algorithms on multiple benchmark CDMWS problems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
ES-RL and SPN-CWS are trained with \(\gamma = 5\) and tested with \(\gamma \in \{1.00, 1.25, 1.50,\) \(1.75, 2.00, 2.25\}\) to evaluate their performance under tight SLA deadline coefficients.
- 3.
References
Ajani, O.S., Mallipeddi, R.: Adaptive evolution strategy with ensemble of mutations for reinforcement learning. Knowl.-Based Syst. 245, 108624 (2022)
Arabnejad, V., Bubendorfer, K., Ng, B.: Budget and deadline aware e-science workflow scheduling in clouds. IEEE Trans. Parallel Distrib. Syst. 30(1), 29–44 (2018)
Chen, G., Qi, J., Sun, Y., Hu, X., Dong, Z., Sun, Y.: A collaborative scheduling method for cloud computing heterogeneous workflows based on deep reinforcement learning. Futur. Gener. Comput. Syst. 141, 284–297 (2023)
Chen, H., Zhu, X., Liu, G., Pedrycz, W.: Uncertainty-aware online scheduling for real-time workflows in cloud service environment. IEEE Trans. Serv. Comput. 14(4), 1167–1178 (2018)
Dong, T., Xue, F., Xiao, C., Zhang, J.: Workflow scheduling based on deep reinforcement learning in the cloud environment. J. Ambient. Intell. Humaniz. Comput. 12(12), 10823–10835 (2021)
Escott, K.-R., Ma, H., Chen, G.: Genetic programming based hyper heuristic approach for dynamic workflow scheduling in the cloud. In: Hartmann, S., Küng, J., Kotsis, G., Tjoa, A.M., Khalil, I. (eds.) DEXA 2020. LNCS, vol. 12392, pp. 76–90. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59051-2_6
Faragardi, H.R., Sedghpour, M.R.S., Fazliahmadi, S., Fahringer, T., Rasouli, N.: Grp-heft: a budget-constrained resource provisioning scheme for workflow scheduling in IAAS clouds. IEEE Trans. Parallel Distrib. Syst. 31(6), 1239–1254 (2019)
Hoseiny, F., Azizi, S., Shojafar, M., Tafazolli, R.: Joint QoS-aware and cost-efficient task scheduling for fog-cloud resources in a volunteer computing system. ACM Trans. Internet Technol. 21(4), 1–21 (2021)
Huang, V., Wang, C., Ma, H., Chen, G., Christopher, K.: Cost-aware dynamic multi-workflow scheduling in cloud data center using evolutionary reinforcement learning. In: Troya, J., Medjahed, B., Piattini, M., Yao, L., Fernández, P., Ruiz-Cortés, A. (eds.) ICSOC 2022. LNCS, vol. 13740, pp. 449–464. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20984-0_32
Jayanetti, A., Halgamuge, S., Buyya, R.: Multi-agent deep reinforcement learning framework for renewable energy-aware workflow scheduling on distributed cloud data centers. IEEE Trans. Parallel Distrib. Syst. (2024)
Khadka, S., Tumer, K.: Evolution-guided policy gradient in reinforcement learning. In: Advances in Neural Information Processing Systems, vol. 31 (2018)
Li, H., Huang, J., Wang, B., Fan, Y.: Weighted double deep q-network based reinforcement learning for bi-objective multi-workflow scheduling in the cloud. Clust. Comput. 25(2), 751–768 (2022)
Liu, J., et al.: Online multi-workflow scheduling under uncertain task execution time in IAAS clouds. IEEE Trans. Cloud Comput. 9(3), 1180–1194 (2019)
Masdari, M., ValiKardan, S., Shahi, Z., Azar, S.I.: Towards workflow scheduling in cloud computing: a comprehensive analysis. J. Netw. Comput. Appl. 66, 64–82 (2016)
Salimans, T., Ho, J., Chen, X., Sidor, S., Sutskever, I.: Evolution strategies as a scalable alternative to reinforcement learning. arxiv 2017. arXiv preprint arXiv:1703.03864 (2017)
Silver, E.A.: An overview of heuristic solution methods. J. Oper. Res. Soc. 55(9), 936–956 (2004)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Wang, Y., et al.: Multi-objective workflow scheduling with deep-q-network-based multi-agent reinforcement learning. IEEE access 7, 39974–39982 (2019)
Wu, L., Garg, S.K., Versteeg, S., Buyya, R.: Sla-based resource provisioning for hosted software-as-a-service applications in cloud computing environments. IEEE Trans. Serv. Comput. 7(3), 465–485 (2013)
Wu, Q., Ishikawa, F., Zhu, Q., Xia, Y., Wen, J.: Deadline-constrained cost optimization approaches for workflow scheduling in clouds. IEEE Trans. Parallel Distrib. Syst. 28(12), 3401–3412 (2017)
Xu, M., et al.: Genetic programming for dynamic workflow scheduling in fog computing. IEEE Trans. Serv. Comput. 16(4), 2657–2671 (2023)
Yang, Y., Chen, G., Ma, H., Hartmann, S., Zhang, M.: Dual-tree genetic programming with adaptive mutation for dynamic workflow scheduling in cloud computing. IEEE Trans. Evol. Comput. (2024)
Yang, Y., Chen, G., Ma, H., Zhang, M.: Dual-tree genetic programming for deadline-constrained dynamic workflow scheduling in cloud. In: Troya, J., Medjahed, B., Piattini, M., Yao, L., Fernández, P., Ruiz-Cortés, A. (eds.) Service-Oriented Computing - ICSOC 2022. Lecture Notes in Computer Science, vol. 13740, pp. 433–448. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20984-0_31
Yang, Y., Chen, G., Ma, H., Zhang, M., Huang, V.: Budget and SLA aware dynamic workflow scheduling in cloud computing with heterogeneous resources. In: 2021 IEEE Congress on Evolutionary Computation (CEC), pp. 2141–2148. IEEE (2021)
Youn, C.H., Chen, M., Dazzi, P.: Cloud broker and cloudlet for workflow scheduling. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-5071-8
Zhou, B., Cheng, L.: Deep reinforcement learning-based scheduling for same day delivery with a dynamic number of drones. In: Monti, F., Rinderle-Ma, S., Ruiz Cortés, A., Zheng, Z., Mecella, M. (eds.) ICSOC 2023. LNCS, vol. 14419, pp. 34–41. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-48421-6_3
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Shen, Y., Chen, G., Ma, H., Zhang, M. (2025). Cost-Aware Dynamic Cloud Workflow Scheduling Using Self-attention and Evolutionary Reinforcement Learning. In: Gaaloul, W., Sheng, M., Yu, Q., Yangui, S. (eds) Service-Oriented Computing. ICSOC 2024. Lecture Notes in Computer Science, vol 15405. Springer, Singapore. https://doi.org/10.1007/978-981-96-0808-9_1
Download citation
DOI: https://doi.org/10.1007/978-981-96-0808-9_1
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-96-0807-2
Online ISBN: 978-981-96-0808-9
eBook Packages: Computer ScienceComputer Science (R0)