Skip to main content
SpringerLink
Log in

Knowledge-based multi-objective estimation of distribution algorithm for solving reliability constrained cloud workflow scheduling

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

With the rapid development of cloud computing, numerous large-scale workflow are executed in the cloud environment. Therefore, the workflow scheduling in cloud environment has become an emerging topic. This paper focuses on a reliability constrained multi-objective workflow scheduling problem (RCMOWSP) with the objectives of minimum execution cost and time. To solve the RCMOWSP, this paper proposes a knowledge-based multi-objective estimation of distribution algorithm (KMOEDA) with several problem-specific operators. First, an idle time-based decoding scheme is applied to sort the permutation of tasks greedily. In the global search strategy, a probability model is constructed to improve the diversity of population. Based on the problem-specific knowledge, a reliability-aware local search strategy is designed to performs local search around the solutions that violate reliability constraint. An elite enhancement strategy with a task perturbation operator and a resource perturbation operator is introduced to further improve the elite non-dominated solutions in the external archive. A comprehensive experiment is conducted to verify the performance of KMOEDA. The comparative results show that the KMOEDA significantly outperforms several relative multi-objective workflow scheduling approaches in solving the RCMOWSP.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The data presented in this study are available on request from the corresponding author.

References

  1. Basu, A., Kumar, A.: Research commentary: workflow management issues in e-business. Info. Syst. Res. 13(1), 1–14 (2002)

    MathSciNet  Google Scholar 

  2. Berriman, G.B., Deelman, E., Good, J.C., Jacob, J.C., Katz, D.S., Kesselman, C., Laity, A.C., Prince, T.A., Singh, G., Su, M.-H.: Montage: a grid-enabled engine for delivering custom science-grade mosaics on demand. In: Optimizing Scientific Return for Astronomy Through Information Technologies, pp. 221–232. SPIE (2004)

    Google Scholar 

  3. Deelman, E., Gannon, D., Shields, M., Taylor, I.: Workflows and e-science: an overview of workflow system features and capabilities. Future Gener. Comput. Syst. 25(5), 528–540 (2009)

    Google Scholar 

  4. Rizvi, N., Ramesh, D.: Hbdcws: heuristic-based budget and deadline constrained workflow scheduling approach for heterogeneous clouds. Soft Comput. 24(24), 18971–18990 (2020)

    Google Scholar 

  5. Du, M., Wang, Y., Ye, K., Xu, C.: Algorithmics of cost-driven computation offloading in the edge-cloud environment. IEEE Trans. Comput. 69(10), 1519–1532 (2020)

    MathSciNet  Google Scholar 

  6. Lin, B., Huang, Y., Zhang, J., Hu, J., Chen, X., Li, J.: Cost-driven off-loading for DNN-based applications over cloud, edge, and end devices. IEEE Trans. Ind. Info. 16(8), 5456–5466 (2019)

    Google Scholar 

  7. Ye, K., Shen, H., Wang, Y., Xu, C.: Multi-tier workload consolidations in the cloud: profiling, modeling and optimization. IEEE Trans. Cloud Comput. (2020)

  8. Lin, B., Zhu, F., Zhang, J., Chen, J., Chen, X., Xiong, N.N., Mauri, J.L.: A time-driven data placement strategy for a scientific workflow combining edge computing and cloud computing. IEEE Trans. Ind. Info. 15(7), 4254–4265 (2019)

    Google Scholar 

  9. Wang, W., Jiang, Y., Wu, W.: Multiagent-based resource allocation for energy minimization in cloud computing systems. IEEE Trans. Syst. Man Cybern. Syst. 47(2), 205–220 (2016)

    Google Scholar 

  10. Sousa, E., Lins, F., Tavares, E., Cunha, P., Maciel, P.: A modeling approach for cloud infrastructure planning considering dependability and cost requirements. IEEE Trans. Syst. Man Cybern. Syst. 45(4), 549–558 (2014)

    Google Scholar 

  11. Shirvani, M.H.: A hybrid meta-heuristic algorithm for scientific workflow scheduling in heterogeneous distributed computing systems. Eng. Appl. Artif. Intel. 90, 103501 (2020)

    Google Scholar 

  12. Qin, S., Pi, D., Shao, Z., Xu, Y.: A knowledge-based adaptive discrete water wave optimization for solving cloud workflow scheduling. IEEE Trans. Cloud Comput. (2021)

  13. Qin, S., Pi, D., Shao, Z., Xu, Y.: Hybrid collaborative multi-objective fruit fly optimization algorithm for scheduling workflow in cloud environment. Swarm Evol. Comput. 68, 101008 (2022)

    Google Scholar 

  14. Zhu, J., Li, X., Ruiz, R., Li, W., Huang, H., Zomaya, A.Y.: Scheduling periodical multi-stage jobs with fuzziness to elastic cloud resources. IEEE Trans. Parallel Distrib. Syst. 31(12), 2819–2833 (2020)

    Google Scholar 

  15. Li, Z., Chang, V., Hu, H., Hu, H., Li, C., Ge, J.: Real-time and dynamic fault-tolerant scheduling for scientific workflows in clouds. Info. Sci. 568, 13–39 (2021)

    Google Scholar 

  16. Zhu, Z., Zhang, G., Li, M., Liu, X.: Evolutionary multi-objective workflow scheduling in cloud. IEEE Trans. Parallel Distrib. Syst. 27(5), 1344–1357 (2015)

    Google Scholar 

  17. Faragardi, H.R., Sedghpour, M.R.S., Fazliahmadi, S., Fahringer, T., Rasouli, N.: Grp-heft: a budget-constrained resource provisioning scheme for workflow scheduling in IAAS clouds. IEEE Trans. Parallel Distrib. Syst. 31(6), 1239–1254 (2019)

    Google Scholar 

  18. Jia, Y.-H., Chen, W.-N., Yuan, H., Gu, T., Zhang, H., Gao, Y., Zhang, J.: An intelligent cloud workflow scheduling system with time estimation and adaptive ant colony optimization. IEEE Trans. Syst. Man Cybern. Syst. 51(1), 634–649 (2018)

    Google Scholar 

  19. Li, H., Wang, D., Zhou, M., Fan, Y., Xia, Y.: Multi-swarm co-evolution based hybrid intelligent optimization for bi-objective multi-workflow scheduling in the cloud. IEEE Trans. Parallel Distrib. Syst. 33(9), 2183–2197 (2021)

    Google Scholar 

  20. Wang, Y., Zuo, X.: An effective cloud workflow scheduling approach combining PSO and idle time slot-aware rules. IEEE/CAA J. Automatica Sinica 8(5), 1079–1094 (2021)

    Google Scholar 

  21. Garg, R., Mittal, M., Son, L.H.: Reliability and energy efficient workflow scheduling in cloud environment. Cluster Comput. 22(4), 1283–1297 (2019)

    Google Scholar 

  22. Topcuoglu, H., Hariri, S., Wu, M.-Y.: Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans. Parallel Distrib. Syst. 13(3), 260–274 (2002)

    Google Scholar 

  23. Wang, Z., Hayat, M.M., Ghani, N., Shaban, K.B.: Optimizing cloud-service performance: efficient resource provisioning via optimal workload allocation. IEEE Trans. Parallel Distrib. Syst. 28(6), 1689–1702 (2016)

    Google Scholar 

  24. Topcuoglu, H., Hariri, S., Wu, M.-Y.: Task scheduling algorithms for heterogeneous processors. In: Proceedings Eighth Heterogeneous Computing Workshop (HCW’99), pp. 3–14. IEEE (1999)

    Google Scholar 

  25. De Coninck, E., Verbelen, T., Vankeirsbilck, B., Bohez, S., Simoens, P., Dhoedt, B.: Dynamic auto-scaling and scheduling of deadline constrained service workloads on IAAS clouds. J. Syst. Softw. 118, 101–114 (2016)

    Google Scholar 

  26. Duan, R., Prodan, R., Li, X.: Multi-objective game theoretic scheduling of bag-of-tasks workflows on hybrid clouds. IEEE Trans. Cloud Comput. 2(1), 29–42 (2014)

    Google Scholar 

  27. Li, S., Zhou, Y., Jiao, L., Yan, X., Wang, X., Lyu, M.R.-T.: Towards operational cost minimization in hybrid clouds for dynamic resource provisioning with delay-aware optimization. IEEE Trans. Serv. Comput. 8(3), 398–409 (2015)

    Google Scholar 

  28. Bittencourt, L.F., Madeira, E.R., Da Fonseca, N.L.: Scheduling in hybrid clouds. IEEE Commun. Magaz. 50(9), 42–47 (2012)

    Google Scholar 

  29. Meng, S., Huang, W., Yin, X., Khosravi, M.R., Li, Q., Wan, S., Qi, L.: Security-aware dynamic scheduling for real-time optimization in cloud-based industrial applications. IEEE Trans. Ind. Info. 17(6), 4219–4228 (2020)

    Google Scholar 

  30. Lu, P., Sun, Q., Wu, K., Zhu, Z.: Distributed online hybrid cloud management for profit-driven multimedia cloud computing. IEEE Trans. Multimedia 17(8), 1297–1308 (2015)

    Google Scholar 

  31. Zhu, J., Li, X., Ruiz, R., Xu, X.: Scheduling stochastic multi-stage jobs to elastic hybrid cloud resources. IEEE Trans. Parallel Distrib. Syst. 29(6), 1401–1415 (2018)

    Google Scholar 

  32. Rodriguez, M.A., Buyya, R.: Deadline based resource provisioning and scheduling algorithm for scientific workflows on clouds. IEEE Trans. Cloud Comput. 2(2), 222–235 (2014)

    Google Scholar 

  33. Wang, Z.-J., Zhan, Z.-H., Yu, W.-J., Lin, Y., Zhang, J., Gu, T.-L., Zhang, J.: Dynamic group learning distributed particle swarm optimization for large-scale optimization and its application in cloud workflow scheduling. IEEE Trans. Cybern. 50(6), 2715–2729 (2019)

    Google Scholar 

  34. Wu, Q., Ishikawa, F., Zhu, Q., Xia, Y., Wen, J.: Deadline-constrained cost optimization approaches for workflow scheduling in clouds. IEEE Tran. Parallel Distrib. Syst. 28(12), 3401–3412 (2017)

    Google Scholar 

  35. Durillo, J.J., Prodan, R.: Multi-objective workflow scheduling in amazon EC2. Cluster Comput. 17(2), 169–189 (2014)

    Google Scholar 

  36. Wu, Q., Zhou, M., Zhu, Q., Xia, Y., Wen, J.: Moels: Multiobjective evolutionary list scheduling for cloud workflows. IEEE Trans. Autom. Sci. Eng. 17(1), 166–176 (2019)

    Google Scholar 

  37. Choudhary, A., Gupta, I., Singh, V., Jana, P.K.: A GSA based hybrid algorithm for bi-objective workflow scheduling in cloud computing. Future Gener. Comput. Syst. 83, 14–26 (2018)

    Google Scholar 

  38. Paknejad, P., Khorsand, R., Ramezanpour, M.: Chaotic improved PICEA-g-based multi-objective optimization for workflow scheduling in cloud environment. Future Gener. Comput. Syst. 117, 12–28 (2021)

    Google Scholar 

  39. Saeedi, S., Khorsand, R., Bidgoli, S.G., Ramezanpour, M.: Improved many-objective particle swarm optimization algorithm for scientific workflow scheduling in cloud computing. Comput. Ind. Eng. 147, 106649 (2020)

    Google Scholar 

  40. Chen, Z.-G., Zhan, Z.-H., Lin, Y., Gong, Y.-J., Gu, T.-L., Zhao, F., Yuan, H.-Q., Chen, X., Li, Q., Zhang, J.: Multiobjective cloud workflow scheduling: a multiple populations ant colony system approach. IEEE Trans. Cybern. 49(8), 2912–2926 (2018)

    Google Scholar 

  41. Larrañaga, P., Lozano, J.A.: Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation, vol. 2. Springer (2001)

    Google Scholar 

  42. Baluja, S.: Population-based incremental learning. a method for integrating genetic search based function optimization and competitive learning. Technical report, Carnegie-Mellon Universiy Pittsburgh, PA, Department of Computer Science (1994)

  43. Shao, W., Pi, D., Shao, Z.: A pareto-based estimation of distribution algorithm for solving multi-objective distributed no-wait flow-shop scheduling problem with sequence-dependent setup time. IEEE Trans. Autom. Sci. Eng. 16(3), 1344–1360 (2019)

    Google Scholar 

  44. Shao, W., Shao, Z., Pi, D.: Modeling and multi-neighborhood iterated greedy algorithm for distributed hybrid flow shop scheduling problem. Knowl.-Based Syst. 194, 105527 (2020)

    Google Scholar 

  45. Zhao, F., He, X., Wang, L.: A two-stage cooperative evolutionary algorithm with problem-specific knowledge for energy-efficient scheduling of no-wait flow-shop problem. IEEE Trans. Cybern. 51(11), 5291–5303 (2020)

    Google Scholar 

  46. Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multi-objective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)

    Google Scholar 

  47. Juve, G., Chervenak, A., Deelman, E., Bharathi, S., Mehta, G., Vahi, K.: Characterizing and profiling scientific workflows. Future Gener. Comput. Syst. 29(3), 682–692 (2013)

    Google Scholar 

  48. Zitzler, E., Thiele, L.: Multi-objective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE Trans. Evol. Comput. 3(4), 257–271 (1999)

    Google Scholar 

Download references

Funding

Not applicable.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Shengtai West Road, Nanjing, 211100, China

    Ming Li, Dechang Pi & Shuo Qin

Authors
  1. Ming Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dechang Pi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuo Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

ML and SQ wrote the main manuscript text and prepared all figures. All authors reviewed the manuscript.

Corresponding author

Correspondence to Shuo Qin.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, M., Pi, D. & Qin, S. Knowledge-based multi-objective estimation of distribution algorithm for solving reliability constrained cloud workflow scheduling. Cluster Comput 27, 1401–1419 (2024). https://doi.org/10.1007/s10586-023-04022-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-023-04022-w

Keywords

Search

Navigation