Skip to main content
SpringerLink
Log in

A divide and conquer approach to deadline constrained cost-optimization workflow scheduling for the cloud

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

The modeling of complex computational applications as giant computational workflows has been a critically effective means of better understanding the intricacies of applications and of determining the best approach to their realization. It is a challenging assignment to schedule such workflows in the cloud while also considering users’ different quality of service requirements. The present paper introduces a new direction based on a divide-and-conquer approach to scheduling these workflows. The proposed Divide-and-conquer Workflow Scheduling algorithm (DQWS) is designed with the objective of minimizing the cost of workflow execution while respecting its deadline. The critical path concept is the inspiration behind the divide-and-conquer process. DQWS finds the critical path, schedules it, removes the critical path from the workflow, and effectively divides the leftover into some mini workflows. The process continues until only chain structured workflows, called linear graphs, remain. Scheduling linear graphs is performed in the final phase of the algorithm. Experiments show that DQWS outperforms its competitors, both in terms of meeting deadlines and minimizing the monetary costs of executing scheduled workflows.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Notes

  1. https://confluence.pegasus.isi.edu/display/pegasus/ Workflow Generator.

References

  1. Buyya, R., Vecchiola, C., Selvi, S.T.: High-throughput computing. In: Proceedings of the Mastering cloud computing: foundations and applications programming. p. 222. Elsevier, (2013)

  2. Ostermann, S., Iosup, A., Yigitbasi, N., Prodan, R., Fahringer, T., Epema, D.: A performance analysis of EC2 cloud computing services for scientific computing. In: Proceedings of the International Conference on Cloud Computing 2009, pp. 115–131. Springer

  3. Li, X., Cai, Z.: Elastic resource provisioning for cloud workflow applications. IEEE Trans. Autom. Sci. Eng. 14(2), 1195–1210 (2017)

    Article  Google Scholar 

  4. Yu, J., Buyya, R.: A taxonomy of scientific workflow systems for grid computing. ACM Sigmod Record. 34(3), 44–49 (2005)

    Article  Google Scholar 

  5. Singh, L., Singh, S.: A survey of workflow scheduling algorithms and research issues. Int. J. Comput. Appl. 74(15), 21–28 (2013)

    Google Scholar 

  6. Nedić, N., Vukmirović, S., Imre, L., Čapko, D.: A genetic algorithm approach for utility management system workflow scheduling. Inf. Technol. Control 39(4), 310–316 (2010)

    Google Scholar 

  7. Rodriguez, M.A., Buyya, R.: A taxonomy and survey on scheduling algorithms for scientific workflows in IaaS cloud computing environments. Concurrency and Computation: Practice and Experience. 29(8), (2017)

  8. Zhu, X., Wang, J., Guo, H., Zhu, D., Yang, L.T., Liu, L.: Fault-tolerant scheduling for real-time scientific workflows with elastic resource provisioning in virtualized clouds. IEEE Trans. Parallel Distrib. Syst. 27(12), 3501–3517 (2016)

    Article  Google Scholar 

  9. Kc, K., Anyanwu, K.: Scheduling hadoop jobs to meet deadlines. In: 2010 IEEE Second International Conference on Cloud Computing Technology and Science 2010, pp. 388–392. IEEE

  10. Ferguson, A.D., Bodik, P., Kandula, S., Boutin, E., Fonseca, R.: Jockey: Guaranteed job latency in data parallel clusters. In: Proceedings of the 7th ACM European Conference on Computer Systems 2012, pp. 99–112

  11. Dimopoulos, S., Krintz, C., Wolski, R.: Pythia: Admission control for multi-framework, deadline-driven, big data workloads. In: Proceedings of the 2017 IEEE 10th International Conference on Cloud Computing (CLOUD) 2017, pp. 488–495. IEEE

  12. Kwok, Y.-K.K.Y.-K., Ahmad, I.: A static scheduling algorithm using dynamic critical path for assigning parallel algorithms onto multiprocessors. In: Parallel Processing, 1994. ICPP 1994 Volume 2. International Conference on 1994, pp. 155–159. IEEE

  13. Topcuoglu, H., Hariri, S., Wu, M.-Y.: Task scheduling algorithms for heterogeneous processors. In: Heterogeneous Computing Workshop, 1999 (HCW'99) Proceedings. Eighth 1999, pp. 3–14. IEEE

  14. Rahman, M., Hassan, R., Ranjan, R., Buyya, R.: Adaptive workflow scheduling for dynamic grid and cloud computing environment. Concurr. Comput. Pract. Exp. 25(13), 1816–1842 (2013)

    Article  Google Scholar 

  15. Rahman, M., Venugopal, S., Buyya, R.: A dynamic critical path algorithm for scheduling scientific workflow applications on global grids. In: Proceedings of the e-Science and Grid Computing, IEEE International Conference on 2007, pp. 35–42. IEEE

  16. Xue, S., Peng, Y., Xu, X., Zhang, J., Shen, C., Ruan, F.: DSM: a dynamic scheduling method for concurrent workflows in cloud environment. Clust. Comput. 22(1), 693–706 (2019)

    Article  Google Scholar 

  17. Singh, V., Gupta, I., Jana, P.K.: An energy efficient algorithm for workflow scheduling in IaaS Cloud. J. Grid Comput. 18(3), 357–376 (2019)

    Article  Google Scholar 

  18. Wang, S., Li, K., Mei, J., Xiao, G., Li, K.: A reliability-aware task scheduling algorithm based on replication on heterogeneous computing systems. J. Grid Comput. 15(1), 23–39 (2017)

    Article  Google Scholar 

  19. Chakravarthi, K.K., Shyamala, L., Vaidehi, V.: Budget aware scheduling algorithm for workflow applications in IaaS clouds. Clust. Comput. 23(4), 3405–3419 (2020)

    Article  Google Scholar 

  20. Rizvi, N., Ramesh, D.: Fair budget constrained workflow scheduling approach for heterogeneous clouds. Clust. Comput. 23(4), 3185–3201 (2020)

    Article  Google Scholar 

  21. Garg, N., Singh, D., Goraya, M.S.: Energy and resource efficient workflow scheduling in a virtualized cloud environment. Clust. Comput. (in press) (2020)

  22. Sreenu, K., Sreelatha, M.: W-Scheduler: whale optimization for task scheduling in cloud computing. Clust. Comput. 22(S1), 1087–1098 (2019)

    Article  Google Scholar 

  23. Bittencourt, L.F., Madeira, E.R.M.: HCOC: a cost optimization algorithm for workflow scheduling in hybrid clouds. J. Internet Serv. Appl. 2(3), 207–227 (2011)

    Article  Google Scholar 

  24. Abrishami, H., Rezaeian, A., Tousi, G.K., Naghibzadeh, M.: Scheduling in hybrid cloud to maintain data privacy. In: Proceedings of the Innovative Computing Technology (INTECH), 2015 Fifth International Conference on 2015, pp. 83–88. IEEE

  25. Abrishami, S., Naghibzadeh, M., Epema, D.H.: Deadline-constrained workflow scheduling algorithms for Infrastructure as a Service Clouds. Future Gen. Comput. Syst. 29(1), 158–169 (2013)

    Article  Google Scholar 

  26. Byun, E.-K., Kee, Y.-S., Kim, J.-S., Maeng, S.: Cost optimized provisioning of elastic resources for application workflows. Future Gen. Comput. Syst. 27(8), 1011–1026 (2011)

    Article  Google Scholar 

  27. Malawski, M., Figiela, K., Bubak, M., Deelman, E., Nabrzyski, J.: Scheduling multilevel deadline-constrained scientific workflows on clouds based on cost optimization. Sci. Program. 2015, 1–13 (2015)

    Google Scholar 

  28. Arabnejad, V., Bubendorfer, K., Ng, B.: Scheduling deadline constrained scientific workflows on dynamically provisioned cloud resources. Future Gen. Comput. Syst. 75, 348–364 (2017)

    Article  Google Scholar 

  29. Arabnejad, V., Bubendorfer, K., Ng, B., Chard, K.: A deadline constrained critical path heuristic for cost-effectively scheduling workflows. In: Proceedings of the 2015 IEEE/ACM 8th International Conference on Utility and Cloud Computing (UCC) 2015, pp. 242–250. IEEE

  30. Verma, A., Kaushal, S.: Cost-time efficient scheduling plan for executing workflows in the cloud. J. Grid Comput. 13(4), 495–506 (2015)

    Article  MathSciNet  Google Scholar 

  31. Zheng, W., Emmanuel, B., Wang, C., Qin, Y., Zhang, D.: Cost optimization for scheduling scientific workflows on clouds under deadline constraints. In: Proceedings of the Advanced Cloud and Big Data (CBD), 2017 Fifth International Conference on 2017, pp. 51–56. IEEE

  32. Cao, S., Deng, K., Ren, K., Li, X., Nie, T., Song, J.: A deadline-constrained scheduling algorithm for scientific workflows in clouds. In: Proceedings of the 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS) 2019, pp. 98–105. IEEE

  33. Ebrahimi, M., Mohan, A., Lu, S.: scheduling big data workflows in the cloud under deadline constraints. In: Proceedings of the 2018 IEEE Fourth International Conference on Big Data Computing Service and Applications (BigDataService) 2018, pp. 33–40. IEEE

  34. Cadorel, E., Coullon, H., Menaud, J.-M.: A workflow scheduling deadline-based heuristic for energy optimization in Cloud. In: Proceedings of the 2019 International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData) 2019, pp. 719–728. IEEE

  35. Chen, W., Xie, G., Li, R., Li, K.: Execution cost minimization scheduling algorithms for deadline-constrained parallel applications on heterogeneous clouds. Clust. Comput. (in press) (2020)

  36. Sun, T., Xiao, C., Xu, X.: A scheduling algorithm using sub-deadline for workflow applications under budget and deadline constrained. Clust. Comput. 22(3), 5987–5996 (2019)

    Article  Google Scholar 

  37. Wu, Q., Ishikawa, F., Zhu, Q., Xia, Y., Wen, J.: Deadline-constrained Cost Optimization Approaches for Workflow Scheduling in Clouds. IEEE Trans. Parallel Distrib. Syst. 28(12), 3401–3412 (2017)

    Article  Google Scholar 

  38. Chen, Z.-G., Du, K.-J., Zhan, Z.-H., Zhang, J.: Deadline constrained cloud computing resources scheduling for cost optimization based on dynamic objective genetic algorithm. In: Proceedings of the Evolutionary Computation (CEC), 2015 IEEE Congress on 2015, pp. 708–714. IEEE

  39. Wen, Y., Liu, J., Dou, W., Xu, X., Cao, B., Chen, J.: Scheduling workflows with privacy protection constraints for big data applications on cloud. Future Gen. Comput. Syst. 108, 1084–1091 (2018)

    Article  Google Scholar 

  40. Kaur, G., Kalra, M.: Deadline constrained scheduling of scientific workflows on cloud using hybrid genetic algorithm. In: Proceedings of the 2017 7th International Conference on Cloud Computing, Data Science & Engineering-Confluence 2017, pp. 276–280. IEEE

  41. Mojab, S.Z.M., Ebrahimi, M., Reynolds, R., Lu, S.: iCATS: Scheduling big data workflows in the cloud using cultural algorithms. In: Proceedings of the 2019 IEEE Fifth International Conference on Big Data Computing Service and Applications (BigDataService) 2019, pp. 99–106. IEEE

  42. Biswas, T., Kuila, P., Ray, A.K.: A novel workflow scheduling with multi-criteria using particle swarm optimization for heterogeneous computing systems. Clust. Comput. 23(4), 3255–3271 (2020)

    Article  Google Scholar 

  43. Garg, R., Mittal, M.: Reliability and energy efficient workflow scheduling in cloud environment. Clust. Comput. 22(4), 1283–1297 (2019)

    Article  Google Scholar 

  44. Kalra, M., Singh, S.: Multi‐criteria workflow scheduling on clouds under deadline and budget constraints. Concurr. Comput. Pract. Exp. 31(17), (2019)

  45. Bittencourt, L.F., Madeira, E.R.: A performance-oriented adaptive scheduler for dependent tasks on grids. Concurr. Comput. Pract. Exp. 20(9), 1029–1049 (2008)

    Article  Google Scholar 

  46. Yu, J., Buyya, R., Ramamohanarao, K.: Workflow scheduling algorithms for grid computing. Stud. Comput. Intell. 146, 173–214 (2008)

    MATH  Google Scholar 

  47. Zhu, Z., Zhang, G., Li, M., Liu, X.: Evolutionary multi-objective workflow scheduling in cloud. IEEE Trans. Parallel Distrib. Syst. 27(5), 1344–1357 (2016)

    Article  Google Scholar 

  48. https://aws.amazon.com/ec2/pricing/on-demand/. Accessed 18 Sep. 2020

  49. Wu, H., Hua, X., Li, Z., Ren, S.: Resource and instance hour minimization for deadline constrained DAG applications using computer clouds. IEEE Trans. Parallel Distrib. Syst. 27(3), 885–899 (2016)

    Article  Google Scholar 

  50. Naghibzadeh, M.: Modeling and scheduling hybrid workflows of tasks and task interaction graphs on the cloud. Future Gen. Comput. Syst. 65, 33–45 (2016)

    Article  Google Scholar 

  51. Moore, E.F.: The shortest path through a maze. In: Proc. of the Int. Symp. on the Theory of Switching. (1959), pp. 285–292. Harvard University Press

  52. Juve, G., Chervenak, A., Deelman, E., Bharathi, S., Mehta, G., Vahi, K.: Characterizing and profiling scientific workflows. Future Gen. Comput. Syst. 29(3), 682–692 (2013)

    Article  Google Scholar 

  53. Bharathi, S., Chervenak, A., Deelman, E., Mehta, G., Su, M.-H., Vahi, K.: Characterization of scientific workflows. In: Proceedings of the Workflows in Support of Large-Scale Science, 2008. WORKS 2008. Third Workshop on 2008, pp. 1–10. IEEE

  54. https://confluence.pegasus.isi.edu/display/pegasus/WorkflowGenerator. Accessed 7 May 2020

  55. Palankar, M.R., Iamnitchi, A., Ripeanu, M., Garfinkel, S.: Amazon S3 for science grids: a viable solution? In: Proceedings of the 2008 International Workshop on Data-aware Distributed Computing 2008, pp. 55–64. ACM

Download references

Acknowledgments

We would like to express our gratitude to Dr. S. Abrishami, Mr. H. Abrishami, Mrs. H. Taheri, and Mr. M. Hatami for their invaluable comments that greatly contributed to the quality of the paper.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Ferdowsi University of Mashhad, Mashhad, Iran

    Ghazaleh Khojasteh Toussi & Mahmoud Naghibzadeh

Authors
  1. Ghazaleh Khojasteh Toussi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mahmoud Naghibzadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mahmoud Naghibzadeh.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khojasteh Toussi, G., Naghibzadeh, M. A divide and conquer approach to deadline constrained cost-optimization workflow scheduling for the cloud. Cluster Comput 24, 1711–1733 (2021). https://doi.org/10.1007/s10586-020-03223-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-020-03223-x

Keywords

Search

Navigation