Skip to main content
Log in

Scheduling large-scale scientific workflow on virtual machines with different numbers of vCPUs

The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

With the wide deployment of cloud computing in scientific computing, cost minimization is increasingly critical for large-scale scientific workflow. Unfortunately, due to the highly intricate directed acyclic graph (DAG)-based workflow and the flexible usage of virtual machines (VMs) in cloud platform, the existing workflow scheduling approaches are inefficient to strike a balance between the parallelism and the topology of the DAG-based workflow while using the VMs, which causes a low utilization of VMs and consumes more cost. To address these issues, this paper presents a novel task scheduling framework named cost minimization approach with the DAG splitting method (COMSE) for minimizing the cost of running a deadline-constrained large-scale scientific workflow. First, we provide comprehensive theoretical analyses on how to improve the utilization of a resource-balanced multi-vCPU VM for running multiple tasks simultaneously. Second, considering the balance between the parallelism and the topology of a workflow, we simplify the DAG-based workflow, and based on the simplified DAG, a DAG splitting method is devised to preprocess the workflow. Third, since the cloud is charged by hours, we also design an exact algorithm to find the optimal operation pattern for a given schedule to make the consumed instance hours minimum, and this algorithm is named as instance hours minimization by Dijkstra (TOID). Finally, by employing the DAG splitting method and the TOID, the COMSE schedules a deadline-constrained large-scale scientific workflow on the multi-vCPU VMs and incorporates two important objects: minimizing the computation cost and the communication cost. Our solution approach is evaluated through rigorous performance evaluation study using real-word workflows, and the results show that the proposed COMSE approach outperforms existing algorithms in terms of computation cost and communication cost.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. M.R. MEDIA. Global cloud computing market forecast 2019-2024. https://www.marketresearchmedia.com/?p=839

  2. Armbrust M, Fox A, Griffith R, Joseph AD, Katz R, Konwinski A, Lee G, Patterson D, Rabkin A, Stoica I et al (2010) A view of cloud computing. Commun ACM 53(4):50

    Article  Google Scholar 

  3. Zhu X, Yang LT, Chen H, Wang J, Yin S, Liu X (2014) Real-time tasks oriented energy-aware scheduling in virtualized clouds. IEEE Trans Cloud Comput 2(2):168

    Article  Google Scholar 

  4. Vasile MA, Pop F, Tutueanu RI, Cristea V (2015) Resource-aware hybrid scheduling algorithm in heterogeneous distributed computing. Future Gener Comput Syst 51(C):61

    Article  Google Scholar 

  5. Li P, Chen Z, Yang LT, Gao J, Zhang Q, Deen MJ (2018) An incremental deep convolutional computation model for feature learning on industrial big data. IEEE Trans Industr Inf 15(3):1341–1349

    Article  Google Scholar 

  6. Cloud A (2019) Cloud paralleled file system. https://www.alibabacloud.com/help/product/111536.htm

  7. Singh S, Chana I (2016) A survey on resource scheduling in cloud computing: issues and challenges. J Grid Comput 14(2):217

    Article  Google Scholar 

  8. Alkhanak EN, Lee SP, Rezaei R, Parizi RM (2016) Cost optimization approaches for scientific workflow scheduling in cloud and grid computing: a review, classifications, and open. J Syst Softw 113:1

    Article  Google Scholar 

  9. Bojanova I, Zhang J, Voas J (2013) Cloud computing. IT Prof 15(2):12

    Article  Google Scholar 

  10. Alkhanak EN, Lee SP (2018) A hyper-heuristic cost optimisation approach for scientific workflow scheduling in cloud computing. Future Gener Comput Syst 86:480

    Article  Google Scholar 

  11. Boutin E, Ekanayake J, Lin W, Shi B, Zhou J, Qian Z, Wu M, Zhou L (2014) Apollo: scalable and coordinated scheduling for cloud-scale computing. In: Usenix Conference on Operating Systems Design and Implementation, pp 285–300

  12. Dalman T, Wiechert W, Nöh K (2015) A deadline-constrained scheduling algorithm for scientific workflows in clouds. J Biotechnol 232:12

    Article  Google Scholar 

  13. Zhang J, Kuc D, Lu S (2014) Confucius: a tool supporting collaborative scientific workflow composition. IEEE Trans Serv Comput 7(1):2

    Article  Google Scholar 

  14. Wu H, Hua X, Li Z, Ren S (2016) Resource and instance hour minimization for deadline constrained DAG applications using computer clouds. IEEE Trans Parallel Distrib Syst 27(3):885

    Article  Google Scholar 

  15. Topcuoglu H, Hariri S, Wu MY (2002) Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans Parallel Distrib Syst 13(3):260

    Article  Google Scholar 

  16. Abrishami Saeid, Naghibzadeh Mahmoud, Dick Epema, HJ (2013) Deadline-constrained workflow scheduling algorithms for infrastructure; as a service clouds. Future Gener Comput Syst 29(1):158

    Article  Google Scholar 

  17. Cao S, Deng K, Ren K, Li X, Nie T, Song J (2019) 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). IEEE, pp 98–105

  18. Chen H, Zhu X, Qiu D, Ling L, Du Z (2017) Scheduling for workflows with security-sensitive intermediate data by selective tasks duplication in clouds. IEEE Trans Parallel Distrib Syst 28(9):2674

    Article  Google Scholar 

  19. Zong Z, Manzanares A, Ruan X, Qin X (2011) EAD and PEBD: two energy-aware duplication scheduling algorithms for parallel tasks on homogeneous clusters. IEEE Trans Comput 60(3):360

    Article  MathSciNet  MATH  Google Scholar 

  20. Choudhury P, Kumar R, Chakrabarti PP (2008) Hybrid scheduling of dynamic task graphs with selective duplication for multiprocessors under memory and time constraints. IEEE Press, New York

    Book  Google Scholar 

  21. Gerasoulis A, Yang T (1992) A comparison of clustering heuristics for scheduling directed acyclic graphs on multiprocessors. J Parallel Distrib Comput 16(4):276

    Article  MathSciNet  MATH  Google Scholar 

  22. Johnson DB (1973) A note on dijkstra`s shortest path algorithm. JACM 20(3):385–388

    Article  MathSciNet  MATH  Google Scholar 

  23. Smith B, Lucas K (2011) UnixBench: The original BYTE UNIX benchmark suite [Online]. Available: https://github.com/kdlucas/byte-unixbench

  24. Hartmanis J (1982) Computers and intractability:a guide to the theory of np-completeness (Michael R. Garey and David S. Johnson). Siam Rev 24(1):90–92. https://doi.org/10.1137/1024022

  25. Bittencourt LF, Madeira ERM (2011) HCOC: a cost optimization algorithm for workflow scheduling in hybrid clouds. J Internet Serv Appl 2(3):207

    Article  Google Scholar 

  26. Wu H, Chen X, Song X,  Guo H (2019) Cost minimization of scheduling scientific workflow applications on clouds. Concurr Comput Pract Exp e5503  

  27. Chen W, da Silva RF, Deelman E, Sakellariou R (2015) Using imbalance metrics to optimize task clustering in scientific workflow executions. Future Gener Comput Syst 46:69

    Article  Google Scholar 

  28. Zhu Z, Zhang G, Li M, Liu X (2016) Evolutionary multi-objective workflow scheduling in cloud. IEEE Trans Parallel Distrib Syst 27(5):1344

    Article  Google Scholar 

  29. Verma A, Kaushal S (2017) A hybrid multi-objective particle swarm optimization for scientific workflow scheduling. Parallel Comput 62:1

    Article  MathSciNet  Google Scholar 

  30. Talukder AKMKA, Kirley M, Buyya R (2009) Multiobjective differential evolution for scheduling workflow applications on global grids. Concurr Comput Pract Exp 21(13):1742

    Article  Google Scholar 

  31. Braun TD, Siegel HJ, Beck N, Reuther AI, Theys MD, Yao B, Freund RF, Maheswaran M, Robertson JP, Hensgen D (1999) Eighth heterogeneous computing workshop, p 15

  32. Pabla CS (2009) Completely fair scheduler, vol 4. Betascript Publishing, London

    Google Scholar 

  33. Kozich JJ, Westcott SL, Baxter NT, Highlander SK, Schloss PD (2013) Development of a dual-index sequencing strategy and curation pipeline for analyzing amplicon sequence data on the MiSeq Illumina sequencing platform. Appl Environ Microbiol 79(17):5112

    Article  Google Scholar 

  34. Brown DA, Brady PR, Dietz A, Cao J, Johnson B, Mcnabb J (2007) A case study on the use of workflow technologies for scientific analysis: gravitational wave data analysis. Springer, London

    Google Scholar 

  35. Bharathi S, Chervenak A, Deelman E, Mehta G, Su MH, Vahi K (2008) Third Workshop on Workflows in Support of Large-Scale Science. WORKS 2008. IEEE, 2008, pp 1–10

  36. Juve G, Chervenak A, Deelman E, Bharathi S, Mehta G, Vahi K (2013) Characterizing and profiling scientific workflows. Future Gener Comput Syst 29(3):682

    Article  Google Scholar 

  37. Juve G (2014) workflowgenerator. https://confluence.pegasus.isi.edu/display/pegasus/workflowgenerator

  38. Deelman E, Callaghan S, Field E, Francoeur H, Graves R, Gupta N, Gupta V, Jordan TH, Kesselman C, Maechling P (2006) IEEE International Conference on E-Science and Grid Computing, 2006. E-Science, pp 14–14

  39. Callaghan S, Maechling P, Dan G, Beattie K, Jordan T, Deelman E, Vahi K, Mehta G, Juve G, Milner K (2009) IEEE Fourth International Conference on Escience, 2008. Escience, pp 151–158

  40. Berriman GB, Deelman E, Good JC, Jacob JC, Katz DS, Kesselman C, Laity AC, Prince TA, Singh G, Su MH (2004) Optimizing Scientific Return for Astronomy through Information Technologies, vol 5493. International Society for Optics and Photonics, pp 221–233

  41. Livny J, Teonadi H, Livny M, Waldor MK, Livny J, Teonadi H, Livny M, Waldor MK (2008) High-throughput, kingdom-wide prediction and annotation of bacterial non-coding RNAs. PLoS ONE 3(9):e3197

    Article  Google Scholar 

Download references

Acknowledgements

This document is the results of the research project funded by Natural Science Foundation of Liaoning (No. 2019-MS-170), Guangxi Key Laboratory of Hybrid Computation and IC Design Analysis Open Fund (HCIC201605), Guangxi Youth Teacher Project (2018KY0976) and Supercomputing Center of Dalian University of Technology.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xin Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, H., Chen, X., Song, X. et al. Scheduling large-scale scientific workflow on virtual machines with different numbers of vCPUs. J Supercomput 77, 679–710 (2021). https://doi.org/10.1007/s11227-020-03273-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-020-03273-3

Keywords

Navigation