Abstract
With the wide deployment of cloud computing in scientific computing, cost minimization is increasingly critical for large-scale scientific workflow. Unfortunately, due to the highly intricate directed acyclic graph (DAG)-based workflow and the flexible usage of virtual machines (VMs) in cloud platform, the existing workflow scheduling approaches are inefficient to strike a balance between the parallelism and the topology of the DAG-based workflow while using the VMs, which causes a low utilization of VMs and consumes more cost. To address these issues, this paper presents a novel task scheduling framework named cost minimization approach with the DAG splitting method (COMSE) for minimizing the cost of running a deadline-constrained large-scale scientific workflow. First, we provide comprehensive theoretical analyses on how to improve the utilization of a resource-balanced multi-vCPU VM for running multiple tasks simultaneously. Second, considering the balance between the parallelism and the topology of a workflow, we simplify the DAG-based workflow, and based on the simplified DAG, a DAG splitting method is devised to preprocess the workflow. Third, since the cloud is charged by hours, we also design an exact algorithm to find the optimal operation pattern for a given schedule to make the consumed instance hours minimum, and this algorithm is named as instance hours minimization by Dijkstra (TOID). Finally, by employing the DAG splitting method and the TOID, the COMSE schedules a deadline-constrained large-scale scientific workflow on the multi-vCPU VMs and incorporates two important objects: minimizing the computation cost and the communication cost. Our solution approach is evaluated through rigorous performance evaluation study using real-word workflows, and the results show that the proposed COMSE approach outperforms existing algorithms in terms of computation cost and communication cost.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig8_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig9_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig10_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig11_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig12_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig13_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig14_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig15_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig16_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11227-020-03273-3/MediaObjects/11227_2020_3273_Fig17_HTML.png)
Similar content being viewed by others
References
M.R. MEDIA. Global cloud computing market forecast 2019-2024. https://www.marketresearchmedia.com/?p=839
Armbrust M, Fox A, Griffith R, Joseph AD, Katz R, Konwinski A, Lee G, Patterson D, Rabkin A, Stoica I et al (2010) A view of cloud computing. Commun ACM 53(4):50
Zhu X, Yang LT, Chen H, Wang J, Yin S, Liu X (2014) Real-time tasks oriented energy-aware scheduling in virtualized clouds. IEEE Trans Cloud Comput 2(2):168
Vasile MA, Pop F, Tutueanu RI, Cristea V (2015) Resource-aware hybrid scheduling algorithm in heterogeneous distributed computing. Future Gener Comput Syst 51(C):61
Li P, Chen Z, Yang LT, Gao J, Zhang Q, Deen MJ (2018) An incremental deep convolutional computation model for feature learning on industrial big data. IEEE Trans Industr Inf 15(3):1341–1349
Cloud A (2019) Cloud paralleled file system. https://www.alibabacloud.com/help/product/111536.htm
Singh S, Chana I (2016) A survey on resource scheduling in cloud computing: issues and challenges. J Grid Comput 14(2):217
Alkhanak EN, Lee SP, Rezaei R, Parizi RM (2016) Cost optimization approaches for scientific workflow scheduling in cloud and grid computing: a review, classifications, and open. J Syst Softw 113:1
Bojanova I, Zhang J, Voas J (2013) Cloud computing. IT Prof 15(2):12
Alkhanak EN, Lee SP (2018) A hyper-heuristic cost optimisation approach for scientific workflow scheduling in cloud computing. Future Gener Comput Syst 86:480
Boutin E, Ekanayake J, Lin W, Shi B, Zhou J, Qian Z, Wu M, Zhou L (2014) Apollo: scalable and coordinated scheduling for cloud-scale computing. In: Usenix Conference on Operating Systems Design and Implementation, pp 285–300
Dalman T, Wiechert W, Nöh K (2015) A deadline-constrained scheduling algorithm for scientific workflows in clouds. J Biotechnol 232:12
Zhang J, Kuc D, Lu S (2014) Confucius: a tool supporting collaborative scientific workflow composition. IEEE Trans Serv Comput 7(1):2
Wu H, Hua X, Li Z, Ren S (2016) Resource and instance hour minimization for deadline constrained DAG applications using computer clouds. IEEE Trans Parallel Distrib Syst 27(3):885
Topcuoglu H, Hariri S, Wu MY (2002) Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans Parallel Distrib Syst 13(3):260
Abrishami Saeid, Naghibzadeh Mahmoud, Dick Epema, HJ (2013) Deadline-constrained workflow scheduling algorithms for infrastructure; as a service clouds. Future Gener Comput Syst 29(1):158
Cao S, Deng K, Ren K, Li X, Nie T, Song J (2019) 2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). IEEE, pp 98–105
Chen H, Zhu X, Qiu D, Ling L, Du Z (2017) Scheduling for workflows with security-sensitive intermediate data by selective tasks duplication in clouds. IEEE Trans Parallel Distrib Syst 28(9):2674
Zong Z, Manzanares A, Ruan X, Qin X (2011) EAD and PEBD: two energy-aware duplication scheduling algorithms for parallel tasks on homogeneous clusters. IEEE Trans Comput 60(3):360
Choudhury P, Kumar R, Chakrabarti PP (2008) Hybrid scheduling of dynamic task graphs with selective duplication for multiprocessors under memory and time constraints. IEEE Press, New York
Gerasoulis A, Yang T (1992) A comparison of clustering heuristics for scheduling directed acyclic graphs on multiprocessors. J Parallel Distrib Comput 16(4):276
Johnson DB (1973) A note on dijkstra`s shortest path algorithm. JACM 20(3):385–388
Smith B, Lucas K (2011) UnixBench: The original BYTE UNIX benchmark suite [Online]. Available: https://github.com/kdlucas/byte-unixbench
Hartmanis J (1982) Computers and intractability:a guide to the theory of np-completeness (Michael R. Garey and David S. Johnson). Siam Rev 24(1):90–92. https://doi.org/10.1137/1024022
Bittencourt LF, Madeira ERM (2011) HCOC: a cost optimization algorithm for workflow scheduling in hybrid clouds. J Internet Serv Appl 2(3):207
Wu H, Chen X, Song X, Guo H (2019) Cost minimization of scheduling scientific workflow applications on clouds. Concurr Comput Pract Exp e5503
Chen W, da Silva RF, Deelman E, Sakellariou R (2015) Using imbalance metrics to optimize task clustering in scientific workflow executions. Future Gener Comput Syst 46:69
Zhu Z, Zhang G, Li M, Liu X (2016) Evolutionary multi-objective workflow scheduling in cloud. IEEE Trans Parallel Distrib Syst 27(5):1344
Verma A, Kaushal S (2017) A hybrid multi-objective particle swarm optimization for scientific workflow scheduling. Parallel Comput 62:1
Talukder AKMKA, Kirley M, Buyya R (2009) Multiobjective differential evolution for scheduling workflow applications on global grids. Concurr Comput Pract Exp 21(13):1742
Braun TD, Siegel HJ, Beck N, Reuther AI, Theys MD, Yao B, Freund RF, Maheswaran M, Robertson JP, Hensgen D (1999) Eighth heterogeneous computing workshop, p 15
Pabla CS (2009) Completely fair scheduler, vol 4. Betascript Publishing, London
Kozich JJ, Westcott SL, Baxter NT, Highlander SK, Schloss PD (2013) Development of a dual-index sequencing strategy and curation pipeline for analyzing amplicon sequence data on the MiSeq Illumina sequencing platform. Appl Environ Microbiol 79(17):5112
Brown DA, Brady PR, Dietz A, Cao J, Johnson B, Mcnabb J (2007) A case study on the use of workflow technologies for scientific analysis: gravitational wave data analysis. Springer, London
Bharathi S, Chervenak A, Deelman E, Mehta G, Su MH, Vahi K (2008) Third Workshop on Workflows in Support of Large-Scale Science. WORKS 2008. IEEE, 2008, pp 1–10
Juve G, Chervenak A, Deelman E, Bharathi S, Mehta G, Vahi K (2013) Characterizing and profiling scientific workflows. Future Gener Comput Syst 29(3):682
Juve G (2014) workflowgenerator. https://confluence.pegasus.isi.edu/display/pegasus/workflowgenerator
Deelman E, Callaghan S, Field E, Francoeur H, Graves R, Gupta N, Gupta V, Jordan TH, Kesselman C, Maechling P (2006) IEEE International Conference on E-Science and Grid Computing, 2006. E-Science, pp 14–14
Callaghan S, Maechling P, Dan G, Beattie K, Jordan T, Deelman E, Vahi K, Mehta G, Juve G, Milner K (2009) IEEE Fourth International Conference on Escience, 2008. Escience, pp 151–158
Berriman GB, Deelman E, Good JC, Jacob JC, Katz DS, Kesselman C, Laity AC, Prince TA, Singh G, Su MH (2004) Optimizing Scientific Return for Astronomy through Information Technologies, vol 5493. International Society for Optics and Photonics, pp 221–233
Livny J, Teonadi H, Livny M, Waldor MK, Livny J, Teonadi H, Livny M, Waldor MK (2008) High-throughput, kingdom-wide prediction and annotation of bacterial non-coding RNAs. PLoS ONE 3(9):e3197
Acknowledgements
This document is the results of the research project funded by Natural Science Foundation of Liaoning (No. 2019-MS-170), Guangxi Key Laboratory of Hybrid Computation and IC Design Analysis Open Fund (HCIC201605), Guangxi Youth Teacher Project (2018KY0976) and Supercomputing Center of Dalian University of Technology.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Wu, H., Chen, X., Song, X. et al. Scheduling large-scale scientific workflow on virtual machines with different numbers of vCPUs. J Supercomput 77, 679–710 (2021). https://doi.org/10.1007/s11227-020-03273-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-020-03273-3