Skip to main content

Advertisement

Springer Nature Link
Log in

Scheduling multiple scientific workflows using containers on IaaS cloud

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

With the adaptable paradigm of cloud computing and obtainable of data accumulated from largely high-powered scientific devices, workflows have turn into an occurring aim to execute considerable scientific advances at an enhanced speed. Occurring Workflow as a Service (WaaS) frameworks provide scientists an effortless, simply accessible and cost-efficient manner of using their applications from anywhere and at anytime in the cloud. They are multitenant platforms and are developed to handle the execution of heterogeneous workflows continuous workload. To fulfill this, they utilize the compute, network and storage services provided by Infrastructure as a Service (IaaS) vendors. Therefore, at any considerable particular moment, a WaaS framework should be proficient of effectively schedule these continuous workload of workflows with various features and quality of service (QoS). Therefore, we propose a strategy of scheduling and resource provisioning planned particularly for WaaS platforms. The algorithm is dynamic and scalable to adjust to improve in the workload and platform. It supports containers to deal the inefficiency of resource utilization and targets to reduce the overall execution cost of infrastructure resources as fulfilling each single workflow deadline constraint. To our information, this approach that explicitly deals VM sharing in the subject of WaaS by devising the utilization of containers in the heuristics of scheduling and resource provisioning. Our experimental results shows its responsiveness to the uncertainties of the environment, its potential to achieve deadlines, and its cost-effectiveness when compared to other recent algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

References

  • Abrishami S, Naghibzadeh M, Epema DH (2013) Deadline-constrained workflow scheduling algorithms for infrastructure as a service clouds. Future Gener Comput Syst 29(1):158–169

    Article  Google Scholar 

  • Bryk P, Malawski M, Juve G, Deelman E (2016) Storage-aware algorithms for scheduling of workflow ensembles in clouds. J Grid Comput 14(2):359–378

    Article  Google Scholar 

  • Calheiros RN, Ranjan R, Beloglazov A, De Rose CA, Buyya R (2011) CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Softw Pract Exp 41(1):23–50

    Article  Google Scholar 

  • Calheiros RN, Buyya R (2013) Meeting deadlines of scientific workflows in public clouds with tasks replication. IEEE Trans Parallel Distrib Syst 25(7):1787–1796

    Article  Google Scholar 

  • Chai X (2020) Task scheduling based on swarm intelligence algorithms in high performance computing environment. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-020-01994-0

    Article  Google Scholar 

  • Chen W, Lee YC, Fekete A, Zomaya AY (2015) Adaptive multiple-workflow scheduling with task rearrangement. J Supercomput 71(4):1297–1317

    Article  Google Scholar 

  • Chhabra A, Singh G, Kahlon KS (2020) Performance-aware energy-efficient parallel job scheduling in HPC grid using nature-inspired hybrid meta-heuristics. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-020-02255-w

    Article  Google Scholar 

  • Deelman E, Singh G, Livny M, Berriman B, Good J, (2008) The cost of doing science on the cloud: the montage example. In: SC'08: proceedings of the 2008 ACM/IEEE conference on supercomputing, IEEE, pp 1–12

  • Deldari A, Naghibzadeh M, Abrishami S (2017) CCA: a deadline-constrained workflow scheduling algorithm for multicore resources on the cloud. J Supercomput 73(2):756–781

    Article  Google Scholar 

  • Di Tommaso P, Palumbo E, Chatzou M, Prieto P, Heuer ML, Notredame C (2015) The impact of Docker containers on the performance of genomic pipelines. PeerJ 3:e1273

    Article  Google Scholar 

  • Dziok T, Figiela K, Malawski M (2016) Adaptive multi-level workflow scheduling with uncertain task estimates. Parallel processing and applied mathematics. Springer, Cham, pp 90–100

    Chapter  Google Scholar 

  • Esteves S, Veiga L (2016) WaaS: workflow-as-a-service for the cloud with scheduling of continuous and data-intensive workflows. Comput J 59(3):371–383

    Article  Google Scholar 

  • Felter W, Ferreira A, Rajamony R, Rubio J (2015) An updated performance comparison of virtual machines and linux containers. In: 2015 IEEE international symposium on performance analysis of systems and software (ISPASS), IEEE, pp 171–172

  • Filgueira R, Da Silva RF, Krause A, Deelman E, Atkinson M (2016) Asterism: Pegasus and dispel4py hybrid workflows for data-intensive science. In: 2016 Seventh international workshop on data-intensive computing in the clouds (DataCloud), IEEE, pp 1–8

  • Gerlach W, Tang W, Keegan K, Harrison T, Wilke A, Bischof J, DSouza M, Devoid S, Murphy-Olson D, Desai N, Meyer F, (2014) Skyport-container-based execution environment management for multi-cloud scientific workflows. In: 2014 5th International workshop on data-intensive computing in the clouds, IEEE, pp 25–32

  • Gil Y, Deelman E, Ellisman M, Fahringer T, Fox G, Gannon D, Goble C, Livny M, Moreau L, Myers J (2007) Examining the challenges of scientific workflows. Computer 40(12):24–32

    Article  Google Scholar 

  • Gupta A, Bhadauria HS, Singh A (2020) Load balancing based hyper heuristic algorithm for cloud task scheduling. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-020-02127-3

    Article  Google Scholar 

  • Iosup A, Ostermann S, Yigitbasi MN, Prodan R, Fahringer T, Epema D (2011) Performance analysis of cloud computing services for many-tasks scientific computing. IEEE Trans Parallel Distrib Syst 22(6):931–945

    Article  Google Scholar 

  • Jackson KR, Ramakrishnan L, Muriki K, Canon S, Cholia S, Shalf J, Wasserman HJ, Wright NJ (2010) Performance analysis of high performance computing applications on the amazon web services cloud. In: 2010 IEEE second international conference on cloud computing technology and science, IEEE, pp 159–168

  • Jiang HJ, Huang KC, Chang HY, Gu DS, Shih PJ (2011) Scheduling concurrent workflows in HPC cloud through exploiting schedule gaps. In: International conference on algorithms and architectures for parallel processing, Springer, Berlin, pp 282–293

  • Jiang Q, Lee YC, Zomaya AY (2015) Executing large scale scientific workflow ensembles in public clouds. In: 2015 44th International conference on parallel processing, IEEE, pp 520–529

  • Jin H, Wang X, Wu S, Di S, Shi X (2014) Towards optimized fine-grained pricing of IaaS cloud platform. IEEE Trans Cloud Comput 3(4):436–448

    Article  Google Scholar 

  • Juve G, Deelman E, Vahi K, Mehta G, Berriman B, Berman BP, Maechling P (2010) Data sharing options for scientific workflows on amazon ec2. In: SC'10: Proceedings of the 2010 ACM/IEEE international conference for high performance computing, networking, storage and analysis, IEEE, pp 1–9

  • Juve G, Chervenak A, Deelman E, Bharathi S, Mehta G, Vahi K (2013) Characterizing and profiling scientific workflows. Future Gener Comput Syst 29(3):682–692

    Article  Google Scholar 

  • Kouki Y, Ledoux T (2013) RightCapacity: SLA-driven cross-layer cloud elasticity management. Int J Next Gener Comput 4(3):250–262

    Google Scholar 

  • Liu S, Ren K, Deng K, Song J (2016) A task backfill based scientific workflow scheduling strategy on cloud platform. In: 2016 Sixth international conference on information science and technology (ICIST), IEEE, pp 105–110

  • Maddikunta PKR, Gadekallu TR, Kaluri R, Srivastava G, Parizi RM, Khan MS (2020) Green communication in IoT networks using a hybrid optimization algorithm. Comput Commun 159:97–107

    Article  Google Scholar 

  • Maechling P, Deelman E, Zhao L, Graves R, Mehta G, Gupta N, Mehringer J, Kesselman C, Callaghan S, Okaya D, Francoeur H (2007) SCEC CyberShake workflows—automating probabilistic seismic hazard analysis calculations. Workflows for e-Science. Springer, London, pp 143–163

    Chapter  Google Scholar 

  • Malawski M, Juve G, Deelman E, Nabrzyski J (2015) Algorithms for cost-and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds. Future Gener Comput Syst 48:1–18

    Article  Google Scholar 

  • Mao M, Humphrey M (2011) Auto-scaling to minimize cost and meet application deadlines in cloud workflows. In: SC'11: proceedings of 2011 international conference for high performance computing, networking, storage and analysis, IEEE, pp 1–12

  • Mao M, Humphrey M (2012) A performance study on the vm startup time in the cloud. In: 2012 IEEE fifth international conference on cloud computing, IEEE, pp 423–430

  • Ostermann S, Iosup A, Yigitbasi N, Prodan R, Fahringer T, Epema D (2009) A performance analysis of EC2 cloud computing services for scientific computing. In: International conference on cloud computing, Springer, Berlin, pp 115–131

  • Pietri I, Malawski M, Juve G, Deelman E, Nabrzyski J, Sakellariou R (2013) Energy-constrained provisioning for scientific workflow ensembles. In: 2013 International conference on cloud and green computing, IEEE, pp 34–41

  • Piraghaj SF, Dastjerdi AV, Calheiros RN, Buyya R (2017) ContainerCloudSim: an environment for modeling and simulation of containers in cloud data centers. Softw Pract Exp 47(4):505–521

    Article  Google Scholar 

  • Priya RMS, Bhattacharya S, Maddikunta PKR, Somayaji SRK, Lakshmanna K, Kaluri R, Hussien A, Gadekallu TR (2020) Load balancing of energy cloud using wind driven and firefly algorithms in internet of everything. J Parallel Distrib Comput 142:16–26

    Article  Google Scholar 

  • Rajan CDS (2020) Design and implementation of fuzzy priority deadline job scheduling algorithm in heterogeneous grid computing. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-020-02171-z

    Article  Google Scholar 

  • Reddy GT, Sudheer K, Rajesh K, Lakshmanna K (2014) Employing data mining on highly secured private clouds for implementing a security-asa-service framework. J Theor Appl Inf Technol 59(2):317–326

    Google Scholar 

  • Rodriguez MA, Buyya R (2017a) Budget-driven scheduling of scientific workflows in IaaS clouds with fine-grained billing periods. ACM Trans Auton Adapt Syst 12(2):1–22

    Article  Google Scholar 

  • Rodriguez MA, Buyya R (2017b) A taxonomy and survey on scheduling algorithms for scientific workflows in IaaS cloud computing environments. Concur Comput Pract Exp 29(8):e4041

    Article  Google Scholar 

  • Schad J, Dittrich J, Quiané-Ruiz JA (2010) Runtime measurements in the cloud: observing, analyzing, and reducing variance. Proc VLDB Endow 3(1–2):460–471

    Article  Google Scholar 

  • Shi J, Luo J, Dong F, Zhang J (2014) A budget and deadline aware scientific workflow resource provisioning and scheduling mechanism for cloud. In: Proceedings of the 2014 IEEE 18th international conference on computer supported cooperative work in design (CSCWD), IEEE, pp 672–677

  • Stavrinides GL, Karatza HD (2015) A cost-effective and qos-aware approach to scheduling real-time workflow applications in paas and saas clouds. In: 2015 3rd international conference on future internet of things and cloud, IEEE, pp 231–239

  • Thennarasu SR, Selvam M, Srihari K (2020) A new whale optimizer for workflow scheduling in cloud computing environment. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-020-01678-9

    Article  Google Scholar 

  • Vöckler JS, Juve G, Deelman E, Rynge M, Berriman B (2011) Experiences using cloud computing for a scientific workflow application. In: Proceedings of the 2nd international workshop on scientific cloud computing, pp 15–24

  • Wang W, Niu D, Li B, Liang B (2013) Dynamic cloud resource reservation via cloud brokerage. In: 2013 IEEE 33rd international conference on distributed computing systems, IEEE, pp 400–409

  • Wang J, Korambath P, Altintas I, Davis J, Crawl D (2014) Workflow as a service in the cloud: architecture and scheduling algorithms. Procedia Comput Sci 29:546

    Article  Google Scholar 

  • Xu M, Cui L, Wang H, Bi Y (2009) A multiple QoS constrained scheduling strategy of multiple workflows for cloud computing. In: 2009 IEEE international symposium on parallel and distributed processing with applications, IEEE, pp 629–634

  • Yin W, Mavaluru D, Ahmed M, Abbas M, Darvishan A (2020) Application of new multi-objective optimization algorithm for EV scheduling in smart grid through the uncertainties. J Ambient Intell Human Comput 11(5):2071–2103

    Article  Google Scholar 

  • Yu Z, Shi W (2008) A planner-guided scheduling strategy for multiple workflow applications. In: 2008 International conference on parallel processing-workshops, IEEE, pp 1–8

  • Zhou AC, He B, Liu C (2015) Monetary cost optimizations for hosting workflow-as-a-service in IaaS clouds. IEEE Trans Cloud Comput 4(1):34–48

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Science and Technology, College of Engineering, Anna University, Guindy, Chennai, Tamilnadu, India

    P. Rajasekar & Yogesh Palanichamy

Authors
  1. P. Rajasekar
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yogesh Palanichamy
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to P. Rajasekar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rajasekar, P., Palanichamy, Y. Scheduling multiple scientific workflows using containers on IaaS cloud. J Ambient Intell Human Comput 12, 7621–7636 (2021). https://doi.org/10.1007/s12652-020-02483-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-020-02483-0

Keywords