Skip to main content
SpringerLink
Log in

Elastic resource provisioning for scientific workflow scheduling in cloud under budget and deadline constraints

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

With the popularization and development of cloud computing, lots of scientific computing applications are conducted in cloud environments. However, current application scenario of scientific computing is also becoming increasingly dynamic and complicated, such as unpredictable submission times of jobs, different priorities of jobs, deadlines and budget constraints of executing jobs. Thus, how to perform scientific computing efficiently in cloud has become an urgent problem. To address this problem, we design an elastic resource provisioning and task scheduling mechanism to perform scientific workflow jobs in cloud. The goal of this mechanism is to complete as many high-priority workflow jobs as possible under budget and deadline constraints. This mechanism consists of four steps: job preprocessing, job admission control, elastic resource provisioning and task scheduling. We perform the evaluation with four kinds of real scientific workflow jobs under different budget constraints. We also consider the uncertainties of task runtime estimations, provisioning delays, and failures in evaluation. The results show that in most cases our mechanism achieves a better performance than other mechanisms. In addition, the uncertainties of task runtime estimations, VM provisioning delays, and task failures do not have major impact on the mechanism’s performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Ams02 experiment. [Online]. Available: http://www.ams02.org

  2. Malawski, M, Juve, G., Deelman, E., Nabrzyski, J.: Cost-and deadline-constrained provisioning for scientific workflow ensembles in iaas clouds. In: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, p. 2. IEEE Computer Society Press (2012)

  3. Mao, M., Humphrey, M.: Auto-scaling to minimize cost and meet application deadlines in cloud workflows,. in: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, p. 49. ACM (2011)

  4. Ming-Mao, M.H.: Scaling and scheduling to maximize application performance within budget constraints in cloud workflows. In: IEEE International Parallel and Distributed Processing Symposium. IEEE Computer Society Press (2013)

  5. Abrishami, S., Naghibzadeh, M., Epema, D.H.: Deadline-constrained workflow scheduling algorithms for infrastructure as a service clouds. Future Gener. Comput. Syst. 29(1), 158–169 (2013)

    Article  Google Scholar 

  6. Calheiros, R., Buyya, R.: Meeting deadlines of scientific workflows in public clouds with tasks replication. IEEE Trans. Parallel Distrib. Syst. 25(7), 1787–1796 (2014)

    Article  Google Scholar 

  7. Byun, E.-K., Kee, Y.-S., Kim, J.-S., Maeng, S.: Cost optimized provisioning of elastic resources for application workflows. Future Gener. Comput. Syst. 27(8), 1011–1026 (2011)

    Article  Google Scholar 

  8. Lin, X., Wu, C.Q.: On scientific workflow scheduling in clouds under budget constraint. In: 42nd International Conference on Parallel Processing (ICPP), pp. 90–99. IEEE (2013)

  9. Hoseiny Farahabady, M.R., Samani, H.R., Leslie, L.M., Lee, Y.C., Zomaya, A.Y.: Handling uncertainty: Pareto-efficient bot scheduling on hybrid clouds. In: 2013 42nd International Conference on Parallel Processing (ICPP), pp. 419–428. IEEE

  10. Delavar, A.G., Aryan, Y.: Hsga: a hybrid heuristic algorithm for workflow scheduling in cloud systems. Clus. Comput. 17(1), 129–137 (2014)

    Article  Google Scholar 

  11. Lee, Y.C., Han, H., Zomaya, A.Y., Yousif, M.: Resource-efficient workflow scheduling in clouds. Knowl. Based Syst. 80, 153–162 (2015)

    Article  Google Scholar 

  12. Yu, J., Buyya, R., Tham, C.K.: Cost-based scheduling of scientific workflow applications on utility grids. In: First International Conference on e-Science and Grid Computing. IEEE (2005)

  13. Sakellariou, R., Zhao, H., Tsiakkouri, E., Dikaiakos, M.D.: Scheduling workflows with budget constraints. In: Integrated Research in GRID Computing, pp. 189–202. Springer, Berlin (2007)

  14. Zhang, W., Cao, J., Zhong, Y., Liu, L., Wu, C.: Concurrent and storage-aware data streaming for data processing workflows in grid environments. Tsinghua Sci. Technol. 15(3), 335–346 (2010)

    Article  Google Scholar 

  15. Kamthe, A., Lee, S.-Y.: A stochastic approach to estimating earliest start times of nodes for scheduling dags on heterogeneous distributed computing systems. Clust. Comput. 14(4), 377–395 (2011)

    Article  Google Scholar 

  16. Chen, W., Lee, Y.C., Fekete, A., Zomaya, A.Y.: Adaptive multiple-workflow scheduling with task rearrangement. J. Supercomput. 71(4), 1297–1317 (2015)

    Article  Google Scholar 

  17. Lee, Y.C., Zomaya, A.Y.: Stretch out and compact: Workflow scheduling with resource abundance. In: 2013 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pp. 219–226. IEEE (2013)

  18. Fard, H.M., Prodan, R., Fahringer, T.: Multi-objective list scheduling of workflow applications in distributed computing infrastructures. J. Parallel Distrib. Comput. 74(3), 2152–2165 (2014)

    Article  MATH  Google Scholar 

  19. Durillo, J.J., Prodan, R.: Multi-objective workflow scheduling in amazon ec2. Clus. Comput. 17(2), 169–189 (2014)

    Article  Google Scholar 

  20. Nita, M.-C., Pop, F., Voicu, C., Dobre, C., Xhafa, F.: Momth: multi-objective scheduling algorithm of many tasks in hadoop. Clust. Comput. 1–14 (2015)

  21. Wang, L., Shen, J., Luo, J.: Facilitating an ant colony algorithm for multi-objective data-intensive service provision. J. Comput. Syst. Sci. 81(4), 734–746 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  22. Amazon elastic compute cloud (amazon ec2). [Online]. Available: http://aws.amazon.com/ec2/

  23. Abrishami, S., Naghibzadeh, M., Epema, D.H.: Cost-driven scheduling of grid workflows using partial critical paths. IEEE Trans. Parallel Distrib. Syst. 23(8), 1400–1414 (2012)

    Article  Google Scholar 

  24. Berman, F., Casanova, H., Chien, A., Cooper, K., Dail, H., Dasgupta, A., Deng, W., Dongarra, J., Johnsson, L., Kennedy, K., et al.: New grid scheduling and rescheduling methods in the grads project. Int. J. Parallel Program. 33(2–3), 209–229 (2005)

    Article  Google Scholar 

  25. Jang, S., Wu, X., Taylor, V., Mehta, G., Vahi, K., Deelman, E.: Using performance prediction to allocate grid resources. Texas A&M University, College Station, TX, GriPhyN Technical Report 25, (2004)

  26. Dynamic programming algorithm. [Online]. Available: http://en.wikipedia.org/wiki/Dynamic_programming

  27. Shi, J., Luo, J., Dong, F., Zhang, J.: A budget and deadline aware scientific workflow resource provisioning and scheduling mechanism for cloud. In: Proceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 672–677 (2014)

  28. Bin packing problem. [Online]. Available:http://en.wikipedia.org/wiki/Bin_packing_problem

  29. Earliest deadline first scheduling. [Online]. Available: http://en.wikipedia.org/wiki/Earliest_deadline_first_scheduling

  30. Calheiros, R.N., Ranjan, R., Beloglazov, A., De Rose, C.A., Buyya, R.: Cloudsim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Softw Practice Exp. 41(1), 23–50 (2011)

    Article  Google Scholar 

  31. Pegasus workflow generator. [Online]. Available: https://confluence.pegasus.isi.edu/display/pegasus/WorkflowGenerator

Download references

Acknowledgments

This work is supported by National Natural Science Foundation of China under Grants No. 61320106007, No. 61202449, No. 61572129, No. 61502097, No. 61370207, National High-tech R&D Program of China (863 Program) under Grants No. 2013AA013503, China Fundamental Research Funds for the Central Universities under Grans No. 1109007115, Jiangsu research prospective joint research project under Grants No. BY2012202, No. BY2013073-01, Jiangsu Provincial Key Laboratory of Network and Information Security under Grants No. BM2003201, Key Laboratory of Computer Network and Information Integration of Ministry of Education of China under Grants No. 93K-9, and partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Southeast University, Nanjing, People’s Republic of China

    Jiyuan Shi, Junzhou Luo, Fang Dong, Jinghui Zhang & Junxue Zhang

Authors
  1. Jiyuan Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junzhou Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fang Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jinghui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Junxue Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junzhou Luo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shi, J., Luo, J., Dong, F. et al. Elastic resource provisioning for scientific workflow scheduling in cloud under budget and deadline constraints. Cluster Comput 19, 167–182 (2016). https://doi.org/10.1007/s10586-015-0530-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-015-0530-0

Keywords

Search

Navigation