Skip to main content
SpringerLink
Log in

Expansion slot backfill scheduling for concurrent workflows with deadline on heterogeneous resources

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

The cost of using a computational resource is measured from startup to shutdown, including the cost of time slots between tasks. Concurrent scheduling of multiple scientific workflows on heterogeneous resources can improve resources utilization and reduce the cost. To make full use of waste time slots between the tasks and improve the completion rate of workflows, Expansion Slot Backfill (ESB) algorithm is proposed in this paper for scheduling multiple deadline-constrained workflows on a fixed set of resources. All workflows are mapped to the resources with one by one strategy. Each new task tries to backfill the earliest time slot in turn. When the time slot is not enough to backfill, it can be expanded elastically by the slide of the earlier tasks. If these tasks slide cause some workflow to exceed the deadline constraint, such workflow with fewer time slots is discarded and the slide is withdrawn. Experiments with multiple parameter variations show that the algorithm get better performance in resource utilization, workflows throughput and time complexity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Juve, G., Chervenak, A., Deelman, E., Bharathi, S., Mehta, G., Vahi, K.: Characterizing and profiling scientific workflows. Futur. Gener. Comput. Syst. 29(3), 682–692 (2013)

    Article  Google Scholar 

  2. Deelman, E., Singh, G., Su, M.-H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Berriman, G.B., Good, J., Laity, A., Jacob, J.C., Katz, D.S.: Pegasus: a framework for mapping complex scientific workflows onto distributed systems. Sci. Program. 13(3), 219–237 (2005)

    Google Scholar 

  3. Deelman, E., Kesselman, C., Mehta, G., Meshkat, L., Pearlman, L., Blackburn, K., Ehrens, P., Lazzarini, A., Williams, R., Koranda, S.: GriPhyN and LIGO, building a virtual data Grid for gravitational wave scientists. In: HPDC ’02: Proceedings of the 11th IEEE International Symposium on High Performance Distributed Computing, Washington, DC, p. 225. IEEE Computer Society, Los Alamitos (2002)

  4. Zhao, Y., Dobson, J., Foster, I., Moreau, L., Wilde, M.: A notation and system for expressing and executing cleanly typed workflows on messy scientific data. SIGMOD Rec. 34(3), 37–43 (2005)

    Article  Google Scholar 

  5. Annis, J., Zhao, Y., Voeckler, J., Wilde, M., Kent, S., Foster, I.: Applying Chimera virtual data concepts to cluster finding in the Sloan Sky Survey. In: Supercomputing’02: Proceedings of the 2002 ACM/IEEE Conference on Supercomputing, Los Alamitos, CA, pp. 1–14. IEEE Computer Society Press, Silver Spring (2002)

  6. Dogan, A., Özgüner, F.: Biobjective scheduling algorithms for execution time-reliability trade-off in heterogeneous computing systems. Comput. J. 48(3), 300–314 (2005)

    Article  Google Scholar 

  7. Yu, J.: A taxonomy of scientific workflow systems for grid computing. Acm Sigmod Rec. 34(3), 44–49 (2005)

    Article  Google Scholar 

  8. Byun, E.J., Choi, S.J., Baik, M.S., Gil, J.M., Park, C.Y., Hwang, C.S.: Mjsa: Markov job scheduler based on availability in desktop grid computing environment. Futur. Gener. Comput. Syst. 23(4), 616–622 (2007)

    Article  Google Scholar 

  9. Hönig, U., Schiffmann, W.: A Meta-algorithm for scheduling multiple DAGs in homogeneous system environments. In: Proceedings of the eighteenth IASTED International Conference on Parallel and Distributed Computing and Systems (PDCS’06), Dallas USA (2006)

  10. Zhao, H., Sakellariou, R.: Scheduling multiple DAGs onto heterogeneous systems. In: Proceedings of the Parallel and Distributed Processing Symposium, IPDPS’06, IEEE Computer Society, Washington, DC (2006)

  11. Hirales-Carbajal, A., Tchernykh, A., Yahyapour, R., González-García, J.L., Röblitz, T., Ramírez-Alcaraz, J.M.: Multiple workflow scheduling strategies with user run time estimates on a grid. J. Grid Comput. 10(2), 325–346 (2012)

    Article  Google Scholar 

  12. Bittencourt, L.F., Madeira, E.R.M.: Towards the scheduling of multiple workflows on computational grids. J. Grid Comput. 8(3), 419–441 (2010)

    Article  Google Scholar 

  13. Tsai, Y.L., Liu, H.C., Huang, K.C.: Adaptive dual-criteria task group allocation for clustering-based multi-workflow scheduling on parallel computing platform. J. Supercomput. 71(10), 3811–3831 (2015)

    Article  Google Scholar 

  14. Yu, Z., Shi, W.: A planner-guided scheduling strategy for multiple workflow applications. In: Proceedings of the Parallel Processing Workshops (ICPPW), 2008 International Conference, pp. 1–8 (2008)

  15. Hsu, C.C., Huang, K.C., Wang, F.J.: Online scheduling of workflow applications in grid environments. Futur. Gener. Comput. Syst. 27(6), 860–870 (2011)

    Article  Google Scholar 

  16. Arabnejad, H., Barbosa, J.: Fairness resource sharing for dynamic workflow scheduling on heterogeneous systems. In: Proceedings of the 10th IEEE International Symposium on parallel and distributed processing with applications, Madrid, pp. 633–639 (2012)

  17. Stavrinides, G.L., Karatza, H.D.: Scheduling real-time DAGs in heterogeneous clusters by combining imprecise computations and bin packing techniques for the exploitation of schedule holes. Futur. Gener. Comput. Syst. 28(7), 977–988 (2012)

    Article  Google Scholar 

  18. Tang, S., Lee, B.S., He, B.: Dynamicmr: a dynamic slot allocation optimization framework for mapreduce clusters. IEEE Trans. Cloud Comput. 2(3), 333–347 (2014)

    Article  Google Scholar 

  19. Tian, G.Z., Xiao, C.B., Xie, J.Q.: Scheduling and fair cost-optimizing methods for concurrent multiple dags with deadline sharing resources. Chin. J. Comput. 37(7), 1607–1619 (2014). (in Chinese with English abstract)

    Google Scholar 

  20. Topcuouglu, H., Hariri, S., Wu, M.Y.: Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans. Parallel Distrib. Syst. 13(3), 260–274 (2002)

    Article  Google Scholar 

  21. Bochenina, K., Butakov, N., Boukhanovsky, A.: Static scheduling of multiple workflows with soft deadlines in non-dedicated heterogeneous environments. Futur. Gener. Comput. Syst. 55, 51–61 (2015)

    Article  Google Scholar 

  22. Bittencourt, L.F., Madeira, E.R.M.: Hcoc: a cost optimization algorithm for workflow scheduling in hybrid clouds. J. Internet Serv. Appl. 2(3), 207–227 (2011)

    Article  Google Scholar 

  23. Tian, G. Z.: Research several problems of scheduling multiple DAGs sharing resources [Ph.D. Thesis], Beijing University of Technology, 2014 (in Chinese with English abstract)

  24. Abrishami, S., Naghibzadeh, M., Epema, D.H.J.: Deadline-constrained workflow scheduling algorithms for infrastructure as a service clouds. Futur. Gener. Comput. Syst. 29(1), 158–169 (2013)

    Article  Google Scholar 

  25. Boregowda, U., Chakravarthy, V.: Multiple DAG applications scheduling on a cluster of processors. In: International Conference on Computer Science, Engineering and Applications, pp. 63–73 (2014)

  26. Garey, M.R., Johnson, D.S.: Computers and Intractability: A Guide to the Theory of NP-Completeness. W.H. Freeman and Company, New York (1979)

    MATH  Google Scholar 

  27. Rodriguez, M.A., Buyya, R.: Deadline based resource provisioning and scheduling algorithm for scientific workflows on clouds. IEEE Trans. Cloud Comput. 2(2), 222–235 (2014)

    Article  Google Scholar 

  28. Zhu, Z., Zhang, G., Li, M., Liu, X.: Evolutionary multi-objective workflow scheduling in cloud. IEEE Trans. Parallel Distrib. Syst. 27(5), 1344–1357 (2016)

    Article  Google Scholar 

Download references

Acknowledgements

This work was financially supported by Beijing Natural Science Foundation (4162007), the National Nature Science Foundation of China (61363004) and Natural Science Foundation of China (61501008).

Author information

Authors and Affiliations

  1. School of Computer Science, Beijing University of Technology, Beijing, 100124, People’s Republic of China

    Xiujie Xu, Chuangbai Xiao & Ting Sun

  2. School of Management Engineering, Shandong Jianzhu University, Jinan, 250101, People’s Republic of China

    Xiujie Xu

  3. Department of Computer Information and Engineering, Changzhou Institute of Technology, Changzhou, 213002, People’s Republic of China

    Guozhong Tian

Authors
  1. Xiujie Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chuangbai Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guozhong Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ting Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiujie Xu.

Ethics declarations

Conflict of interest

The author confirms that this article content has no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, X., Xiao, C., Tian, G. et al. Expansion slot backfill scheduling for concurrent workflows with deadline on heterogeneous resources. Cluster Comput 20, 471–483 (2017). https://doi.org/10.1007/s10586-017-0751-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-017-0751-5

Keywords

Search

Navigation