Skip to main content
SpringerLink
Log in

Cost-aware DAG scheduling algorithms for minimizing execution cost on cloud resources

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Directed acyclic graph (DAG) scheduling is a well-known problem, because a DAG can be used to describe a wide range of complex applications, including scientific applications and parallel computing jobs. Most DAG scheduling algorithms were proposed to minimize the job makespan (i.e., execution time) on a multiprocessor computer or cluster. However, as the cost-driven public cloud services have become an attractive and popular platform for providing computing resources, cost minimization emerges as a new critical issue. Therefore, the objective of this work is to propose and solve the cost optimization problem for scheduling DAGs on an IaaS cloud platform where task scheduling must cope with resource provisioning to achieve the optimal solution. In this paper, we proposed both optimal and heuristic scheduling algorithms, and we evaluated them across a variety of DAGs using the price model from EC2. Comparing to other cost-oblivious DAG schedules that aim to minimize makespan or resource usage, the results show that our cost-aware heuristic algorithm can reduce cost by 20–50 % and achieve a cost within x1.16 of the optimal one.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Adam TL, Chandy KM, Dickson JR (1974) A comparison of list schedules for parallel processing systems. Commun ACM 17(12):685–690

    Article  MATH  Google Scholar 

  2. Ahmad I, Yu-Kwong K (1998) On exploiting task duplication in parallel program scheduling. IEEE Trans Parallel Distrib Syst 9(9):872–892

    Article  Google Scholar 

  3. Almeida VAF, Vasconcelos IMM, Arabe J, Menasce D (1992) Using random task graphs to investigate the potential benefits of heterogeneity in parallel systems. In: ACM/IEEE conference on supercomputing, pp 683–691

  4. Arabnejad H, Barbosa J (2014) List scheduling algorithm for heterogeneous systems by an optimistic cost table. IEEE Trans Parallel Distrib Syst 25(3):682–694

    Article  Google Scholar 

  5. Armbrust M, Fox A, Griffith R, Joseph AD, Katz R, Konwinski A, Lee G, Patterson D, Rabkin A, Stoica I, Zaharia M (2010) A view of cloud computing. Commun ACM 53(4):50–58

    Article  Google Scholar 

  6. Bajaj R, Agrawal D (2004) Improving scheduling of tasks in a heterogeneous environment. IEEE Trans Parallel Distrib Syst 15(2):107–118

    Article  Google Scholar 

  7. Bittencourt L, Madeira E (2011) HCOC: a cost optimization algorithm for workflow scheduling in hybrid clouds. J Internet Serv Appl 2(3):207–227

    Article  Google Scholar 

  8. Canon L, Jeannot E (2010) Evaluation and optimization of the robustness of dag schedules in heterogeneous environments. IEEE Trans Parallel Distrib Syst 21(4):532–546

    Article  Google Scholar 

  9. Casavant TL, Kuhl JG (1988) A taxonomy of scheduling in general-purpose distributed computing systems. IEEE Trans Softw Eng 14(2):141–154

    Article  Google Scholar 

  10. Chen G (2010) Simplified particle swarm optimization algorithm based on particles classification. Int Conf Nat Comput 5:2701–2705

    Google Scholar 

  11. Chen H, Shirazi B, Kavi K, Hurson A (1993) Static scheduling using linear clustering and task duplication. In: Parallel and distributed computing and systems, pp 285–290

  12. Colin J, Chretienne P (1991) C.P.M. scheduling with small computation delays and task duplication. Oper Res 39:680–684

    Article  MATH  Google Scholar 

  13. Cordeiro D, Mounié G, Perarnau S, Trystram D, Vincent J-M, Wagner F (2010) Random graph generation for scheduling simulations. In: Proceedings of the 3rd international ICST conference on simulation tools and techniques, pp 60:1–60:10

  14. Darbha S, Agrawal DP (1998) Optimal scheduling algorithm for distributed-memory machines. IEEE Trans Parallel Distrib Syst 9(1):87–95

    Article  Google Scholar 

  15. Dean J, Ghemawat S (2008) MapReduce: simplified data processing on large clusters. Commun ACM 51(1):107–113

    Article  Google Scholar 

  16. Fujimoto N, Hagihara K (2003) Near-optimal dynamic task scheduling of precedence constrained coarse-grained tasks onto a computational grid. In: Proceedings of international conference on parallel and distributed computing, pp 80–87

  17. Graham R, Lawler L, Lenstra J, Kan A (1979) Optimization and approximation in deterministic sequencing and scheduling: a survey. Ann Discrete Math 5:287–326

    Article  MathSciNet  MATH  Google Scholar 

  18. Hu T (1961) Parallel sequencing and assembly line problems. Oper Res 9(6):841–848

    Article  MathSciNet  Google Scholar 

  19. Juve G, Chervenak A, Deelman E, Bharathi S, Mehta G, Vahi K (2013) Characterizing and profiling scientific workflows. Future Gener Comput Syst 29(3):682–692

    Article  Google Scholar 

  20. Kvas A, Ojstersek M, Zumer V (1994) Evaluation of static program allocation schemes for macro data-flow computer. In: Proceedings of the 20th EUROMICRO conference, pp 573–580

  21. Kwok Y-K, Ahmad I (1996) Dynamic critical-path scheduling: an effective technique for allocating task graphs to multiprocessors. IEEE Trans Parallel Distrib Syst 7(5):506–521

    Article  Google Scholar 

  22. Mao M, Humphrey M (2011) Auto-scaling to minimize cost and meet application deadlines in cloud workflows. ACM/IEEE Conf Supercomput 49(1–49):12

    Google Scholar 

  23. Olston C, Reed B, Srivastava U, Kumar R, Tomkins A (2008) Pig latin: a not-so-foreign language for data processing. In: ACM SIGMOD, pp 1099–1110

  24. Radulescu A, van Gemund A (2000) Fast and effective task scheduling in heterogeneous systems. In: Proceedings of 9th heterogeneous computing workshop, 2000 (HCW 2000), pp 229–238

  25. Raju R, Amudhavel J, Pavithra M, Anuja S, Abinaya B (2014) A heuristic fault tolerant “MapReduce framework for minimizing makespan in hybrid cloud environment. In: International conference on green computing communication and electrical engineering (ICGCCEE), pp 1–4

  26. Rodriguez MA, Buyya R (2014) Deadline based resource provisioningand scheduling algorithm for scientific workflows on clouds. IEEE Trans Cloud Comput 2(2):222–235

    Article  Google Scholar 

  27. Shirazi B, Wang M, Pathak G (1990) Analysis and evaluation of heuristic methods for static task scheduling. J Parallel Distrib Comput 10(3):222–2232

    Article  Google Scholar 

  28. Thusoo A, Sarma JS, Jain N, Shao Z, Chakka P, Anthony S, Liu H, Wyckoff P, Murthy R (2009) Hive: a warehousing solution over a Map-Reduce framework. Proc VLDB Endow 2(2):1626–1629

    Article  Google Scholar 

  29. Topcuouglu H, Hariri S, Wu M-Y (2002) Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans Parallel Distrib Syst 13(3):260–274

    Article  Google Scholar 

  30. Wu M-Y, Gajski D (1990) Hypertool: a programming aid for message-passing systems. IEEE Trans Parallel Distrib Syst 1(3):330–343

    Article  Google Scholar 

  31. Yang C-H, Lee P, Chung Y-C (2007) Improving static task scheduling in heterogeneous and homogeneous computing systems. In: ICPP, pp 45–45

  32. Yang T, Gerasoulis A (1994) DSC: scheduling parallel tasks on an unbounded number of processors. IEEE Trans Parallel Distrib Syst 5(9):951–967

    Article  Google Scholar 

  33. Yu WH (1984) Lu decomposition on a multiprocessing system with communications delay, Ph.D. thesis. University of California, Berkeley. AAI8427141

  34. Yu Y, Isard M, Fetterly D, Budiu M, Erlingsson U, Gunda PK, Currey J (2008) DryadLINQ: a system for general-purpose distributed data-parallel computing using a high-level language. In: Proceedings of the USENIX conference on operating systems design and implementation, pp 1–14

  35. Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I (2010) Spark: cluster computing with working sets. In: Proceedings of the 2Nd USENIX conference on hot topics in cloud computing, HotCloud’10. USENIX Association, Berkeley, p 10

  36. Zhang C, De Sterck H (2010) Cloudbatch: a batch job queuing system on clouds with hadoop and hbase. In: IEEE international conference on cloud computing technology and science (CloudCom), pp 368–375

  37. Zhou AC, He B (2014) Transformation-based monetary cost optimizations for workflows in the cloud. IEEE Trans Cloud Comput 2(1):85–98

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Information Systems and Applications, National Tsing Hua University, Hsinchu, Taiwan

    Moïse W. Convolbo

  2. Department of Computer Science, National Tsing Hua University, Hsinchu, Taiwan

    Jerry Chou

Authors
  1. Moïse W. Convolbo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jerry Chou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jerry Chou.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Convolbo, M.W., Chou, J. Cost-aware DAG scheduling algorithms for minimizing execution cost on cloud resources. J Supercomput 72, 985–1012 (2016). https://doi.org/10.1007/s11227-016-1637-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-016-1637-7

Keywords

Search

Navigation