Skip to main content
SpringerLink
Log in

Scheduling Concurrent Workflows in HPC Cloud through Exploiting Schedule Gaps

  • Conference paper
Algorithms and Architectures for Parallel Processing (ICA3PP 2011)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 7016))

Abstract

Many large-scale scientific applications are usually constructed as workflows due to large amounts of interrelated computation and communication. Workflow scheduling has long been a research topic in parallel and distributed computing. However, most previous research focuses on single workflow scheduling. As cloud computing emerges, users can now have easy access to on-demand high performance computing resources, usually called HPC cloud. Since HPC cloud has to serve many users simultaneously, it is common that many workflows submitted from different users are running concurrently. Therefore, how to schedule concurrent workflows efficiently becomes an important issue in HPC cloud environments. Due to the dependency and communication costs between tasks in a workflow, there usually are gaps formed in the schedule of a workflow. In this paper, we propose a method which exploits such schedule gaps to efficiently schedule concurrent workflows in HPC cloud. The proposed scheduling method was evaluated with a series of simulation experiments and compared to the existing method in the literature. The results indicate that our method can deliver good performance and outperform the existing method significantly in terms of average makespan, up to 18% performance improvement.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bittencourt, L.F., Madeira, E.R.M.: Towards the Scheduling of Multiple Workflows on Computational Grids. Journal of Grid Computing 8, 419–441 (2009)

    Article  Google Scholar 

  2. Kwok, Y.K., Ahmad, I.: Static Scheduling Algorithms for Allocating Directed Task Graphs to Multiprocessors. ACM Computing Surveys 31(4), 406–471 (1999)

    Article  Google Scholar 

  3. Adam, T.L., Chandy, K.M., Dickson, J.R.: A Comparison of List Schedules for Parallel Processing Systems. Communications of the ACM 17(12), 685–690 (1974)

    Article  MATH  Google Scholar 

  4. Bittencourt, L.F., Madeira, E.R.M.: Fulfilling Task Dependence Gaps for Workflow Scheduling on Grids. In: 3rd IEEE International Conference on Signal-Image Technology and Internet Based Systems, pp. 468–475 (2007)

    Google Scholar 

  5. Stavrinides, G.L., Karatza, H.D.: Scheduling Multiple Task Graphs in Heterogeneous Distributed Real-Time Systems by Exploiting Schedule Holes with Bin Packing Techniques. Simulation Modelling Practice and Theory, vol 19(1), 540–552 (2011)

    Article  Google Scholar 

  6. Bittencourt, L.F., Sakellariou, R., Madeira, E.R.M.: DAG Scheduling Using a Lookahead Variant of the Heterogeneous Earliest Finish Time Algorithm. In: 18th ‘Conference on Parallel, Distributed and Network-based Processing, pp. 27–34 (2010)

    Google Scholar 

  7. Wieczorek, M., Prodan, R., Hoheisel, A.: Taxonomies of the Multi-Criteria Grid Workflow Scheduling Problem. In: Grid Middleware and Services, pp. 237–264 (2008)

    Google Scholar 

  8. Rahman, M., Ranjan, R., Buyya, R.: Cooperative and Decentralized Workflow Scheduling in Global Grids. Future Generation Computer Systems 26, 753–768 (2010)

    Article  Google Scholar 

  9. Ding, F., Zhang, R., Ruan, K., Lin, J., Zhao, Z.: A QoS-based Scheduling Approach for Complex Workflow Applications. In: 5th Annual ChinaGrid Conference, pp. 67–73 (2010)

    Google Scholar 

  10. Bittencourt, L.F., Madeira, E.R.M.: A Performance-Oriented Adaptive Scheduler for Dependent Tasks on Grids. Concurrency and Computation: Practice and Experience 20, 1029–1049 (2008)

    Article  Google Scholar 

  11. Gary, M.R., Johnson, D.S.: Computers and Intractability: A Guide to the Theory of NP-Completeness. W.H. Freeman and Co, New York (1979)

    Google Scholar 

  12. Ullman, J.D.: NP-Complete Scheduling Problems. Journal of Computer and Systems Sciences 10, 384–393 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  13. Zhao, H., Sakellarious, R.: Scheduling Multiple DAGs onto Heterogeneous Systems. In: 15th Heterogeneous Computing Workshop, 14 pp (2006)

    Google Scholar 

  14. Yu, Z., Shi, W.: A Planner-Guided Scheduling Strategy for Multiple Workflow Applications. In: 37th International Conference on Parallel Processing Workshops, pp. 8–12 (2008)

    Google Scholar 

  15. N’takpé, T., Suter, F.: Concurrent Scheduling of Parallel Task Graphs on Multi-Clusters Using Constrained Resource Allocations. In: IEEE International Symposium on Parallel and Distributed Processing, pp. 1–8 (2009)

    Google Scholar 

  16. Kwok, Y., Ahmad, I.: Dynamic Critical-Path Scheduling: An Effective Technique for Allocation Task Graphs to Multi-processors. IEEE Transactions on Parallel and Distributed Systems 7(5), 506–521 (1996)

    Article  Google Scholar 

  17. Sih, G.C., Lee, E.A.: A Compile-Time Scheduling Heuristic for Interconnection-Constrained Heterogeneous Processor Architectures. IEEE Transactions on Parallel and Distributed Systems 4(2), 175–186 (1993)

    Article  Google Scholar 

  18. EI-Rewini, H., Lewis, T.G.: Scheduling Parallel Program Tasks onto Arbitrary Target Machines. Journal of Parallel and Distributed Computing 9, 138–153 (1990)

    Article  Google Scholar 

  19. Yang, T., Gerasoulis, A.: DSC: Scheduling Parallel Tasks on an Unbounded Number of Processors. IEEE Transactions on Parallel and Distributed Systems 5(9), 951–967 (1994)

    Article  Google Scholar 

  20. Park, G., Shirazi, B., Marquis, J.: DFRN: A New Approach for Duplication Based Scheduling for Distributed Memory Multi-processor Systems. In: International Conference on Parallel Processing, pp. 157–166 (1997)

    Google Scholar 

  21. Mandal, A., Kennedy, K., Koelbel, C., Marin, G., Mellor-Crummey, J., Liu, B., Johnsson, L.: Scheduling Strategies for Mapping Application Workflows onto the Grid. In: 14th IEEE Symposium on High Performance Distributed Computing, pp. 125–134 (2005)

    Google Scholar 

  22. Braun, T.D., Siegel, H.J., Beck, N., Bölöni, L.L., Maheswaran, M., Reuther, A.I., Robertson, J.P., Theys, M.D., Yao, B., Hensgen, D., Freund, R.F.: A Comparison of Eleven Static Heuristics for Mapping a Class of Independent Tasks onto Heterogeneous Distributed Computing Systems. Journal of Parallel and Distributed Computing 61(6), 810–837 (2001)

    Article  MATH  Google Scholar 

  23. Hofmann, P., Woods, D.: Cloud Computing: The Limits of Public Clouds for Business Applications. IEEE Internet Computing, 90–93 (November 2010)

    Google Scholar 

  24. Wei, Y., Blake, M.B.: Service-Oriented Computing and Cloud Computing: Challenges and Opportunities. IEEE Internet Computing, 72–75 (November 2010)

    Google Scholar 

  25. Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R.H., Konwinski, A., Lee, G., Patterson, D.A., Rabkin, A., Stoica, I., Zaharia, M.: Above the Clouds: A Berkeley View of Cloud Computing. Technical report no. UCB/EECS-2009-28, EECS Department, University of California, Berkeley (2009)

    Google Scholar 

  26. salesforce.com, http://www.salesforce.com

  27. Gmail, http://gamil.com

  28. Google App Engine, http://code.google.com/intl/en/appengine

  29. Microsoft Azure Platform, http://www.microsoft.com/windowsazure/

  30. Amazon Elastic Compute Cloud, http://aws.amazon.com/ec2/

  31. Akioka, S., Muraoka, Y.: HPC Benchmarks on Amazon EC2. In: 24th IEEE International Conference on Advanced Information Networking and Applications Workshops, pp. 1029–1034 (2010)

    Google Scholar 

  32. Kim, H.: el-Khamra, Y., Jha, S., Parashar, M.: An Autonomic Approach to Integrated HPC Grid and Cloud Usage. In: 5th IEEE International Conference on e-Science, pp. 366–373 (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer and Information Science, National Taichung University of Education, No. 140, Min-Shen Road, Taichung, Taiwan

    He-Jhan Jiang, Kuo-Chan Huang, Di-Syuan Gu & Po-Jen Shih

  2. National Center for High-Performance Computing, National Applied Research Laboratories, P.O. Box 19-136, Hsinchu, Taiwan

    Hsi-Ya Chang

Authors
  1. He-Jhan Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kuo-Chan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hsi-Ya Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Di-Syuan Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Po-Jen Shih
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Information Technology, Deakin University, Melbourne Burwood Campus, 221 Burwood Highway, 3125,, Burwood, VIC, Australia

    Yang Xiang

  2. ICAR-CNR and University of Calabria, Via P. Bucci 41 C, 87036, Rende (CS), Italy

    Alfredo Cuzzocrea

  3. School of Information Technology, Deakin University, Geelong Waurn Ponds Campus, Pigdons Road, 3217, Geelong, VIC, Australia

    Michael Hobbs

  4. School of Information Technology, Deakin University, Melbourne Burwood Campus, 221 Burwood Highway, 3125, Burwood, VIC, Australia

    Wanlei Zhou

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Jiang, HJ., Huang, KC., Chang, HY., Gu, DS., Shih, PJ. (2011). Scheduling Concurrent Workflows in HPC Cloud through Exploiting Schedule Gaps. In: Xiang, Y., Cuzzocrea, A., Hobbs, M., Zhou, W. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2011. Lecture Notes in Computer Science, vol 7016. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-24650-0_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-24650-0_24

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-24649-4

  • Online ISBN: 978-3-642-24650-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation