Skip to main content
Springer Nature Link
Log in

Scheduling and planning job execution of loosely coupled applications

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Growth in availability of data collection devices has allowed individual researchers to gain access to large quantities of data that needs to be analyzed. As a result, many labs and departments have acquired considerable compute resources. However, effective and efficient utilization of those resources remains a barrier for the individual researchers because the distributed computing environments are difficult to understand and control. We introduce a methodology and a tool that automatically manipulates and understands job submission parameters to realize a range of job execution alternatives across a distributed compute infrastructure. Generated alternatives are presented to a user at the time of job submission in the form of tradeoffs mapped onto two conflicting objectives, namely job cost and runtime. Such presentation of job execution alternatives allows a user to immediately and quantitatively observe viable options regarding their job execution, and thus allows the user to interact with the environment at a true service level. Generated job execution alternatives have been tested through simulation and on real-world resources and, in both cases, the average accuracy of the runtime of the generated and perceived job alternatives is within 5%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Similar content being viewed by others

References

  1. Segaran T, Hammerbacher J (eds) (2009) Beautiful data: the stories behind elegant data solutions. O’Reilly Media, Sebastopol

    Google Scholar 

  2. Afgan E, Bangalore P (2007) Performance characterization of BLAST for the grid. In: IEEE 7th international symposium on bioinformatics & bioengineering (IEEE BIBE 2007). Boston, MA, pp 1394–1398

    Chapter  Google Scholar 

  3. Lee CB, Snavely A (2006) On the user–scheduler dialogue: studies of user-provided runtime estimates and utility functions. Int J High Perform Comput Appl 20:495–506

    Article  Google Scholar 

  4. Berman F (1998) High-performance schedulers. In: Foster I, Kesselman C (eds) The grid: blueprint for a new computing infrastructure. Morgan Kaufmann, San Francisco, pp 279–309

    Google Scholar 

  5. Afgan E, Bangalore P (2008) Embarrassingly parallel jobs are not embarrassingly easy to schedule on the grid. Presented at the SC08 international conference for high performance, networking, storage and analysis—workshop on many-task computing on grids and supercomputers, Austin, TX

  6. Leung JY-T (ed) (2004) Handbook of scheduling: algorithms, models, and performance analysis. CRC Press, Boca Raton

    MATH  Google Scholar 

  7. Buyya R, Murshed M, Abramson D, Venugopa S (2005) Scheduling parameter sweep applications on global grids: a deadline and budget constrained cost-time optimization algorithm. Softw Pract Exp 35:491–512

    Article  Google Scholar 

  8. Iosup A, Dumitrescu C, Epema DH, Li H, Wolters L (2006) How are real grids used? The analysis of four grid traces and its implications. In: International conference on grid computing 2006, Barcelona, Spain, pp 262–269

    Google Scholar 

  9. Abramson D, Giddy J, Kotler L (2000) High performance parametric modeling with Nimrod/G: killer application for the global grid. In: International parallel and distributed processing symposium (IPDPS), Cancun, Mexico, pp 520–528

    Chapter  Google Scholar 

  10. Foster I, Kesselman C (eds) (1999) The grid: blueprint for a new computing infrastructure. Morgan Kaufmann, San Mateo

    Google Scholar 

  11. Berman FD, Wolski R, Figueira S, Schopf J, Shao G (1996) Application-level scheduling on distributed heterogeneous networks. In: Supercomputing ’96, Pittsburgh, PA, p 28

    Google Scholar 

  12. Casanova H, Obertelli G, Berman F, Wolski R (2000) The AppLeS parameter sweep template: user-level middleware for the grid. In: Supercomputing 2000. Dallas, TX

    Google Scholar 

  13. Berman F, Chien A et al (2001) The GrADS project: software support for high-level grid application development. Int J High Perform Comput Appl 15:327–344

    Article  Google Scholar 

  14. Dail H, Berman F, Casanova H (2003) A decoupled scheduling approach for grid application development environments. J Parallel Distrib Comput 63:505–524

    Article  MATH  Google Scholar 

  15. Buyya R, Abramson D, Giddy J (2000) Nimrod-G: an architecture for a resource management and scheduling in a global computational grid. In: 4th international conference and exhibition on high performance computing in Asia-Pacific region (HPC ASIA 2000), Beijing, China, pp 283–289

    Chapter  Google Scholar 

  16. Smith C (2003) Open source metascheduling for virtual organizations with the community scheduler framework (CSF). Platform Comput Whitepaper, August

  17. Dvořák F, Kouř D et al (2006) gLite job provenance. In: Provenance and annotation of data. Springer, Berlin, pp 246–253

    Google Scholar 

  18. Montero RS, Huedo E, Llorente IM (2006) Grid scheduling infrastructures based on the GridWay meta-scheduler. IEEE Technical Committee on Scalable Computing (TCSC) Newsletter, vol 8

  19. Kertesz A, Kacsuk P (2007) A taxonomy of grid resource brokers. In: Kacsuk P et al (eds) Distributed and parallel systems from cluster to grid computing. Springer, Berlin, pp 201–210

    Google Scholar 

  20. Solomon M (2004) The classad language reference manual, May. Available: http://www.cs.wisc.edu/condor/classad/refman/

  21. Wieczorek M, Podlipnig S, Prodan R, Fahringer T (2008) Bi-criteria scheduling of scientific workflows for the grid. In: 2008 eighth IEEE international symposium on cluster computing and the grid (ccGrid), Lyon, France, pp 9–16

    Chapter  Google Scholar 

  22. Yu J, Kirley M, Buyya R (2007) Multi-objective planning for workflow execution on grids. In: Grid 2007, Austin, TX, pp 10–17

    Google Scholar 

  23. Li C, Li L (2007) Utility-based QoS optimisation strategy for multi-criteria scheduling on the grid. J Parallel Distrib Comput 67:142–153

    Article  MATH  Google Scholar 

  24. Kumar V (2002) Introduction to parallel computing. Addison-Wesley, Longman, Boston

    Google Scholar 

  25. Afgan E, Bangalore P (2007) Application specification language (ASL)—a language for describing applications in grid computing. In: The 4th international conference on grid services engineering and management—GSEM 2007, Leipzig, Germany, pp 24–38

    Google Scholar 

  26. Elmroth E, Tordsson J, Fahringer T, Nadeem F, Gruber R, Keller V (2008) Three complementary performance prediction methods for grid applications. In: CoreGRID integration workshop 2008, Heraklion, Greece

    Google Scholar 

  27. Afgan E, Bangalore P, Duncan D (2009) GridAtlas—a grid application and resource configuration repository and discovery service. In: IEEE Cluster 2009, New Orleans, LA, p 10

    Google Scholar 

  28. Afgan E, Bangalore P, Mukkai S, Yammanuru S (2008) Design and implementation of a readily available historical application performance database (AppDB) for grid. University of Alabama at Birmingham (UAB), Birmingham, AL UABCIS-TR-2008-0506-1, May 6

  29. Siddiqui M, Villazón A, Fahringer T (2006) Grid allocation and reservation—grid capacity planning with negotiation-based advance reservation for optimized QoS. In: 2006 ACM/IEEE conference on supercomputing, Tampa, FL, pp 21–35

    Google Scholar 

  30. Sotomayor B, Keahey K, Foster I (2008) Combining batch execution and leasing using virtual machines. In: ACM/IEEE international symposium on high performance distributed computing 2008 (HPDC 2008), Boston, MA, pp 87–96

    Google Scholar 

  31. Olofsson P (2005) Probability, statistics, and stochastic processes, 1st edn. Wiley-Interscience, New York

    Book  MATH  Google Scholar 

  32. Standard Performance Evaluation Corporation, July 24. Available: http://www.spec.org/

  33. Buyya R, Murshed M (2002) GridSim: a toolkit for the modeling and simulation of distributed resource management and scheduling for grid computing. J Concurr Comput Pract Exp (CCPE) 14:1175–1220

    Article  MATH  Google Scholar 

  34. GridWay (2009) Job template options, Feb 16. Available: http://www.gridway.org/documentation/stable5.4/user/gridway-user-functionality.html#id2578278

  35. Pfeiffer W, Wright NJ (2008) Modeling and predicting application performance on parallel computers using HPC challenge benchmarks. In: IEEE symposium on parallel and distributed processing (IPDPS), Miami, FL, pp 1–12

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Biology, Emory University, Atlanta, GA, 30322, USA

    Enis Afgan

  2. Department of Computer and Information Sciences, University of Alabama at Birmingham, Birmingham, AL, 35294, USA

    Purushotham Bangalore

  3. University of Zagreb, Getaldiceva 2, 10 000, Zagreb, Croatia

    Tibor Skala

Authors
  1. Enis Afgan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Purushotham Bangalore
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tibor Skala
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Enis Afgan.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Afgan, E., Bangalore, P. & Skala, T. Scheduling and planning job execution of loosely coupled applications. J Supercomput 59, 1431–1454 (2012). https://doi.org/10.1007/s11227-011-0555-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-011-0555-y

Keywords

Search

Navigation