Skip to main content

Advertisement

Springer Nature Link
Log in

Optimal job packing, a backfill scheduling optimization for a cluster of workstations

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

    We’re sorry, something doesn't seem to be working properly.

    Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

In this paper, we have proposed two backfill scheduling optimizations, i.e., Shortest Width First Backfill (SWFBF) and Shortest Area First Backfill (SAFBF). A near optimal simple, but effective job packing algorithm called the Select-Replace algorithm has also been presented to minimize external fragmentation. Proof of the concept has been given with the help of a simulation study. Five workloads which were derived from a clean version of the parallel workload archive (CTC, LANL, and SDSC. NASA) have been used to evaluate and compare proposed heuristics with previous techniques. With the simple but effective optimizations, significant (56.1%) performance improvement has been achieved as compared to EASY scheduler.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Baker MA, Fox GC, Yau HW (2008) New challenges of parallel job scheduling. In: Lecture notes in computer science, vol 4942. Springer, Berlin, pp 1–23. ISBN 978-3-540-78698-6

    Google Scholar 

  2. Baeza-Yates R, Castillo C, Junqueira F, Plachouras V, Silvestri F (2007) Challenges on distributed web retrieval. In: Data engineering ICDE 2007. IEEE 23rd international conference on, April 2007, pp 6–20

  3. Baker M, Apon A, Buyya R, Jin H (2002) Cluster computing and applications. In: Kent A, Williams J (eds) Encyclopedia of computer science and technology. Dekker, New York, pp 87–125

    Google Scholar 

  4. Mishima H, Lidral AC, Ni J (2008) Application of the Linux cluster for exhaustive window haplotype analysis using the FBAT and unphased programs. Dows Institute for Dental Research, University of Iowa, Iowa City, IA 52242, USA

  5. Shmueli E, Feitelson DG (2005) Backfilling with look ahead to optimize the packing of parallel jobs. J Parallel Distrib Comput 65(9):1090–1107. ISSN 0743-7315

    Article  MATH  Google Scholar 

  6. Zhang Y, Franke H, Moreira JE, Sivasubramaniam A (2002) Improving parallel job scheduling by combining gang scheduling and Backfilling techniques. In: Parallel and distributed processing symposium, IPDPS 2002, pp 133–142. ISBN: 0-7695-0574-0

  7. Lawson B, Smirni E, Puiu D (2002) Self-adaptive backfill scheduling for parallel systems. In: Proceedings of the international conference on parallel processing (ICPP 2002), pp 583–592

  8. Tsafrir D, Etsion Y, Feitelson DG (2007) Backfilling using system-generated predictions rather than user runtime estimates. IEEE Trans Parallel Distrib Syst 18(6):789–803. ISSN: 1045-9219

    Article  Google Scholar 

  9. Tsafrir D, Feitelson DG (2006) The dynamics of backfilling: solving the mystery of why increased inaccuracy may help. In: IEEE international symposium on workload characterization, pp 131–141

  10. Bode B, Halstead DM, Kendall R, Lei Z (2000) Portable batch system and the MAUI scheduler on Linux clusters. In: The proceedings of the 4th annual Linux showcase and conference, Atlanta, October 10–14, 2000

  11. Feitelson D, Weil A (1998) Utilization and predictability in scheduling the IBM SP2 with backfilling. In: Proceedings of the 1st merged international parallel processing symposium and symposium on parallel and distributed processing (IPPS/SPDP-98), Los Alamitos, March 1998, pp 542–547

  12. Jones JP, Nitzberg B (1999) Scheduling for parallel supercomputing: a historical perspective of achievable utilization. In: Job scheduling strategies for parallel processing. Lect notes comput sci, vol 1659. Springer, Berlin, pp 1–16

    Chapter  Google Scholar 

  13. Ward WA Jr, Mahood CL, West JE (2002) Scheduling jobs on parallel systems using a relaxed backfill strategy. In: Job scheduling strategies for parallel processing. Lect notes comput sci, vol 2537. Springer, Berlin, pp 88–102

    Chapter  Google Scholar 

  14. Srinivasan S, Kettimuthu R, Subramani V, Sadayappan P (2002) Selective reservation strategies for backfill job scheduling. In: Job scheduling strategies for parallel processing. Lect notes comput sci, vol 2537. Springer, Berlin, pp 55–71

    Chapter  Google Scholar 

  15. Talby D, Feitelson DG (1999) Supporting priorities and improving utilization of the IBM SP scheduler using slack-based backfilling. In: 13th intl parallel processing symp (IPPS), Apr 1999, pp 513–517

  16. Lawson BG, Smirni E (2002) Multiple-queue backfilling scheduling with priorities and reservations for parallel systems. In: Job scheduling strategies for parallel processing. Lect notes comput sci, vol 2537. Springer, Berlin, pp 72–87

    Chapter  Google Scholar 

  17. Chiang S-H, Arpaci-Dusseau A, Vernon MK (2002) The impact of more accurate requested runtimes on production job scheduling performance. In: Job scheduling strategies for parallel processing. Lect notes comput sci, vol 2537. Springer, Berlin, pp 103–127

    Chapter  Google Scholar 

  18. Parallel workloads archive. http://www.cs.huji.ac.il/labs/parallel/workload/: accessed July 5, 2008

  19. Terashima-Marin H et al (2007) Comparing two models to generate hyper-heuristics for the 2d-regular bin-packing problem. In: Proceedings of the 9th annual conference on genetic and evolutionary computation. ACM, New York, pp 2182–2189. ISBN 978-1-59593-697-4

    Chapter  Google Scholar 

  20. Klusáček D, Matyska L, Rudová H (2008) Alea—grid scheduling simulation environment. In: 7th international conference on parallel processing and applied mathematics (PPAM 2007). Springer, Berlin

    Google Scholar 

  21. Buyya R, Murshed M (2003) GridSim: a toolkit for the modeling and simulation of distributed resource management and scheduling for grid computing. Concurr Comput Pract Exp 14(13–15):1175–1220. Special issue: grid computing environments

    Google Scholar 

  22. Feitelson DG, Tsafrir D (2006) Workload sanitation for performance evaluation. In: Performance analysis of systems and software, 2006 IEEE international symposium on, pp 221–230. ISBN: 1-4244-0186-0

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, COMSATS Institute of Information Technology, Abbottabad, Pakistan

    Syed Munir Hussain Shah & Haroon Rasheed

  2. Department of Information Science, Kuwait University, Kuwait City, Kuwait

    Kalim Qureshi

Authors
  1. Syed Munir Hussain Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kalim Qureshi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haroon Rasheed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Syed Munir Hussain Shah.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hussain Shah, S.M., Qureshi, K. & Rasheed, H. Optimal job packing, a backfill scheduling optimization for a cluster of workstations. J Supercomput 54, 381–399 (2010). https://doi.org/10.1007/s11227-009-0332-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-009-0332-3