Skip to main content
SpringerLink
Log in

A Data Structure for Planning Based Workload Management of Heterogeneous HPC Systems

  • Conference paper
  • First Online:
Job Scheduling Strategies for Parallel Processing (JSSPP 2017)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 10773))

Included in the following conference series:

  • 616 Accesses

Abstract

This paper describes a data structure and a heuristic to plan and map arbitrary resources in complex combinations while applying time dependent constraints. The approach is used in the planning based workload manager OpenCCS at the Paderborn Center for Parallel Computing (PC\(^2\)) to operate heterogeneous clusters with up to 10000 cores. We also show performance results derived from four years of operation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Battre, D., Hovestadt, M., Kao, O., Keller, A., Voss, K.: Planning-based scheduling for SLA-awareness and grid integration. In: Proceedings of the 26th Workshop of the UK Planning and Scheduling Special Interest Group (PlansSIG 2007) (2007)

    Google Scholar 

  2. Brune, M., Gehring, J., Keller, A., Reinefeld, A.: RSD - resource and service description. In: Schaeffer, J. (ed.) High Performance Computing Systems and Applications (HPCS 1998), pp. 193–206. Kluwer Academic Press, Dordrecht (1998)

    Chapter  Google Scholar 

  3. OpenCCS Manual, July 2017. https://www.openccs.eu

  4. Chlumský, V., Klusáček, D., Ruda, M.: The extension of torque scheduler allowing the use of planning and optimization in grids. Comput. Sci. 13(2), 5–19 (2012). https://doi.org/10.7494/csci.2012.13.2.5

    Article  Google Scholar 

  5. Curino, C., Difallah, D.E., Douglas, C., et al.: Reservation-based scheduling: if you’re late don’t blame us! Tech-report MSR-TR-2013-108, Microsoft (2013)

    Google Scholar 

  6. Hovestadt, M., Kao, O., Keller, A., Streit, A.: Scheduling in HPC resource management systems: queuing vs. planning. In: Feitelson, D., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2003. LNCS, vol. 2862, pp. 1–20. Springer, Heidelberg (2003). https://doi.org/10.1007/10968987_1

    Chapter  Google Scholar 

  7. Jyothi, S.A., Curino, C., Menache, I., et al.: Morpheus: towards automated SLOs for enterprise clusters. In: Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2016), November 2016

    Google Scholar 

  8. Kay, J., Lauder, P.: A fair share scheduler. Commun. ACM 31, 44–55 (1998)

    Article  Google Scholar 

  9. Kleban, S.D., Clearwater, S.: Fair share on high performance computing systems: what does fair really mean? In: Proceedings of 3rd IEEE International Symposium on Cluster Computing and the Grid (CCGrid 2003), pp. 145–153. IEEE Computer Society (2003)

    Google Scholar 

  10. Lifka, D.A.: The ANL/IBM SP scheduling system. In: Feitelson, D.G., Rudolph, L. (eds.) JSSPP 1995. LNCS, vol. 949, pp. 295–303. Springer, Heidelberg (1995). https://doi.org/10.1007/3-540-60153-8_35

    Chapter  Google Scholar 

  11. Mu’alem, A., Feitelson, D.G.: Utilization, predictability, workloads, and user runtime estimates in scheduling the IBM SP2 with backfilling. IEEE Trans. Parallel Distrib. Syst. 12(6), 529–543 (2001)

    Article  Google Scholar 

  12. PBSPro Open Source, January 2017. http://www.pbspro.org

  13. PC\(^2\): Paderborn Center for Parallel Computing, July 2017. https://pc2.uni-paderborn.de

  14. Schneider, J., Linnert, B.: List-based data structures for efficient management of advance reservations. Int. J. Parallel Prog. 42, 77–93 (2014). https://doi.org/10.1007/s10766-012-0219-4

    Article  Google Scholar 

  15. Torque, January 2017. http://www.adaptivecomputing.com/products/open-source/torque/

  16. Tumanov, A., Zhu, T., Park, J.W., et al.: TetriSched: global rescheduling with adaptive plan-ahead in dynamic heterogeneous clusters. In: Proceedings of the 11th European Conference on Computer Systems (EuroSys 2016), April 2016. https://doi.org/10.1145/2901318.2901355

  17. Vavilapalli, V.K., Murthy, A.C., Douglas, C., et al.: Apache Hadoop YARN: yet another resource negotiator. In: Proceedings of the 4th Annual Symposium on Cloud Computing (SOCC 2013), October 2013. https://doi.org/10.1145/2523616.2523633

Download references

Acknowledgements

I would like to thank Christoph Kleineweber, Dr. Lars Schäfers, and Dr. Jörn Schumacher for their valuable contribution to the current OpenCCS release.

Author information

Authors and Affiliations

  1. Paderborn Center for Parallel Computing, Paderborn University, 33098, Paderborn, Germany

    Axel Keller

Authors
  1. Axel Keller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Axel Keller .

Editor information

Editors and Affiliations

  1. CESNET, Prague, Czech Republic

    Dalibor Klusáček

  2. Google, Mountain View, California, USA

    Walfredo Cirne

  3. Google, Seattle, Washington, USA

    Narayan Desai

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Keller, A. (2018). A Data Structure for Planning Based Workload Management of Heterogeneous HPC Systems. In: Klusáček, D., Cirne, W., Desai, N. (eds) Job Scheduling Strategies for Parallel Processing. JSSPP 2017. Lecture Notes in Computer Science(), vol 10773. Springer, Cham. https://doi.org/10.1007/978-3-319-77398-8_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-77398-8_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-77397-1

  • Online ISBN: 978-3-319-77398-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation