Skip to main content
SpringerLink
Log in

Exploring the Impact of Node Failures on the Resource Allocation for Parallel Jobs

  • Conference paper
  • First Online:
Euro-Par 2021: Parallel Processing Workshops (Euro-Par 2021)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13098))

Included in the following conference series:

  • 711 Accesses

Abstract

Increasing the size and complexity of modern HPC systems also increases the probability of various types of failures. Failures may disrupt application execution and waste valuable system resources due to failed executions. In this work, we explore the effect of node failures on the completion times of MPI parallel jobs. We introduce a simulation environment that generates synthetic traces of node failures, assuming that the times between failures for each node are independently distributed, following the same distribution but with different parameters. To highlight the importance of failure-awareness for resource allocation, we compare two failure-oblivious resource allocation approaches with one that considers node failure probabilities before assigning a partition to a job: a heuristic that randomly selects the partition for a job, and Slurm’s linear resource allocation policy. We present results for a case study that assumes a 4D-torus topology and a Weibull distribution for each node’s time between failures, and considers several different traces of node failures, capturing different failure patterns. For the synthetic traces explored, the benefit is more prominent for longer jobs, up to 82% depending on the trace, when compared with Slurm and a failure-oblivious heuristic. For shorter jobs, benefits are noticeable for systems with more frequent failures.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Cappello, F., Al, G., Gropp, W., Kale, S., Kramer, B., Snir, M.: Toward exascale resilience: 2014 update. Supercomput. Front. Innov.: Int. J. 1(1), 5–28 (2014)

    Google Scholar 

  2. Casanova, H., Giersch, A., Legrand, A., Quinson, M., Suter, F.: Versatile, scalable, and accurate simulation of distributed applications and platforms. J. Parallel Distrib. Comput. 74(10), 2899–2917 (2014)

    Article  Google Scholar 

  3. Dogan, A., Ozguner, F.: Matching and scheduling algorithms for minimizing execution time and failure probability of applications in heterogeneous computing. IEEE Trans. Parallel Distrib. Syst. 13(3), 308–323 (2002)

    Article  Google Scholar 

  4. El-Sayed, N., Zhu, H., Schroeder, B.: Learning from failure across multiple clusters: a trace-driven approach to understanding, predicting, and mitigating job terminations. In: 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), pp. 1333–1344 (2017)

    Google Scholar 

  5. Elnozahy, E.N.: System resilience at extreme scale. Technical report. Defense Advanced Research Project Agency (2008)

    Google Scholar 

  6. Engelmann, C., Lauer, F.: Facilitating co-design for extreme-scale systems through lightweight simulation. In: 2010 IEEE International Conference on Cluster Computing Workshops and Posters (CLUSTER WORKSHOPS), pp. 1–8 (2010)

    Google Scholar 

  7. Engelmann, C., Naughton, T.: Toward a performance/resilience tool for hardware/software co-design of high-performance computing systems. In: 2013 42nd International Conference on Parallel Processing, pp. 960–969 (2013)

    Google Scholar 

  8. ETP4HPC-SRA 4: Strategic Research Agenda for High Performance Computing in Europe. https://www.etp4hpc.eu/pujades/files/ETP4HPC_SRA4_2020_web.pdf

  9. Fu, S.: Failure-aware resource management for high-availability computing clusters with distributed virtual machines. J. Parallel Distrib. Comput. 70(4), 384–393 (2010)

    Article  Google Scholar 

  10. Gottumukkala, N.R., Leangsuksun, C.B., Taerat, N., Nassar, R., Scott, S.L.: Reliability-aware resource allocation in HPC systems. In: 2007 IEEE International Conference on Cluster Computing, pp. 312–321 (2007)

    Google Scholar 

  11. Gottumukkala, N.R., Nassar, R., Paun, M., Leangsuksun, C.B., Scott, S.L.: Reliability of a system of k nodes for high performance computing applications. IEEE Trans. Reliab. 59(1), 162–169 (2010)

    Article  Google Scholar 

  12. Hakem, M., Butelle, F.: Reliability and scheduling on systems subject to failures. In: 2007 International Conference on Parallel Processing (ICPP 2007), pp. 38–38 (2007)

    Google Scholar 

  13. Heien, E., LaPine, D., Kondo, D., Kramer, B., Gainaru, A., Cappello, F.: Modeling and tolerating heterogeneous failures in large parallel systems. In: SC 2011: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–11 (2011)

    Google Scholar 

  14. Choo, H., Yoo, S.-M., Youn, H.Y.: Processor scheduling and allocation for 3D torus multicomputer systems. IEEE Trans. Parallel Distrib. Syst. 11(5), 475–484 (2000)

    Article  Google Scholar 

  15. Jauk, D., Yang, D., Schulz, M.: Predicting faults in high performance computing systems: an in-depth survey of the state-of-the-practice. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019, pp. 30:1–30:13. ACM, New York (2019)

    Google Scholar 

  16. Levy, S., Topp, B., Ferreira, K.B., Arnold, D., Hoefler, T., Widener, P.: Using simulation to evaluate the performance of resilience strategies at scale. In: Jarvis, S.A., Wright, S.A., Hammond, S.D. (eds.) PMBS 2013. LNCS, vol. 8551, pp. 91–114. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10214-6_5

    Chapter  Google Scholar 

  17. Machida, F., Kawato, M., Maeno, Y.: Redundant virtual machine placement for fault-tolerant consolidated server clusters. In: 2010 IEEE Network Operations and Management Symposium - NOMS 2010, pp. 32–39 (2010)

    Google Scholar 

  18. Martino, C.D., Kramer, W., Kalbarczyk, Z., Iyer, R.: Measuring and understanding extreme-scale application resilience: a field study of 5,000,000 HPC application runs. In: 2015 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, pp. 25–36 (2015)

    Google Scholar 

  19. Oliner, A.J., Sahoo, R.K., Moreira, J.E., Gupta, M., Sivasubramaniam, A.: Fault-aware job scheduling for BlueGene/L systems. In: Proceedings of the 18th International Parallel and Distributed Processing Symposium, p. 64 (2004)

    Google Scholar 

  20. Schroeder, B., Gibson, G.: Understanding failures in Petascale computers. J. Phys.: Conf. Ser. 78 (2007)

    Google Scholar 

  21. Schroeder, B., Gibson, G.A.: A large-scale study of failures in high-performance computing systems. IEEE Trans. Depend. Secure Comput. 7(4), 337–350 (2010)

    Article  Google Scholar 

  22. Slurm Resource Selection Plugin. https://slurm.schedmd.com/selectplugins.html

  23. Snir, M., et al.: Addressing failures in exascale computing. In: ICiS Workshop ANL/MCS-TM-332, April 2013

    Google Scholar 

  24. Tikotekar, A., Vallee, G., Naughton, T., Scott, S.L., Leangsuksun, C.: Evaluation of fault-tolerant policies using simulation. In: 2007 IEEE International Conference on Cluster Computing, pp. 303–311 (2007)

    Google Scholar 

  25. Tiwari, D., Gupta, S., Vazhkudai, S.S.: Lazy checkpointing: exploiting temporal locality in failures to mitigate checkpointing overheads on extreme-scale systems. In: 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, pp. 25–36 (2014)

    Google Scholar 

  26. Vardas, I., Ploumidis, M., Marazakis, M.: Towards communication profile, topology and node failure aware process placement. In: 2020 IEEE 32nd International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), pp. 241–248 (2020)

    Google Scholar 

  27. Li, Y., Lan, Z.: Exploit failure prediction for adaptive fault-tolerance in cluster computing. In: Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGRID 2006), vol. 1, pp. 8 p. 538 (2006)

    Google Scholar 

  28. Yigitbasi, N., Gallet, M., Kondo, D., Iosup, A., Epema, D.: Analysis and modeling of time-correlated failures in large-scale distributed systems. In: 2010 11th IEEE/ACM International Conference on Grid Computing, pp. 65–72 (2010)

    Google Scholar 

  29. Yoo, A.B., Jette, M.A., Grondona, M.: SLURM: simple Linux utility for resource management. In: Feitelson, D., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2003. LNCS, vol. 2862, pp. 44–60. Springer, Heidelberg (2003). https://doi.org/10.1007/10968987_3

    Chapter  Google Scholar 

  30. Zhang, Y., Squillante, M.S., Sivasubramaniam, A., Sahoo, R.K.: Performance implications of failures in large-scale cluster scheduling. In: Feitelson, D.G., Rudolph, L., Schwiegelshohn, U. (eds.) JSSPP 2004. LNCS, vol. 3277, pp. 233–252. Springer, Heidelberg (2005). https://doi.org/10.1007/11407522_13

    Chapter  Google Scholar 

Download references

Acknowledgments

This research has received funding from the European Union’s Horizon 2020/EuroHPC research and innovation programme under grant agreements 955606 (DEEP-SEA) and 754337 (EuroEXA). National contributions from the involved state members match the EuroHPC funding.

Author information

Authors and Affiliations

  1. Institute of Computer Science (ICS), Foundation for Research and Technology – Hellas (FORTH) Greece, 100 N. Plastira Av., Vassilika Vouton, Heraklion, 70013, Greece

    Ioannis Vardas, Manolis Ploumidis & Manolis Marazakis

Authors
  1. Ioannis Vardas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manolis Ploumidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manolis Marazakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ioannis Vardas .

Editor information

Editors and Affiliations

  1. University of Lisbon, Lisbon, Portugal

    Ricardo Chaves

  2. Department of Computer Engineering, CiTIUS, University of Santiago de Compostela, Santiago de Compostela, La Coruña, Spain

    Dora B. Heras

  3. University of Lisbon, Lisbon, Portugal

    Aleksandar Ilic

  4. Koç University, Istanbul, Turkey

    Didem Unat

  5. Barcelona Supercomputing Center, Barcelona, Spain

    Rosa M. Badia

  6. University of Stirling, Stirling, UK

    Andrea Bracciali

  7. Louisiana State University, Baton Rouge, USA

    Patrick Diehl

  8. Mathematics and Computer Science, Argonne National Laboratory, Lemont, IL, USA

    Anshu Dubey

  9. Ajou University, Suwon, Korea (Republic of)

    Oh Sangyoon

  10. Tennessee Technological University, Cookeville, TN, USA

    Stephen L. Scott

  11. University of Pisa, Pisa, Italy

    Laura Ricci

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Vardas, I., Ploumidis, M., Marazakis, M. (2022). Exploring the Impact of Node Failures on the Resource Allocation for Parallel Jobs. In: Chaves, R., et al. Euro-Par 2021: Parallel Processing Workshops. Euro-Par 2021. Lecture Notes in Computer Science, vol 13098. Springer, Cham. https://doi.org/10.1007/978-3-031-06156-1_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-06156-1_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06155-4

  • Online ISBN: 978-3-031-06156-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation