Skip to main content
SpringerLink
Log in

Checkpointing Kernel Executions of MPI+CUDA Applications

  • Conference paper
  • First Online:
Euro-Par 2019: Parallel Processing Workshops (Euro-Par 2019)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11997))

Included in the following conference series:

  • 1321 Accesses

Abstract

This paper proposes a new approach to checkpointing MPI applications that use long-running CUDA kernels. It becomes possible to take snapshots of data residing on the GPUs without waiting for kernels to complete. The proposed technique is implemented in the context of the state of the art high performance fault tolerance library FTI. As a result we get an elegant solution to the problem of developing resilient MPI applications where GPU kernels run longer than the mean time between hardware failures. We describe in detail how we checkpoint/restart collaborative MPI-CUDA applications, and we provide an initial evaluation of the proposed approach using the Livermore Unstructured Lagrangian Explicit Shock Hydrodynamics (LULESH) application as a case study.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Available at https://bitbucket.org/maxbaird/cuda_backup.

  2. 2.

    The FTI extension is available at https://github.com/leobago/fti/tree/cuda-dev-kernel-interrupt.

References

  1. Baird, M., Fensch, C., Scholz, S.-B., Šinkarovs, A.: A lightweight approach to gpu resilience. In: Mencagli, G., et al. (eds.) Euro-Par 2018. LNCS, vol. 11339, pp. 826–838. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10549-5_64

    Chapter  Google Scholar 

  2. Bautista-Gomez, L., Tsuboi, S., et al.: FTI: high performance fault tolerance interface for hybrid systems. In: SC 2011, pp. 1–12 (2011). https://doi.org/10.1145/2063384.2063427

  3. Duato, J., Peña, A.J., et al.: rCUDA: reducing the number of GPU-based accelerators in high performance clusters. In: 2010 International Conference on High Performance Computing Simulation, pp. 224–231 (2010). https://doi.org/10.1109/HPCS.2010.5547126

  4. Garg, R., Mohan, A., et al.: CRUM: checkpoint-restart support for CUDA’s unified memory. In: CLUSTER 2018, pp. 302–313 (2018). https://doi.org/10.1109/CLUSTER.2018.00047

  5. Giunta, G., Montella, R., Agrillo, G., Coviello, G.: A GPGPU transparent virtualization component for high performance computing clouds. In: D’Ambra, P., Guarracino, M., Talia, D. (eds.) Euro-Par 2010, Part I. LNCS, vol. 6271, pp. 379–391. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15277-1_37

    Chapter  Google Scholar 

  6. Gupta, V., Gavrilovska, A., et al.: GViM: GPU-accelerated virtual machines. In: ACM Workshop on System-level Virtualization for High Performance Computing, pp. 17–24. ACM (2009), https://doi.org/10.1145/1519138.1519141

  7. Hargrove, P.H., Duell, J.C.: Berkeley lab checkpoint/restart (BLCR) for linux clusters. J. Phys. Conf. Ser. 46, 494–499 (2006). https://doi.org/10.1088/1742-6596/46/1/067

    Article  Google Scholar 

  8. Kannan, S., Farooqui, N., et al.: HeteroCheckpoint: efficient checkpointing for accelerator-based systems. In: 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, pp. 738–743 (2014). https://doi.org/10.1109/DSN.2014.76

  9. Karlin, I., et al.: LULESH Programming Model and Performance Ports Overview. Technical report. LLNL-TR-608824, December 2012. https://computing.llnl.gov/projects/co-design/lulesh_ports1.pdf

  10. Lagar-Cavilla, H.A., et al.: VMM-independent graphics acceleration. In: Proceedings of the 3rd International Conference on Virtual Execution Environments, pp. 33–43. ACM (2007). https://doi.org/10.1145/1254810.1254816

  11. Nukada, A., Takizawa, H., et al.: NVCR: a transparent checkpoint-restart library for NVIDIA CUDA. In: 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and PhD Forum, pp. 104–113 (2011). https://doi.org/10.1109/IPDPS.2011.131

  12. NVIDIA Corporation: NVIDIA CUDA Compute Unified Device Architecture Programming Guide version 10.1.105 (2019). https://bit.ly/2EcQ4hN

  13. Oikawa, M., Kawai, A., et al.: DS-CUDA: a middleware to use many GPUs in the cloud environment. In: 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, pp. 1207–1214 (2012). https://doi.org/10.1109/SC.Companion.2012.146

  14. Peña, A.J., Bland, W., et al.: VOCL-FT: introducing techniques for efficient soft error coprocessor recovery. In: SC 2015: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1–12 (2015). https://doi.org/10.1145/2807591.2807640

  15. Pourghassemi, B., Chandramowlishwaran, A.: cudaCR: an In-Kernel application-level checkpoint/restart scheme for CUDA-enabled GPUs. In: CLUSTER 2017, pp. 725–732 (2017). https://doi.org/10.1109/CLUSTER.2017.100

  16. Shi, L., Chen, H., Sun, J., et al.: vCUDA: GPU-accelerated high-performance computing in virtual machines. IEEE Trans. Comput. 61(6), 804–816 (2012). https://doi.org/10.1109/TC.2011.112

    Article  MathSciNet  MATH  Google Scholar 

  17. Suzuki, T., Akira Nukada, S.M.: Transparent Checkpoint and Restart Technology for CUDA applications (2016). https://bit.ly/2DzHGbO. Accessed 25 April 2019

  18. Takizawa, H., Sato, K., et al.: CheCUDA: a checkpoint/restart tool for CUDA applications. In: 2009 International Conference on Parallel and Distributed Computing, Applications and Technologies, pp. 408–413 (2009). https://doi.org/10.1109/PDCAT.2009.78

  19. Takizawa, H., et al.: CheCL: transparent checkpointing and process migration of OpenCL applications. In: 2011 IEEE International, IPDPS. IEEE (2011). https://doi.org/10.1109/IPDPS.2011.85

  20. Xu, X., et al.: HiAL-Ckpt: a hierarchical application-level checkpointing for CPU-GPU hybrid systems, pp. 1895–1899 (2010). https://doi.org/10.1109/ICCSE.2010.5593819

Download references

Acknowledgement

This work was supported in part by grants EP/N028201/1 and EP/L00058X/1 from the Engineering and Physical Sciences Research Council (EPSRC) and partially sponsored by the European Union’s Horizon 2020 Programme under the LEGaTO Project (www.legato-project.eu), grant agreement 780681.

Author information

Authors and Affiliations

  1. Heriot-Watt University, Edinburgh, UK

    Max Baird, Sven-Bodo Scholz & Artjoms Šinkarovs

  2. Barcelona Supercomputing Center, Barcelona, Spain

    Leonardo Bautista-Gomez

Authors
  1. Max Baird
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sven-Bodo Scholz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Artjoms Šinkarovs
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Leonardo Bautista-Gomez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Max Baird .

Editor information

Editors and Affiliations

  1. Gesellschaft für Wissenschaftliche Datenverarbeitung mbH, Göttingen, Germany

    Ulrich Schwardmann

  2. Gesellschaft für Wissenschaftliche Datenverarbeitung mbH, Göttingen, Germany

    Christian Boehme

  3. CiTIUS, Santiago de Compostela, Spain

    Dora B. Heras

  4. University of Rome "Tor Vergata", Rome, Italy

    Valeria Cardellini

  5. Inria Bordeaux Sud-Ouest, Talence, France

    Emmanuel Jeannot

  6. Engineering Sardegna, Cagliari, Italy

    Antonio Salis

  7. University of Turin, Torino, Italy

    Claudio Schifanella

  8. University College Dublin, Dublin, Ireland

    Ravi Reddy Manumachu

  9. DLR-AS, Göttingen, Germany

    Dieter Schwamborn

  10. University of Pisa, Pisa, Italy

    Laura Ricci

  11. Ajou University, Suwon, Korea (Republic of)

    Oh Sangyoon

  12. RRZE Friedrich-Alexander-Universität, Erlangen, Germany

    Thomas Gruber

  13. ICAR-CNR, Napoli, Italy

    Laura Antonelli

  14. Tennessee Technological University, Cookeville, TN, USA

    Stephen L. Scott

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Baird, M., Scholz, SB., Šinkarovs, A., Bautista-Gomez, L. (2020). Checkpointing Kernel Executions of MPI+CUDA Applications. In: Schwardmann, U., et al. Euro-Par 2019: Parallel Processing Workshops. Euro-Par 2019. Lecture Notes in Computer Science(), vol 11997. Springer, Cham. https://doi.org/10.1007/978-3-030-48340-1_53

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-48340-1_53

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-48339-5

  • Online ISBN: 978-3-030-48340-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation