Abstract
Many of the largest supercomputers are based on heterogeneous architectures with multiple general-purpose graphics processing units (GPGPUs) per compute node. While many APIs for GPU programming are vendor-specific, OpenMP offers a portable alternative. Therefore OpenMP target offloading is advantageous in terms of long-term code sustainability. Further, many applications have already been parallelized with OpenMP. Hence the amount of work needed to port the code to GPUs may be limited. However, the support for the OpenMP 5.x specification is not equally mature across different compilers. Additionally, the multi-GPU support in the OpenMP 5.x specification is limited. We explore what is possible with the Nvidia NVC compiler.
We present a case study of solving the Poisson equation on multiple GPGPUs to outline which approaches for multi-target offloading give good results. We find that a task-based multi-GPU implementation leads to better performance than generating deferrable tasks with the
clause.
We demonstrate that data transfers and computations can be fully overlapped by using only the subset of the OpenMP specifications, which is supported in the 22.3 release of the Nvidia NVC compiler. For compute nodes with multiple Nvidia A100 or V100, we obtain close to ideal strong scaling when increasing the number of accelerators.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
NVIDIA Corporation: NVIDIA HPC Compilers User’s Guide, section 7. Using OpenMP. https://docs.nvidia.com/hpc-sdk/archive/22.3/compilers/hpc-compilers-user-guide/index.html#openmp-use. Accessed 12 July 2022
Strohmaier, E., Dongarra, J., Horst, S., Meuer, M.: TOP500 List - June 2022. https://top500.org/lists/top500/list/2022/06/. Accessed 7 July 2022
Oak Ridge Leadership Computing Facility: Frontier - Direction of Discovery. https://www.olcf.ornl.gov/frontier/. Accessed 7 July 2022
Oak Ridge Leadership Computing Facility: Summit - Oak Ridge National Laboratory’s 200 petaflop supercomputer. https://www.olcf.ornl.gov/olcf-resources/compute-systems/summit/. Accessed 25 May 2022
Lawrence Livermore National Laboratory: Using LC’s Sierra Systems. https://hpc.llnl.gov/documentation/tutorials/using-lc-s-sierra-systems. Accessed 25 May 2022
van der Pas, R., Stotzer, E., Terboven, C.: Using OpenMP-The Next Step, 1st edn. The MIT Press, Cambridge (2017)
Demmel, J.: Solving the Discrete Poisson Equation using Jacobi, SOR, Conjugate Gradients, and the FFT. https://people.eecs.berkeley.edu/~demmel/cs267/lecture24/lecture24.html. Accessed 27 May 2022
Burkardt, J.: Jacobi Iterative Solution of Poisson’s Equation in 1D. https://people.sc.fsu.edu/~jburkardt/presentations/jacobi_poisson_1d.pdf. Accessed 15 July 2022
NVIDIA Corporation: NVIDIA HPC Compilers User’s Guide. https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html. Accessed 11 May 2022
Cheng, L.: Finite Difference Methods for Poisson Equation. https://www.math.uci.edu/~chenlong//226/FDM.pdf. Accessed 15 July 2022
Xu, R., Tian, X., Chandrasekaran, S., Chapman, B.: Multi-GPU Support on Single Node Using Directive-Based Programming Model. Sci. Program. 2015 (2015). https://doi.org/10.1155/2015/621730
Hasbestan, J.J., Xiao, C., Senocak, I.: PittPack: an open-source Poisson’s equation solver for extreme-scale computing with accelerators. Comput. Phys. Commun. 254, 107272 (2020). https://doi.org/10.1016/j.cpc.2020.107272
Kale, V., Lu, W., Curtis, A., Malik, A.M., Chapman, B., Hernandez, O.: Toward supporting multi-GPU targets via taskloop and user-defined schedules. In: Milfeld, K., de Supinski, B.R., Koesterke, L., Klinkenberg, J. (eds.) IWOMP 2020. LNCS, vol. 12295, pp. 295–309. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58144-2_19
Patel, A., Doerfert, J.: Remote OpenMP offloading. In: Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP, pp. 441–442. Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3503221.3508416
Chapman, B., et al.: Outcomes of OpenMP hackathon: OpenMP application experiences with the offloading model (Part I). In: McIntosh-Smith, S., de Supinski, B.R., Klinkenberg, J. (eds.) IWOMP 2021. LNCS, vol. 12870, pp. 67–80. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85262-7_5
Rydahl, A.: Why does ‘default(none)’ not always take effect in OpenMP? https://forums.developer.nvidia.com/t/why-does-default-none-not-always-take-effect-in-openmp/213247. Accessed 26 May 2022
Rydahl, A.: OpenMP Target Offloading Bug - Making Target Region in Task. https://forums.developer.nvidia.com/t/openmp-target-offloading-bug-making-target-region-in-task/216308. Accessed 12 July 2022
Acknowledgments
This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 951732. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and Germany, Bulgaria, Austria, Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, Greece, Hungary, Ireland, Italy, Lithuania, Latvia, Poland, Portugal, Romania, Slovenia, Spain, Sweden, United Kingdom, France, Netherlands, Belgium, Luxembourg, Slovakia, Norway, Switzerland, Turkey, Republic of North Macedonia, Iceland, Montenegro.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Rydahl, A., Gammelmark, M., Karlsson, S. (2022). Feasibility Studies in Multi-GPU Target Offloading. In: Klemm, M., de Supinski, B.R., Klinkenberg, J., Neth, B. (eds) OpenMP in a Modern World: From Multi-device Support to Meta Programming. IWOMP 2022. Lecture Notes in Computer Science, vol 13527. Springer, Cham. https://doi.org/10.1007/978-3-031-15922-0_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-15922-0_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15921-3
Online ISBN: 978-3-031-15922-0
eBook Packages: Computer ScienceComputer Science (R0)