Skip to main content
Springer Nature Link
Log in

CI/CD Efforts for Validation, Verification and Benchmarking OpenMP Implementations

  • Conference paper
  • First Online:
Advancing OpenMP for Future Accelerators (IWOMP 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15195))

Included in the following conference series:

  • 286 Accesses

Abstract

Software developers must adapt to keep up with the changing capabilities of platforms so that they can utilize the power of High-Performance Computers (HPC), including exascale systems. OpenMP, a directive-based parallel programming model, allows developers to include directives to existing C, C++, or Fortran code to allow node level parallelism without compromising performance. This paper describes our CI/CD efforts to provide easy evaluation of the support of OpenMP across different compilers using existing testsuites and benchmark suites on HPC platforms. Our main contributions include (1) the set of a Continuous Integration (CI) and Continuous Development (CD) workflow that captures bugs and provides faster feedback to compiler developers, (2) an evaluation of OpenMP (offloading) implementations supported by AMD, HPE, GNU, LLVM, and Intel, and (3) evaluation of the quality of compilers across different heterogeneous HPC platforms. With the comprehensive testing through the CI/CD workflow, we aim to provide a comprehensive understanding of the current state of OpenMP (offloading) support in different compilers and heterogeneous platforms consisting of CPUs and GPUs from NVIDIA, AMD, and Intel.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://lab.llvm.org/buildbot/.

  2. 2.

    https://github.com/llvm/llvm-test-suite/tree/main/External.

  3. 3.

    https://crpl.cis.udel.edu/lnt-sollve/.

  4. 4.

    zenodo.org/doi/10.5281/zenodo.12571032: allCompilerSystemsResults.json.

  5. 5.

    zenodo.org/doi/10.5281/zenodo.12571032: compilerVersionsPerlmutterFrontierResults.json.

  6. 6.

    zenodo.org/doi/10.5281/zenodo.12571032: Frontier_SPEC_PaperResults.zip.

  7. 7.

    zenodo.org/doi/10.5281/zenodo.12571032: Perlmutter_SPEC_PaperResults.zip.

  8. 8.

    zenodo.org/doi/10.5281/zenodo.12571032: SqPatch.hpp.diff.

References

  1. AMD: AMD ROCm Smoke Tests. https://github.com/ROCm/aomp/tree/aomp-dev/test/smoke

  2. Anderson, J., Burns, P., Milroy, D., Ruprecht, P., Hauser, T., Siegel, H.: Deploying RMACC summit: an HPC resource for the rocky mountain region, pp. 1–7, July 2017. https://doi.org/10.1145/3093338.3093379

  3. Argonne National Laboratory: Argonne’s aurora supercomputer breaks exascale barrier, May 2024. https://www.anl.gov/article/argonnes-aurora-supercomputer-breaks-exascale-barrier

  4. Bak, S., et al.: Openmp application experiences: porting to accelerated nodes. Parallel Comput. 109, 102856 (2022)

    Article  Google Scholar 

  5. Bauman, P., et al.: Introduction to AMD GPU programming with hip. Presentation at Oak Ridge National Laboratory (2019). https://www.olcf.ornl.gov/calendar/intro-to-amd-gpu-programming-with-hip

  6. Brunst, H.,et al.: First experiences in performance benchmarking with the new SPEChpc 2021 suites. In: 2022 22nd International Symposium on Cluster, Cloud and Internet Computing (CCGrid), pp. 675–684. IEEE Computer Society, Los Alamitos, CA, USA, May 2022. https://doi.org/10.1109/CCGrid54584.2022.00077, https://doi.ieeecomputersociety.org/10.1109/CCGrid54584.2022.00077

  7. Daley, C., Ahmed, H., Williams, S., Wright, N.: A case study of porting HPGMG from CUDA to OpenMP target offload. In: Milfeld, K., de Supinski, B.R., Koesterke, L., Klinkenberg, J. (eds.) IWOMP 2020. LNCS, vol. 12295, pp. 37–51. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58144-2_3

    Chapter  Google Scholar 

  8. Denny, J.E., Lee, S., Vetter, J.S.: CLACC: translating OpenACC to OpenMP in clang. In: 2018 IEEE/ACM 5th Workshop on the LLVM Compiler Infrastructure in HPC (LLVM-HPC), pp. 18–29. IEEE (2018)

    Google Scholar 

  9. Huber, J., et al.: Efficient execution of OpenMP on GPUs. In: 2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), pp. 41–52. IEEE (2022)

    Google Scholar 

  10. Huber, T., et al.: ECP SOLLVE: validation and verification testsuite status update and compiler insight for openMP. In: 2022 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), pp. 123–135 (2022). https://doi.org/10.1109/P3HPC56579.2022.00017

  11. Jin, Z., Vetter, J.S.: A benchmark suite for improving performance portability of the SYCL programming model. In: 2023 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 325–327 (2023). https://doi.org/10.1109/ISPASS57527.2023.00041

  12. John, M.M., Olsson, H.H., Bosch, J.: Towards MLOps: a framework and maturity model. In: 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 1–8 (2021). https://doi.org/10.1109/SEAA53835.2021.00050

  13. Karlin, I., et al.: Early experiences porting three applications to OpenMP 4.5. In: Maruyama, N., de Supinski, B.R., Wahib, M. (eds.) IWOMP 2016. LNCS, vol. 9903, pp. 281–292. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45550-1_20

    Chapter  Google Scholar 

  14. Kim, J., et al.: QMCPACK: an open source ab initio quantum Monte Carlo package for the electronic structure of atoms, molecules and solids. J. Phys.: Condens. Matter 30(19), 195901 (2018)

    MathSciNet  Google Scholar 

  15. Lambert, J., Monil, M.A.H., Lee, S., Malony, A.D., Vetter, J.S.: Leveraging compiler-based translation to evaluate a diversity of exascale platforms. In: 2022 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), pp. 14–25 (2022). https://doi.org/10.1109/P3HPC56579.2022.00007

  16. LLVM Compiler Infrastructure: lit - LLVM Integrated Tester. https://llvm.org/docs/CommandGuide/lit.html

  17. LLVM Compiler Infrastructure: LLVM Test Suite. https://github.com/llvm/llvm-test-suite

  18. LLVM Compiler Infrastructure: LNT infrastructure for performance testing. https://llvm.org/docs/lnt/tests.html

  19. Munley, C., Jarmusch, A., Chandrasekaran, S.: LLM4VV: developing LLM-Driven testsuite for compiler validation. Futur. Gener. Comput. Syst. 160, 1–13 (2024)

    Google Scholar 

  20. NVIDIA: CUDA. https://developer.nvidia.com/cuda-toolkit

  21. Oak Ridge National Laboratory: Frontier supercomputer debuts as world’s fastest, breaking exascale barrier, May 2022. https://www.ornl.gov/news/frontier-supercomputer-debuts-worlds-fastest-breaking-exascale-barrier

  22. OpenACC Organization: OpenACC. https://www.openacc.org/

  23. OpenMP Architecture Review Board: OpenMP application program interface version 4.0 (2013). https://www.openmp.org/wp-content/uploads/OpenMP4.0.0.pdf

  24. OpenMP Architecture Review Board: OpenMP application program interface version 5.2 (2021). https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification-5-2.pdf

  25. ORNL and University of Delaware: OpenMP validation and verification suite. https://github.com/OpenMP-Validation-and-Verification/OpenMP_VV

  26. Pennycook, S.J., Sewall, J.D., Hammond, J.R.: Evaluating the impact of proposed OpenMP 5.0 features on performance, portability and productivity. In: 2018 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC), pp. 37–46. IEEE (2018)

    Google Scholar 

  27. S4PST: Sustainability for Programming Systems and Tools: S4PST. https://ornl.github.io/events/s4pst2023/

  28. Sampedro, Z., Holt, A., Hauser, T.: Continuous integration and delivery for HPC: using singularity and Jenkins. In: Proceedings of the Practice and Experience on Advanced Research Computing. PEARC ’18, Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3219104.3219147

  29. SOLLVE: Scaling OpenMP With LLVm for Exascale Performance and Portability : SOLLVE. https://www.exascaleproject.org/research-project/sollve/

  30. Standard Performance Evaluation Corporation: SPEChpc™ 2021 benchmark suites. https://www.spec.org/hpc2021/

  31. The Khronos Group: OpenCL. https://www.khronos.org/opencl/

  32. Varrette, S., Bouvry, P., Cartiaux, H., Georgatos, F.: Management of an academic HPC cluster: the UL experience, July 2014. https://doi.org/10.1109/HPCSim.2014.6903792

  33. Yu, L., Alégroth, E., Chatzipetrou, P., Gorschek, T.: A roadmap for using continuous integration environments. Commun. ACM 67(6), 82–90 (2024). https://doi.org/10.1145/3631519

Download references

Acknowledgments

This research used resources of the OLCF at ORNL supported by the Office of Science of the U.S. DOE under Contract No. DE-AC05-00OR22725; used resources of the ALCF, a U.S. DOE Office of Science user facility at Argonne National Laboratory and is based on research supported by the U.S. DOE Office of Science-ASCR program, under Contract No. DE-AC02-06CH11357; used resources of NERSC, a U.S. DOE Office of Science User Facility located at LBNL, operated under Contract No. DE-AC02-05CH11231 using NERSC ERCAP0029463. This material is also based upon work supported by the U.S. DOE under Contract DE-FOA-0003177, S4PST: Next Generation Science Software Technologies Project.

Author information

Authors and Affiliations

  1. University of Delaware, Newark, DE, USA

    Aaron Jarmusch, Felipe Cabarcas, Andrew Kallai & Sunita Chandrasekaran

  2. Lawrence Livermore National Laboratory, Livermore, CA, USA

    Johannes Doerfert

  3. Universidad de Antioquia, Medellin, Colombia

    Felipe Cabarcas

  4. University of Oregon, Eugene, OR, USA

    Luke Peyralans

  5. Oak Ridge National Laboratory, Bethel Valley Road Oak Ridge, Oak Ridge, TN, USA

    Swaroop Pophale, Seyong Lee & Joel Denny

Authors
  1. Aaron Jarmusch
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Felipe Cabarcas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Swaroop Pophale
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andrew Kallai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Johannes Doerfert
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Luke Peyralans
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Seyong Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Joel Denny
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Sunita Chandrasekaran
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sunita Chandrasekaran .

Editor information

Editors and Affiliations

  1. Pawsey Supercomputing Centre, Kensington, WA, Australia

    Alexis Espinosa

  2. AMD and OpenMP ARB, Beaverton, OR, USA

    Michael Klemm

  3. Lawrence Livermore National Laboratory, Livermore, CA, USA

    Bronis R. de Supinski

  4. Pawsey Supercomputing Centre, Kensington, WA, Australia

    Maciej Cytowski

  5. RWTH Aachen University, Aachen, Germany

    Jannis Klinkenberg

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jarmusch, A. et al. (2024). CI/CD Efforts for Validation, Verification and Benchmarking OpenMP Implementations. In: Espinosa, A., Klemm, M., de Supinski, B.R., Cytowski, M., Klinkenberg, J. (eds) Advancing OpenMP for Future Accelerators. IWOMP 2024. Lecture Notes in Computer Science, vol 15195. Springer, Cham. https://doi.org/10.1007/978-3-031-72567-8_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72567-8_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72566-1

  • Online ISBN: 978-3-031-72567-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics