Abstract
MPI’s low-level interface is prone to errors, leading to bugs that can remain dormant for years. MPI correctness tools can aid in writing correct code but lack a standardized benchmark for comparison. This makes it difficult for users to choose the best tool and difficult for developers to gauge their tools’ effectiveness. MPI correctness benchmarks, MPI-CorrBench, the MPI Bugs Initiative, and RMARaceBench have emerged to address this problem. However, comparability is hindered by having separate benchmarks, and none fully reflects real-world MPI usage patterns. Hence, we present MPI-BugBench, a unified MPI correctness benchmark replacing previous efforts. It addresses the shortcomings of its predecessors by providing a single, standardized test harness for assessing tools and incorporates a broader range of real-world MPI usage scenarios. MPI-BugBench is available at https://git-ce.rwth-aachen.de/hpc-public/mpi-bugbench.
T. Jammer, E. Saillard and S. Schwitanski—These authors contributed equally to this research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Droste, A., Kuhn, M., Ludwig, T.: MPI-checker: static analysis for MPI. In: Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC, LLVM 2015. ACM (2015). https://doi.org/10.1145/2833157.2833159
GASPI Forum: GASPI: Global Address Space Programming Interface 17.1 (2017). https://raw.githubusercontent.com/GASPI-Forum/GASPI-Forum.github.io/master/standards/GASPI-17.1.pdf. Accessed 28 May 2024
Hilbrich, T., Schulz, M., de Supinski, B.R., Müller, M.S.: MUST: a scalable approach to runtime error detection in MPI programs. In: Tools for High Performance Computing 2009, pp. 53–66, Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-11261-4_5
Hück, A., Jammer, T., Jenke, J., Bischof, C.: Investigating the real-world applicability of MPI correctness benchmarks. In: Proceedings of the SC 2023 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023, pp. 230–233. ACM (2023). https://doi.org/10.1145/3624062.3624091
Hück, A., Jammer, T., Protze, J., Bischof, C.: Investigating the usage of MPI at argument-granularity in HPC codes. In: Proceedings of EuroMPI2023: the 30th European MPI Users’ Group Meeting, EuroMPI2023, pp. 1–10. ACM (2023). https://doi.org/10.1145/3615318.3615322
Hück, A., et al.: Compiler-aided type tracking for correctness checking of MPI applications. In: IEEE/ACM 2nd International Workshop on Software Correctness for HPC Applications (Correctness), pp. 51–58 (2018). https://doi.org/10.1109/Correctness.2018.00011
Intel: Intel Trace Analyzer and Collector (2023). https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/trace-analyzer.html. Accessed 28 May 2024
Laguna, I., Marshall, R., Mohror, K., Ruefenacht, M., Skjellum, A., Sultana, N.: A large-scale study of MPI usage in open-source HPC applications. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019. ACM (2019). https://doi.org/10.1145/3295500.3356176
Laurent, M., Saillard, E., Quinson, M.: The MPI bugs initiative: a framework for MPI verification tools evaluation. In: IEEE/ACM 5th International Workshop on Software Correctness for HPC Applications (Correctness), pp. 1–9 (2021). https://doi.org/10.1109/Correctness54621.2021.00008
Lehr, J.P., Jammer, T., Bischof, C.: MPI-CorrBench: towards an MPI correctness benchmark suite. In: Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2021, pp. 69–80. ACM (2021). https://doi.org/10.1145/3431379.3460652
Lin, P.H., Liao, C.: High-precision evaluation of both static and dynamic tools using DataRaceBench. In: 2021 IEEE/ACM 5th International Workshop on Software Correctness for HPC Applications (Correctness), pp. 1–8 (2021). https://doi.org/10.1109/Correctness54621.2021.00011
Luecke, G., et al.: The importance of run-time error detection, pp. 145–155 (2009). https://doi.org/10.1007/978-3-642-11261-4_10
Message Passing Interface Forum: MPI: A Message-Passing Interface Standard Version 4.1 (2023). https://www.mpi-forum.org/docs/mpi-4.1/mpi41-report.pdf. Accessed 28 May 2024
OpenSHMEM Committee: OpenSHMEM: Application Programming Interface Version 1.5 (2020). http://openshmem.org/site/sites/default/site_files/OpenSHMEM-1.5.pdf. Accessed 28 May 2024
Saillard, E., Carribault, P., Barthou, D.: PARCOACH: combining static and dynamic validation of MPI collective communications. Int. J. High Perform. Comput. Appl. 28(4), 425–434 (2014). https://doi.org/10.1145/2488551.2488555
Schwitanski, S., Jenke, J., Klotz, S., Müller, M.S.: RMARaceBench: a microbenchmark suite to evaluate race detection tools for RMA programs. In: Proceedings of the SC ’23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W ’23, pp. 205–214. ACM (2023). https://doi.org/10.1145/3624062.3624087
Schwitanski, S., Jenke, J., Tomski, F., Terboven, C., Müller, M.S.: On-the-fly data race detection for MPI RMA programs with MUST. In: 2022 IEEE/ACM Sixth International Workshop on Software Correctness for HPC Applications (Correctness), pp. 27–36. IEEE (2022). https://doi.org/10.1109/Correctness56720.2022.00009
Vinayagame, R., Saillard, E., Thibault, S., Nguyen, V.M., Sergent, M.: Rethinking data race detection in MPI-RMA programs. In: Proceedings of the SC 2023 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, pp. 196–204 (2023). https://doi.org/10.1145/3624062.3624086
Virouleau, P., Saillard, E., Sergent, M., Lemarinier, P.: Highlighting PARCOACH improvements on MBI. In: Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023 (2023). https://doi.org/10.1145/3624062.3624093
Acknowledgments
This work was supported by the Hessian Ministry for Higher Education, Research and the Arts through the Hessian Competence Center for High-Performance Computing, and by the Federal Ministry of Education and Research (BMBF) and the states of Hesse and North Rhine-Westphalia as part of the NHR program.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Jammer, T. et al. (2025). MPI-BugBench: A Framework for Assessing MPI Correctness Tools. In: Blaas-Schenner, C., Niethammer, C., Haas, T. (eds) Recent Advances in the Message Passing Interface. EuroMPI 2024. Lecture Notes in Computer Science, vol 15267. Springer, Cham. https://doi.org/10.1007/978-3-031-73370-3_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-73370-3_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-73369-7
Online ISBN: 978-3-031-73370-3
eBook Packages: Computer ScienceComputer Science (R0)