SPMD IR: Unifying SPMD and Multi-value IR Showcased for Static Verification of Collectives

Abstract

To effectively utilize modern HPC clusters, inter-node communication and related single program, multiple data (SPMD) parallel programming models such as mpi are inevitable. Current tools and compilers that employ analyses of SPMD models often have the limitation of only supporting one model or implementing the necessary abstraction internally. This makes the analysis and effort for the abstraction neither reusable nor the tool extensible to other models without extensive changes to the tool itself.

This work proposes an spmd ir as part of a multi-layer program representation and accompanying compiler passes to explicitly express the results of abstraction and multi-value analysis. The spmd ir makes the executing processes of operations explicit and differentiates between static and dynamic cases. It is implemented as a prototype in the mlir llvm infrastructure and is comprised of the spmd dialect and two compiler passes, supporting mpi, shmem, and nccl, including hybrid cases.

To evaluate the proposed IR, verification of collective communication was chosen as a use case. For that, this work reimplements and extends parcoach’s static approach on the spmd ir and assesses it by an expanded micro-benchmark suite in mpi, shmem, and nccl. Achieving similar detection accuracy, the evaluation shows that the spmd ir’s level of abstraction is strong enough for parcoach’s analyses and generic enough for increased extensibility. The prototype also constitutes the first collectives verification of shmem, nccl, and their combinations (with mpi).