A Shim Layer for Transparently Adding Meta Data to MPI Handles

MPI tool or abstraction libraries often have the need to bind meta data information to different kinds of MPI opaque handles. Communicator, window and datatype handles allow to associate key-value pairs as a mean to store meta information. The more short-living request handles, however, do not provide such functionality. As a result, several tool libraries use map-like data structures to bind internal meta information to MPI opaque handles and track those handles across their lifetime, using the handle value as a key. This results in several challenges. In this paper we make the case that request handles associated with different concurrent communication operations are not guaranteed to be unique when returned from the MPI library, so that a simple map may result in conflicts. Furthermore, MPI handles are not guaranteed to be constant over their lifetime, making the use of a map even more questionable. In this work, we present a shim layer wrapping MPI opaque handles that is transparent to the application and that allows tool and abstraction libraries to uniquely distinguish semantically different handles. At the same time, the handle shim layer allows to store the meta information in the wrapped handle avoiding the map-like data structures and also sand-boxes the MPI handle so that changes over its lifetime do not cause any harm. We provide a thread-safe proof-of-concept implementation for most relevant MPI-4 functions that can be used with multi-threaded MPI applications. The implementation transparently supports the different underlying base-language types for handles chosen by the MPI implementations. We evaluate the integration into several tools and abstraction libraries. In an overhead evaluation on synthetic benchmarks, our handle shim layer reduces the runtime overhead compared to a map-like data structure by about in most single-threaded cases.