A schedule cache for data parallel unstructured computations

Abstract

High Performance Fortran (HPF) is the de facto standard language for writing data parallel programs. This study describes a dynamic approach for optimizing unstructured communication in codes with indirect addressing. The basic idea is that the run-time data reflecting the communication patterns will be reused if possible. The user has only to specify which data in the program have to be traced for modifications. The experiments and results show the effectiveness of the chosen approach.

Introduction

Data parallel languages, and especially High Performance Fortran (HPF) [11], [12] have been designed to make possible efficient and high-level parallel programming for distributed memory machines, in contrast to the low-level concurrent programming model based on explicit message-passing primitives. For distributed memory architectures, HPF compilers emulate the global address space by distributing the arrays among the processors according to the mapping directives of the user and by generating automatically explicit interprocessor communication.

Data parallel languages are challenged by irregular applications. These applications use indirect addressing on distributed arrays, i.e., gather and scatter operations. The dynamic nature of such computations does not provide much information to compute at compile-time, because the addressing scheme depends on the indirection array. For irregular parallel assignments, complex data structures, called schedules, are required for accessing remote items of distributed arrays and for communication optimization. The structures can be worked out only at run-time. The code design which first builds the schedule, then uses it to carry out the actual communication and computation, has been coined as the inspector/executor scheme [19], and is widely used by research [21], [13], [22], [6] and commercial [18] compilers.

The pre-processing associated with the inspector creates a large run-time overhead, because it involves a large amount of computation and all-to-all communications. Hence, the inspector/executor scheme targets applications which exhibit spatial or temporal locality. Spatial locality eliminates communication for non-local data whose actual value is already available or that will be accessed more than once. This study focuses on temporal locality. Temporal locality makes it possible to reuse a previously computed schedule when the communication pattern has not changed, even if the data have been modified. In this case, the inspector cost is amortized over multiple schedule reuses.

This study presents a new protocol to handle temporal locality. It has been implemented and evaluated in the framework of the automatic data parallelism translator (ADAPTOR) HPF compilation system [1]. The protocol is called URB, because the run-time system can either Use, Refresh or Build a schedule for each parallel irregular assignment. At the language level, the protocol implementation is limited to a directive, namely TRACE. This directive provides an attribute to the indirection arrays, which is coherent with the semantics of other HPF array-related directives (ALIGN, DISTRIBUTE). The main properties of URB are:

• a schedule can be reused for all parallel assignments that share the same structure parameters and the same values of indirection arrays;

• the schedule information is completely handled by the compiler and the run-time system;

• reuse through procedure calls does not require inter-procedural analysis;

• codes using the trace directive and other codes (e.g. library routines) can securely be mixed, of course losing the performance benefits of schedule reuse for non-TRACE-aware routines.

The main result of the paper is that HPF (and more generally, data-parallel languages) can efficiently handle data-parallel irregular applications, as shown in the performance results section. Most of the work targeting irregular applications has chosen another path, namely parallel libraries based on task parallelism, possibly coordinating data-parallel regular programs, such as in KeLP [16]. Both approaches require hard work for data localization, which is outside the scope of this study. However, when this work is done, a very modest effort of adding an intuitive directive suffices to get good and scalable performance inside the HPF framework.

The rest of the paper is organized as follows: Section 2 describes the realization of indirect addressing of distributed arrays via the inspector/executor scheme, and the opportunities to reuse schedules. Section 3 describes the protocol that is needed for tracking indirection arrays and how it can be realized. Related work is discussed in Section 4. Section 5 describes the implementation of our ideas in the HPF compilation system ADAPTOR. Section 6 presents performance results, and we conclude in Section 7.