“Optimal” parallelism through integration of data and control parallelism: A case study in complete parallelization

Abstract

This paper presents a detailed case study of programming in a parallel programming system which targets complete and controlled parallelization of array-oriented computations. The purpose is to demonstrate how coherent integration of control and data parallelism enables both effective realization of the potential parallelism of applications and matching of the degree of parallelism in a program to the resources of the execution environment. (“Our ability to reason is constrained by the language in which we reason.”) The programming system is based on an integrated graphical, declarative representation of control parallelism and data partitioning parallelism. The computation used for the example is even-odd reduction of block tridiagonal matrices. This computation has three phases, each with a different parallel structure. We derive, implement and measure the execution of a dynamic parallel computation structure which employs differing levels of control and data parallelism in each phase of the computation to give load balanced execution across a range of number of processors. The program formulated in the integrated representation revealed parallelism not shown in the original algorithm and has a constant level of actual parallelism throughout the computation where the original algorithm had unbalanced levels of parallelism during different phases of the computation. The resulting program shows near-linear speed-up across all phases of the computation for number of processors ranging from 2 to 32.