A novel MPI reduction algorithm resilient to imbalances in process arrival times

Abstract

Reduction algorithms are optimized only under the assumption that all processes commence the reduction simultaneously. Research on process arrival times has shown that this is rarely the case. Thus, all benchmarking methodologies that take into account only balanced arrival times might not portray a true picture of real-world algorithm performance. In this paper, we select a subset of four reduction algorithms frequently used by library implementations and evaluate their performance for both balanced and imbalanced process arrival times. The main contribution of this paper is a novel imbalance robust algorithm that uses pre-knowledge of process arrival times to construct reduction schedules. The performance of selected algorithms was empirically evaluated on a 128 node subset of the Partnership for Advanced Computing in Europe CURIE supercomputer. The reported results show that the new imbalance robust algorithm universally outperforms all the selected algorithms, whenever the reduction schedule is precomputed. We find that when the cost of schedule construction is included in the total runtime, the new algorithm outperforms the selected algorithms for problem sizes greater than 1 MiB.

Appendices

Appendix 1: Algorithm definitions

Appendix 2: Autocorrelograms of Helsim simulation image rendering time

See Fig. 14.

Fig. 14

Autocorrelograms of the four principal clusters of image render times across the 100 iterations of the Helsim simulation, for $P <24$. The data were gathered on the Lynx cluster with $P =128$ and eight processes per node. In the autocorrelation plots, the green line indicates a $P =0.95$ significance level, and the red line indicates a $P =0.99$ significance level

Appendix 3: Radix-k reduction schedule illustration

See Fig. 15.

Fig. 15

Example of Radix-k execution for $P=12$, factored into two rounds $R=\{4,3\}$. Reprinted from Peterka et al. [16]. Reprinted according to fair use

Appendix 4: Time series data for algorithm parallel ring

See Fig. 16.

Fig. 16

Time series data of two independent samples of algorithm parallel ring. Blue color denotes sample 1 and red sample 2. Axis x denotes iteration number, while axis y denotes the observed algorithm runtime in seconds. a Parallel ring, $I(x) = 1.75\times 10^{-1}$ ms. b Parallel ring, $I(x) = 2\times 10^{-1}$ ms. c Parallel ring, $I(x) = 2.25\times 10^{-1}$ ms. d Parallel ring, $I(x) = 2.5\times 10^{-1}$ ms

Appendix 5: Autocorrelation of time series data for parallel ring

See Fig. 17.

Fig. 17

Autocorrelation of time series data of two independent executions of algorithm parallel ring. Green line denotes the 95% confidence level, while the red denotes the 99 % confidence level. a Parallel ring, $I(x) = 2.5\times 10^{-1}$ ms. Sample 1. b Parallel ring, $I(x) = 2.5\times 10^{-1}$ ms. Sample 2