Stragglers in Distributed Matrix Multiplication

Abstract

A delay in a single processor may affect an entire system since the slowest processor typically determines the runtime. Problems with such stragglers are often mitigated using dynamic load balancing or redundancy solutions such as task replication. Unfortunately, the former option incurs high communication cost, and the latter significantly increases the arithmetic cost and memory footprint, making high resource overhead seem inevitable. Matrix multiplication and other numerical linear algebra kernels typically have structures that allow better straggler management. Redundancy based solutions tailored for such algorithms often combine codes in the algorithm’s structure. These solutions add fixed cost overhead and may perform worse than the original algorithm when little or no delays occur. We propose a new load-balancing solution tailored for distributed matrix multiplication. Our solution reduces latency overhead by $O \left( P/\log {P} \right)$ compared to existing dynamic load-balancing solutions, where P is the number of processors. Our solution overtakes redundancy-based solutions in all parameters: arithmetic cost, bandwidth cost, latency cost, memory footprint, and the number of stragglers it can tolerate. Moreover, our overhead costs depend on the severity of delays and are negligible when delays are minor. We compare our solution with previous ones and demonstrate significant improvements in asymptotic analysis and simulations: up to x4.4 and x5.3 compared to general-purpose dynamic load balancing and redundancy-based solutions, respectively.

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 818252). This work was supported by the Federmann Cyber Security Center in conjunction with the Israel national cyber directorate. This research was supported by a grant from the United States-Israel Binational Science Foundation (BSF), Jerusalem, Israel.

A Existing Solutions

In this section, we provide an analysis of existing straggler mitigation solutions.

1.1 A.1 Dynamic Load Balancing

We review receiver-initiated load-balancing algorithms (also called work-stealing), which often perform better in decentralized models (such as ours). These solutions often share the following structure: when a target processor receives a work request, it acts in one of the following ways: if it has more than s tasks, it transfers a fraction $\delta$ of the work to the processor that requested the work. If it has less than s tasks, it rejects the request, which is then passed on to the next candidate. Choosing targets at random incurs the optimal asymptotic costs (see [59] for further details).

Claim

Let $\gamma _{P} \cdot F,\ BW,\ L$, and M denote an algorithm’s arithmetic cost, bandwidth cost, latency cost, and memory footprint, respectively. Let ${F}^{'},\ {BW}^{'},\ {L}^{'}$, and $M^{'}$ denote the costs of the algorithm applying the work-stealing solution on C. Then ${F}^{'} = \bar{\gamma }_{a} \cdot (1+\frac{P}{S}) \cdot F$, ${BW}^{'} = BW + M$, ${L}^{'} = L + O \left( P\log {P} \right)$, and $M^{'} = M$.

Proof

We follow the proof of Theorem 1 with slight modifications. The main difference between SLB and work-stealing is in the number of performed task requests, which affects the latency cost. According to Theorem 2 in [59], the work-stealing technique is expected to perform $O \left( P\log {S} \right)$ work requests. Similarly to SLB, we bound S by a polynomial function with P (by grouping tasks together) so that $\log {S} = O \left( \log {P} \right)$. In total, ${F}^{'} = \bar{\gamma }_{a} \cdot (1+\frac{P}{S}) \cdot F$, ${BW}^{'} = BW + O \left( M \right)$, ${L}^{'} = L + O \left( P\log {P} \right)$, and $M^{'} = M$.

1.2 A.2 Redundancy

Here, we compare our solution to three commonly used erasure code based solutions: replication, MDS, and LT.

Replication. In the Replication solution, the algorithm divides the processors into P/r groups of r processors, where processors within the same group perform the exact same computations. The algorithm constructs the final output using the results of the fastest processor in each group.

Claim

Let $\gamma _{P} \cdot F,\ BW,\ L$, and M denote an algorithm’s arithmetic cost, bandwidth cost, latency cost, and memory footprint, respectively. Let ${F}^{'},\ {BW}^{'},\ {L}^{'}$, and $M^{'}$ denote the costs of the algorithm applying the replication solution on C. Then ${F}^{'} = \gamma _{P-r+1} \cdot r \cdot F$, ${BW}^{'} = BW + 2r\cdot M$, ${L}^{'} = L + O \left( \log {r} \right)$, and $M^{'} = r\cdot M$.

Proof

Since the workload is shared between P/r processor (rather than P), each processor computes a factor of r more computations and uses a factor of r more memory. The algorithm uses an all-broadcast operator to share the input, and a scatter operator to share the output. This costs $2\log {r}$ messages and $2r \cdot M$ words. The algorithm halts when the first processor from each set completes its tasks. In the worst-case scenario, the algorithm halts when $P-r+1$ processors have completed their tasks. Hence, the arithmetic cost is $\gamma _{P-r+1} \cdot r \cdot F$. Summing up, we obtain ${F}^{'} = \gamma _{P-r+1} \cdot r \cdot F$, ${BW}^{'} = BW + 2r\cdot M$, ${L}^{'} = L + O \left( \log {r} \right)$, and $M^{'} = r \cdot M$.

Erasure Codes. An erasure code (cf. [48, 53]) is a linear transformation T that takes a vector v of size x1, and outputs a vector w of size x2, where x1, x2, and $\rho = \frac{x2}{x1}$ are called the rank, length, and rate of the code, respectively. We represent a code T by a generator matrix G of size $x2 \times x1$, where applying the code is equivalent to multiplying the input vector from the left with the matrix G. The generator matrix of a replication code is an identity matrix where each row is duplicated r times. We say the code has a distance d if any two code vectors differ in at least d coordinates. A code with distance d can recover from $d-1$ erasures. Maximal Distance Separable codes (MDS) are a family of codes with maximal distance.

Random codes combine randomness in the construction of the generator matrix. Luby-Transforms codes [40] (LT) are a family of random codes that obey the following conditions: I) The values of its generator matrix are either one or zero. II) The density of each row (number of non-zeroes elements) is sampled randomly from some distribution. III) The locations of the non-zeroes are sampled uniformly. A popular choice for the density distribution is the Robust Soliton degree distribution [40].

MDS. Given an MDS code with rank K and length P, the algorithm partitions the problem into K tasks and constructs P new tasks in place of the original ones. The $i'th$ processor computes the $i'th$ task, and the final output is produced from the outcomes of the first K processors to finish.

Claim

Let $\gamma _{P} \cdot F,\ BW,\ L$, and M denote an algorithm’s arithmetic cost, bandwidth cost, latency cost, and memory footprint, respectively. Let ${F}^{'},\ {BW}^{'},\ {L}^{'}$, and $M^{'}$ denote the costs of the algorithm applying the MDS solution² on C. Then ${F}^{'} = \gamma _{K} \cdot O \left( \frac{P}{K} \cdot F \right)$, ${BW}^{'} = BW + 2P\cdot M$, ${L}^{'} = L + O \left( P \right)$, and $M^{'} = \frac{P}{K} \cdot M$.

Proof

Input (resp. output) redistribution involves code encoding (resp. decoding) and an all-reduce operation. By Corollary 1, this adds $2P \cdot M$ bandwidth cost, $O \left( P \right)$ latency cost, and an arithmetic cost, which is often negligible. Moreover, each processor performs a factor of $\frac{P}{K}$ more arithmetic computations (since the workload is distributed among K processors instead of among P) and uses a factor of $\frac{P}{K}$ more memory. The algorithm halts when K processors have completed their tasks. Thus, the arithmetic cost is $\gamma _{K} \cdot O \left( \frac{P}{K} \cdot F \right)$.

LT. Mallick et al. [41] proposed a new variation of the LT coding solution (denoted here as LT+) that utilizes partial computations performed by all processors and attains near-ideal load balancing. In the LT+ solution, each processor broadcasts an update message every time it completes a task. The algorithm generates $\rho \cdot S$ new tasks (in place of the originals) using the LT code and halts when enough tasks have been completed.

Remark 1

Using the Robust Soliton degree distribution [40], the encoding and decoding complexity of a code of size x with probability $1-\epsilon$ is $O(x\log {\frac{x}{\epsilon }})$. Any $x + \sqrt{x}\cdot \log ^{2}{\frac{x}{\epsilon }}$ symbols are sufficient to construct the output.

Claim

Let $\gamma _{P} \cdot F,\ BW,\ L$, and M denote an algorithm’s arithmetic cost, bandwidth cost, latency cost, and memory footprint, respectively, and let ${F}^{'},\ {BW}^{'},\ {L}^{'}$, and $M^{'}$ denote the costs of the algorithm applying the LT+ solution to C. Then ${F}^{'} = \bar{\gamma }_{a} \cdot O \left( 1 + \frac{\log ^{2}{s}}{\sqrt{S}} \right) \cdot F$, ${BW}^{'} = BW + 2P \cdot M$, ${L}^{'} = L + O \left( S\cdot \log {P} \right)$, and $M^{'} = \rho \cdot M$.

Proof

Input and output distribution phases are similar to the MDS solution but with different weights. Since both use an all-reduce operation, the overhead costs remain the same. The algorithm constructs the final output when processors have computed a sufficient amount of tasks globally (maintaining a near-perfect³ load balancing. The number of completed tasks expected to be sufficient for recovery is $O \left( S + \sqrt{S}\log ^{2}{S} \right)$ (Claim 1), which means that each processor is expected to perform a factor of $O \left( 1 + \frac{\log ^{2}{S}}{\sqrt{S}} \right)$ more computations compared to the non-mitigated algorithm. Notifying the processors when each task is completed adds $S \cdot P$ bandwidth and $O \left( S \cdot \log {P} \right)$ latency (Corollary 1). The algorithm generates a factor of $\rho$ more tasks and thus uses a factor of $\rho$ more memory. Summing up, the additional costs are: $\left( {F}^{'}, {BW}^{'}, {L}^{'} \right) = \left( \bar{\gamma }_{a} \cdot O \left( 1 + \frac{\log ^{2}{S}}{\sqrt{S}} \right) \cdot F,\ BW + 2P\cdot M,\ L + O \left( S\log {P} \right) \right)$.