Communication Optimization in Heterogeneous Edge Networks Using Dynamic Grouping and Gradient Coding

Abstract

Communication load in heterogeneous edge networks is becoming heavier because of excessive computation and delay caused by straggler dropout, leading to high electricity cost and serious greenhouse gas emissions. To create a green edge environment, we focus on mitigating computation and straggler dropout to improve the communication efficiency during the distributed training. Therefore, we propose a novel scheme named Dynamic Grouping and Heterogeneity-aware Gradient Coding (DGHGC) to speed up average iteration time. The average iteration time is used as a metric reflecting the effect of mitigating computation and straggler dropout. Specifically, DGHGC firstly uses the static grouping to evenly distribute stragglers in each group. After the static grouping, considering the nonuniform distribution of nodes due to straggler dropout during the training process, a dynamic grouping depending on dropout frequency of stragglers is employed. The dynamic grouping tolerates more stragglers by examining the dropout threshold to improve the rationality of the static grouping for stragglers. In addition, DGHGC applies a heterogeneity-aware gradient coding to allocate reasonable data to stragglers based on their computing capacity and encode gradients to prevent stragglers from dropping out. Numerical results demonstrate that the average iteration time of DGHGC can be reduced largely compared to the state-of-art benchmark schemes.