A two-phase half-async method for heterogeneity-aware federated learning

Abstract

Federated learning (FL) is a distributed machine learning paradigm that allows training models on decentralized data over large-scale edge/mobile devices without collecting raw data. However, existing methods are still far from efficient and stable under extreme statistical and environmental heterogeneity. In this work, we propose FedHA (Federated Heterogeineity Awareness), a novel half-async algorithm which simultaneously incorporates the merits of asynchronous and synchronous methods. It separates the training into two phases by estimating the consistency of optimization directions of collected local models. It applies different strategies to facilitate fast and stable training, namely model selection, adaptive local epoch, and heterogeneity weighted aggregation in these phases. We provide theoretical convergence and communication guarantees on both convex and non-convex problems without introducing extra assumptions. In the first phase (the consistent phase), the convergence rate of FedHA is $\mathcal{O}\left(\frac{1}{e^T}\right)$, which is faster than existing methods while reducing communication. In the second phase (inconsistent phase), FedHA retains the best-known results in convergence ($\mathcal{O}\left(\frac{1}{T}\right)$) and communication ($\mathcal{O}\left(\frac{1}{∊}\right)$). We validate our proposed algorithm on different tasks with both IID (Independently and Identically Distributed) and non-IID data, and results show that our algorithm is efficient, stable, and flexible under the twofold heterogeneity using the proposed strategies.

Introduction

With the rise of the Internet of Things (IoT), and the fifth-generation mobile network (5G), massive edge/mobile devices must process ever-more data. It introduces additional requirements and challenges for future use cases: 1) communication latency will be in 1 mill-second range, and 2) data rates per user device will be in 1–10 Gigabits-per second range, which implies 5–10 times reduction on latency and 10–100 times increase on user data rate, respectively.¹ In this case, as the data volume, data rates, and the number of edge devices are increasing tremendously, the whole life-cycle of data mining is becoming more challenging. Such scenarios require higher communication bandwidth and computing capability, which is costly in both time and energy-consuming. Thus, training strategies for deep learning models shift from centralized to distributed and decentralized paradigms in recent years, e.g., the distributed gradient descent based methods and the prevalent parameter server [39], [6], [7], [16], [11], [21].

However, distributed learning approaches always suffer from communication overhead and privacy risk [24]. Federated Learning (FL) [9] was proposed to alleviate the challenges mentioned above, and in the 5G era, it becomes one of the candidate paradigms for the next generation of machine learning on edge/mobile devices [22]. First, it avoids sharing sensitive data kept in local devices, thus protecting user privacy. Second, it deploys and trains models distributively without the need for dense computation in center servers. Resource-limited edge/mobile devices receive a randomly initialized model from the central server and train their local models quickly when they are idle with the local data only, which are relatively smaller in size (e.g., only dozens or hundreds of records per device). Then, the central server samples a subset of devices, next collects encrypted model parameters (e.g., gradients, weights, and biases), and aggregates collected models into a global model using specific aggregation algorithms, e.g., FedAvg [24] and its variants [31], [8], [17], [13], [26], [27]. Finally, the aggregated global model is sent to each device to replace their latest local models. These collaborative processes are shown in Fig. 1.

However, though FL was deliberately designed for statistical heterogeneity (e.g., the non-Independently and Identically Distributed data) and environmental heterogeneity (e.g., staleness and unreliable connections), most existing methods are still suffering from the performance degradation caused by them. First, data between devices are usually different and exhibit long-tail distribution. Such heterogeneity in data also introduces bias into models, e.g., the model may be biased towards commonly occurring devices or the devices with a more considerable amount of data [4], [18]. Solving statistical heterogeneity issues is not easy since FL does not allow data sharing between devices due to its fundamental requirements on privacy preservation, which makes estimation of non-IIDness difficult. Second, environmental heterogeneity causes stagnancy for some devices, which hinders the convergence of models [5]. Specifically, local models trained in these slow devices may degrade the global model’s performance since their distributions may be inconsistent with the current global model. Besides, such effect continuously accumulates in these devices since the running environment of stale devices may keep unstable during the whole life-cycle of training processes. The “slow” models have less chance to be aggregated into the global model, magnifying the degree of differences. Furthermore, synchronous federated learning methods are costly in aggregation [5], and they are potentially inefficient for large-scale devices since the server halts when collecting models, even from a much smaller subset of devices [34].

Scholars have proposed asynchronous federated learning (AFL) approaches that address these problems without waiting for slow devices [25], [20], [33]. However, asynchrony adversely magnifies the staleness effect on non-IID data because such data are usually long-tailed in distributed heterogeneous devices, where “Matthew Effect” exists—the staleness continuously accumulates in slow devices. Eventually, it adversely affects the overall performance.

This paper focuses on performance issues in FL caused by non-IID data (statistical heterogeneity) and slow devices (environmental heterogeneity). The main contributions of this paper are summarized as follows:

• We propose a half-async method named FedHA (Federated Heterogeineity Awareness). It trains models in two-phase using different strategies, namely model selection, adaptive local epoch, and heterogeneity weighted aggregation. It retains the efficiency of asynchronous methods and the communication efficiency of synchronous methods simultaneously. FedHA is the first FL method incorporating half-async model training from a two-phase perspective to the best of our knowledge.

• We provide the theoretical guarantees of FedHA for both convex and non-convex problems. Our analysis provides the first complete convergence guarantees for async/half-async FL that do not require additional assumptions to the best of our knowledge. In comparison, FedHA reaches $\mathcal{O}\left(\frac{1}{e^T}\right)$ in convergence rate and $\mathcal{O}\left(1\right)$ in communication bound in phase I, superior to the best-known methods. In phase II, FedHA gives comparable results as the best-known methods.

• We verify the efficiency of FedHA and its strategies by conducting empirical experiments. The experiments are performed with convex and non-convex models on multiple datasets. Results show that FedHA consistently outperforms state-of-the-art FL/AFL methods on the accuracy, communication efficiency, and flexibility.