Customized Federated Learning for accelerated edge computing with heterogeneous task targets

doi:10.1016/j.comnet.2020.107569

Computer Networks

Volume 183, 24 December 2020, 107569

https://doi.org/10.1016/j.comnet.2020.107569 Get rights and content

Abstract

As a dominant edge intelligence technique, Federated Learning (FL) can reduce the data transmission volume, shorten the communication latency and improve the collaboration efficiency among end-devices and edge servers. Existing works on FL-based edge computing only take device- and resource-heterogeneity into consideration under a fixed loss-minimization objective. As heterogeneous end-devices are usually assigned with various tasks with different target accuracies, task heterogeneity is also a significant issue and has not yet been investigated. To this end, we propose a Customized FL (CuFL) algorithm with an adaptive learning rate to tailor for heterogeneous accuracy requirements and to accelerate the local training process. We also present a fair global aggregation strategy for the edge server to minimize the variance of accuracy gaps among heterogeneous end-devices. We rigorously analyze the convergence property of the CuFL algorithm in theory. We also verify the feasibility and effectiveness of the CuFL algorithm in the vehicle classification task. Evaluation results demonstrate that our algorithm performs better in terms of the accuracy rate, training time, and fairness during aggregation than existing efforts.

Introduction

With the rapid development of smart applications (e.g., smart transportation, smart healthcare, and smart agriculture), wireless devices at the network termination are assigned with various intelligent tasks (e.g., identification, prediction, and policy optimization). As most scenarios are user-interactive and latency-sensitive, real-time response is of great importance to the user experience. In the Mobile Edge Computing (MEC) paradigm, communication latencies of end-devices can be reduced since MEC servers (i.e., edge servers) are configured with certain computing capabilities and are deployed in the vicinity of end-devices [1], [2], [3]. Considering that resource-constrained end-devices have difficulties in tackling complex computing tasks, it has become an inevitable trend to coordinate multiple end-devices to execute edge tasks intelligently [4]. In traditional edge intelligence methods, end-devices need to offload local data to the adjacent MEC server, which naturally costs transmission time and poses threats to data privacy [5].

In order to further reduce the communication latency and protect data privacy, Google initiates a collaborative machine learning framework, termed Federated Learning (FL) [6], [7]. Without exposing any raw data, distributed end-devices can independently train a specific machine learning model (e.g., support vector machine) based on their local data sets. After several iterations of local training, end-devices will upload model parameters to the MEC server, which aggregates diverse model parameters and sends the updated parameters back after global aggregation [8]. Compared with conventional edge intelligence approaches, FL makes some breakthroughs: (i) FL breaks the isolation of individual data and facilitates the joint execution of dispersed end-devices; (ii) it naturally avoids data leakage and protects privacy of end users; (iii) FL can reduce the training time and save energy due to the fact that only model parameters are transmitted instead of the whole data set [9]. Due to the incomparable advantages, FL has aroused widespread attention recently.

Existing FL methods over mobile edge networks mainly focus on the heterogeneities in terms of devices (e.g., CPU-cycle frequency) and resources (e.g., power, wireless channels) [10]. To tackle the above heterogeneity bottlenecks, a number of works have been proposed to maximize the training accuracy of the predefined model while reducing training time and energy consumption [11].

Moreover, existing FL-based edge computing methods generally assume that different end-devices share a consistent training target. Considering that heterogeneous end-devices may be assigned with various tasks and different targets (e.g., different levels of training accuracies) [12], it is necessary to investigate the individualized FL scheme to adapt to various training accuracies. Under the individualized FL framework, there is no need to cater for an extravagant accuracy. Instead, the early satisfaction of the predefined accuracy requirement can make end-devices timely quit the training process, and evidently reduce training latency and save energy.

In order to accelerate the FL process while adapting to heterogeneous tasks with different levels of accuracy requirements, we propose a Customized FL (CuFL) algorithm with an adaptive learning rate. In our algorithm, each end-device has a distinctive accuracy target. We utilize a large learning rate to quickly search for the target accuracy, and utilize a small learning rate when the current accuracy approaches the target one. Once training targets are satisfied, end-devices can quit training processes before predefined deadlines. Besides the optimization in the local training stage, we design a parameter aggregation method that minimizes the variance between current accuracies and target accuracies to maintain the fair global aggregation.

Our main contributions are summarized as follows:

•
Considering that heterogeneous devices are usually assigned with various tasks under individualized accuracy targets, we propose the CuFL algorithm to accelerate the FL process while ensuring that all end-devices can satisfy their specific task requirements. To our knowledge, CuFL is the first algorithm that can tackle the multi-dimensional heterogeneities on devices, resources and tasks simultaneously.
•
In order to accelerate the local training process, we innovatively introduce an adaptive learning rate to tailor for end-devices’ individualized training requirements. By defining a gap function between the current accuracy and the target accuracy, the local learning rate is set to a large value initially to speed up the model convergence and gradually decreases as the gap becomes smaller.
•
To further accelerate the local model training for end-devices, we propose an early termination scheme to shorten the training time by cutting down the aggregation rounds. In the early termination scheme, end-devices can quit the FL process in advance when they satisfy the accuracy requirements. As a result, the energy cost is reduced and communication resources will be abundant for the remaining devices.
•
At the side of the MEC server, we optimize the global aggregation method. In order to achieve a fair parameter aggregation at the MEC server, we introduce a fairness coefficient to minimize the variance between the current and target accuracies.
•
We rigorously analyze the convergence property of the CuFL algorithm in theory. We also verify the effectiveness of CuFL on the vehicle classification task. Evaluation results show its superiorities in terms of the accuracy rate, training time and fairness during aggregation.

The reminder of this paper is organized as follows. Section 2 reviews the existing researches about FL. Section 3 presents the system model and problem formulation. Section 4 presents the CuFL algorithm and analyzes its convergence property. In Section 5, we conduct experiments to evaluate the CuFL algorithm. Finally, Section 6 concludes this paper.

Section snippets

Related work

FL emerges as a novel technology that aggregates distributed end-devices to collaboratively learn an intelligent algorithm with the aid of a central MEC server [13]. As FL includes end-devices’ local training and the MEC server’s global aggregation, we thus review related works in these two aspects.

Preliminaries on FL

We consider a wireless edge network that consists of one MEC server and multiple end-devices. Let $K$ denote the set of end-devices, where $K = | K |$ denotes the device number. Each end-device $k (k \in K)$ owns its local data set $D_{k}$ , the size of which is denoted as $D_{k}$ . The total data size of all participating devices is $D = \sum_{k = 1}^{K} D_{k}$ .

As shown in Fig. 1, FL includes the local training stage and the global aggregation stage [8]. During the local training stage, smart devices train a specific machine learning

CuFL algorithm

Based on the aforementioned optimizations, the redesigned FL architecture is shown in Fig. 3. In the CuFL algorithm, the device-edge system aims at minimizing the completion time in executing FL. Except for model parameters, end-devices transmit extra data sizes and loss bounds to the MEC server. During the global aggregation, we replace the original FedAvg method with our fair aggregation method. The details of the CuFL algorithm are described in Section 4.1.

Performance evaluation

In this section, we conduct three sets of simulations to evaluate the algorithm performances in terms of the model accuracy, training time, and fairness during aggregation. We select the classic FedAvg algorithm [7] and the fair-aggregation q-FFL method [24] as the baselines for comparison purposes.

Conclusion

In this paper, we proposed the CuFL algorithm with an adaptive learning rate to minimize the total learning time. Considering the task heterogeneity of end-devices, CuFL allows end-devices to quit the training in advance once they have met their distinctive accuracy requirements. We also introduced a fairness coefficient in the global aggregation step to minimize the variance between the current and target accuracies. We proved the deterministic convergence property theoretically. Extensive

CRediT authorship contribution statement

Hui Jiang: Conceptualization, Methodology, Writing - original draft, Investigation. Min Liu: Conceptualization, Project administration. Bo Yang: Writing - review & editing. Qingxiang Liu: Software, Validation. Jizhong Li: Project administration. Xiaobing Guo: Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgment

This work was supported by the National Natural Science Foundation of China under Grant No. 61732017, No. 62072436, No. 61872028.

Hui Jiang received her B.S. degree from the Department of Internet of Things, Nanjing University of Posts and Telecommunications, Nanjing, China, in 2017. Since 2017, she is currently a Ph.D. candidate at the Networking Technology Research Centre, Institute of Computing Technology, Chinese Academy of Sciences. Her current research interests include federated learning, mobile edge computing and edge intelligence.

References (34)

CasadeiR. et al.
A development approach for collective opportunistic edge-of-things services
Inf. Sci.
(2019)
YangQ. et al.
Federated machine learning: Concept and applications
ACM Trans. Intell. Syst. Technol.
(2019)
ChenY. et al.
Communication-efficient federated deep learning with layerwise asynchronous model update and temporally weighted aggregation
IEEE Trans. Neural Netw. Learn. Syst.
(2019)
DuarteM.F. et al.
Vehicle classification in distributed sensor networks
J. Parallel Distrib. Comput.
(2004)
MachP. et al.
Mobile edge computing: A survey on architecture and computation offloading
IEEE Commun. Surv. Tutor.
(2017)
LeppänenT. et al.
Service modeling for opportunistic edge computing systems with feature engineering
Comput. Commun.
(2020)
ParkJ. et al.
Wireless network intelligence at the edge
Proc. IEEE
(2019)
TeerapittayanonS. et al.
Distributed deep neural networks over the cloud, the edge and end devices
KonečnỳJ. et al.
Federated optimization: Distributed machine learning for on-device intelligence
(2016)
McMahanH.B. et al.
Communication-efficient learning of deep networks from decentralized data

WangL. et al.

CMFL: Mitigating communication overhead for federated learning

NiknamS. et al.

Federated learning for wireless communications: Motivation, opportunities and challenges

(2019)

LimW.Y. et al.

Federated learning in mobile edge networks: A comprehensive survey

IEEE Commun. Surv. Tutor.

(2020)

FangB. et al.

Nestdnn: Resource-aware multi-tenant on-device deep learning for continuous mobile vision

WangX. et al.

In-edge AI: Intelligentizing mobile edge computing, caching and communication by federated learning

IEEE Netw.

(2019)

WangS. et al.

Adaptive federated learning in resource constrained edge computing systems

IEEE J. Sel. Areas Commun.

(2019)

TranN.H. et al.

Federated learning over wireless networks: Optimization model design and analysis

Cited by (18)

Federated learning in smart cities: Privacy and security survey
2023, Information Sciences
Over the last decade, smart cities (SC) have been developed worldwide. Implementing big data and the internet of things improves the monitoring and integration of different infrastructure systems in SC and thus makes our cities more efficient, livable, and sustainable. However, big data is more vulnerable to attacks and hacking, posing significant challenges to the SC's privacy and security. Moreover, more regulations and rules about protecting user data are being enforced worldwide. Compared with centralized machine learning, federated learning (FL) provides a natural method to protect users' privacy by distributing learning over decentralized data and offers artificial intelligence advantages for sensitive, heterogeneous data domains. Researchers have recently implemented FL, motivated by the improvement in privacy and security of integrated systems in SC. However, many technological problems and risks have been identified. This study provides a comprehensive review of FL's application to improve privacy and security in SC systems (i.e., transportation, healthcare, communication, etc.). The benefits, drawbacks, open research issues and future directions of the implementation of FL in SC are thoroughly discussed. The study concluded that the existing FL systems required comprehensive testing for different types of attacks, further improving the data protection and performance in SC.
An efficient federated learning framework for multi-channeled mobile edge network with layered gradient compression
2023, Computer Networks
Citation Excerpt :
Federated learning has emerged as an efficient solution to analyze and process distributed data for data-driven tasks (e.g., autonomous driving, virtual reality, image classification, etc.) in Mobile Edge Computing (MEC) [1–6].
A fundamental issue for federated learning (FL) is how to achieve efficient training performance under complex dynamic communication environments. This issue can be alleviated by the fact that modern edge devices usually can connect to the edge server via multiple communication channels (e.g., 4G, LTE, and 5G) because multi-channel communication can increase the communication bandwidth and has lower communication costs and energy consumption than a single high-speed communication channel. However, if the communication data cannot be properly allocated to multiple channels in a complex dynamic communication network, multi-channel communication will still waste resources (e.g., bandwidth, battery life, and monetary cost). In this paper, we propose an efficient FL framework called, which consists of two parts, the layered gradient compression (LGC), and a learning-driven control algorithm. Specifically, with LGC, local gradients from a device are coded into several layers, and each layer is sent to the server along a different channel. The FL server aggregates the received layers of local gradients from devices to update the global model and sends the result back to the devices. Furthermore, we prove the convergence of LGC and formally define the problem of resource-efficient with LGC. We then propose a learning-driven algorithm for each device to dynamically adjust its local computation (i.e., the number of local stochastic descent) and communication decisions (i.e., the compression level of different layers and the layer-to-channel mapping) in each iteration. Results from extensive experiments show that significantly reduces the training time and improves the resource utilization (energy consumption and money cost) while achieving a similar test accuracy compared with well-known FL baselines.
Smart-3DM: Data-driven decision making using smart edge computing in hetero-crowdsensing environment
2022, Future Generation Computer Systems
Mobile Edge Computing (MEC) has recently emerged as a promising paradigm for Mobile Crowdsensing (MCS) environments. In a given Area of Interest (AoI), the sensing process is performed based on task requirements, which usually ask for a specific quality of the sensing outcome. In this work, a two-stage Data-Driven Decision-making Mechanism using smart edge computing (Smart-3DM) is proposed. It advocates the use of smart edge to better fulfill the data-related task requirements. Depending on the type of data to be collected, the minimum quality of the data required, and the heuristics to apply for each type of crowdsensing service, the smart edge orchestrates the selection of workers in MEC. Our approach relies on (a) smart-edge deployment: where a cluster-based distributed architecture using smart edge nodes is considered. Here, two entities are defined: the main edge node (MEN) and the local edge nodes (LENs); and (b) data management offloading where a two-layer re-selection strategy that considers data type and context-awareness is adopted, to reduce data computation complexity and to increase data quality while meeting the task target. The proposed Smart-3DM is evaluated using a real-life dataset and is compared to one-stage local and global approaches. The overall results show that by using two-stage re-selection strategies, better performance with lower processing power (CPU), less Storage(RAM), and improved execution time is achieved, when compared to the benchmarks.
Federation based joint client and server side Machine Learning for 5G and beyond Wireless Channel Estimation
2022, Computer Networks
Citation Excerpt :
The article provided a fundamental framework for FL and tested their proposed algorithms on public Google+ posts, where the authors analysed the frequency of different features of these posts as key indicators. The authors in [6], delineated that use of FL in edge intelligence technique can reduce transmission volume for data and improve the communication efficiency between end devices. The authors in [7] performed signal recognition using federation learning, where their focus is on elaborating the benefits of using federation in the future wireless network system in terms of data privacy, data size, data security, and high quality.
In this paper, we propose a Machine Learning (ML) based approach to address Wireless Channel Estimation (WCE) problem for 5G and beyond wireless networks. Accurate wireless channel estimation plays a crucial role for provision of high quality of service to billions of wireless devices in current and future wireless networks. Our proposed approach uses contemporary ML technique, Federated Learning (FL), in conjunction with the Stochastic Gradient Descent (SGD) algorithm to optimise the WCE problem. Our proposed approach leverages the local user wireless channel information to locally optimise the objective at users and then uses this locally optimised information at the server to optimise the global objective function. The proposed approach is referred to as joint Federated Server Learning and Federated Client Learning (j-FSL-FCL) in the paper. We formulate the WCE problem with a novel loss function to be used for the optimisation problem. To evaluate the performance of our proposed j-FSL-FCL approach (with and without SGD), we consider a Down Link (DL) wireless channel model with Multiple-Input Multiple-Output (MIMO) setting that mimic closely to the wireless channel for 5G and beyond wireless channel models. The performance measuring parameters for j-FSL-FCL are to minimise the difference between actual and estimated wireless channel parameters (channel strength and direction). We compare the results of our proposed approach with the other techniques in the literature based on Least Squares (LS), Linear Regression (LR) and Mean Square Error (MSE). It is shown that the proposed algorithm converges to the optimal solution quickly when used with SGD compared to other existing techniques. It is also shown that the efficiency of the proposed approach for WCE problem is much higher compared to other LS and MSE based techniques. Finally, we present some interesting futuristic applications of FL in the context of 5G and beyond wireless networks.
A survey on federated learning
2021, Knowledge-Based Systems
Citation Excerpt :
The challenge of heterogeneity hinders federated learning. Some recent studies [83–86] have shown that if the heterogeneity of the devices in the system can be obtained in advance, all mobile devices can be grouped according to the heterogeneity, and a local central server can be assigned to each group. We can first aggregate a group of similarly heterogeneous device models, and then send them to the server to aggregate into a global model.
Federated learning is a set-up in which multiple clients collaborate to solve machine learning problems, which is under the coordination of a central aggregator. This setting also allows the training data decentralized to ensure the data privacy of each device. Federated learning adheres to two major ideas: local computing and model transmission, which reduces some systematic privacy risks and costs brought by traditional centralized machine learning methods. The original data of the client is stored locally and cannot be exchanged or migrated. With the application of federated learning, each device uses local data for local training, then uploads the model to the server for aggregation, and finally the server sends the model update to the participants to achieve the learning goal. To provide a comprehensive survey and facilitate the potential research of this area, we systematically introduce the existing works of federated learning from five aspects: data partitioning, privacy mechanism, machine learning model, communication architecture and systems heterogeneity. Then, we sort out the current challenges and future research directions of federated learning. Finally, we summarize the characteristics of existing federated learning, and analyze the current practical application of federated learning.
Age-Aware Data Selection and Aggregator Placement for Timely Federated Continual Learning in Mobile Edge Computing
2024, IEEE Transactions on Computers

View all citing articles on Scopus

Min Liu received her B.S. and M.S. degrees in computer science from Xi’an Jiaotong University, China, in 1999 and 2002, respectively. She got her Ph.D in computer science from the Graduate University of the Chinese Academy of Sciences in 2008. She is currently a professor at the Networking Technology Research Centre, Institute of Computing Technology, Chinese Academy of Sciences. Her current research interests include network coordination for unmanned system, wireless networks and mobile computing.

Bo Yang received the Ph.D. degree in control science and engineering from the University of Chinese Academy of Sciences, Beijing, China, in 2017. He is currently a Research Associate with the Institute of Computing Technology, Chinese Academy of Sciences. His current research interests include wireless networks, edge computing, and multi-agent learning.

Qingxiang Liu received his B.S. degree from Nanjing University of Information Science and Technology, Nanjing, China in 2020. Since 2020, he is currently a master candidate at the Networking Technology Research Centre, Institute of Computing Technology, Chinese Academy of Sciences. His current research interests include big data and mobile cloud computing.

Jizhong Li is currently working in Huawei. His current research interest is focused on distributed computing and middleware.

Xiaobing Guo received the B.S. and M.S. degrees in computer science from Xi’an Jiaotong University in 1999 and 2002, respectively, and the Ph.D. degree in computer science from the Graduate University of Chinese Academy of Sciences in 2013. He is currently a Principal Researcher in Lenovo Research. His current research interests include the blockchain and confidential computing.

View full text

Customized Federated Learning for accelerated edge computing with heterogeneous task targets

Abstract

Introduction

Section snippets

Related work

Preliminaries on FL

CuFL algorithm

Performance evaluation

Conclusion

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgment

Inf. Sci.

ACM Trans. Intell. Syst. Technol.

IEEE Trans. Neural Netw. Learn. Syst.

J. Parallel Distrib. Comput.

Mobile edge computing: A survey on architecture and computation offloading

IEEE Commun. Surv. Tutor.

Service modeling for opportunistic edge computing systems with feature engineering

Comput. Commun.

Wireless network intelligence at the edge

Proc. IEEE

Distributed deep neural networks over the cloud, the edge and end devices

Federated optimization: Distributed machine learning for on-device intelligence

Communication-efficient learning of deep networks from decentralized data

CMFL: Mitigating communication overhead for federated learning

Federated learning for wireless communications: Motivation, opportunities and challenges

Federated learning in mobile edge networks: A comprehensive survey

IEEE Commun. Surv. Tutor.

Nestdnn: Resource-aware multi-tenant on-device deep learning for continuous mobile vision

In-edge AI: Intelligentizing mobile edge computing, caching and communication by federated learning

IEEE Netw.

Adaptive federated learning in resource constrained edge computing systems

IEEE J. Sel. Areas Commun.

Federated learning over wireless networks: Optimization model design and analysis