skip to main content
10.1145/3437984.3458839acmconferencesArticle/Chapter ViewAbstractPublication PageseurosysConference Proceedingsconference-collections
research-article
Open access

Towards Mitigating Device Heterogeneity in Federated Learning via Adaptive Model Quantization

Published: 26 April 2021 Publication History

Abstract

Federated learning (FL) is increasingly becoming the norm for training models over distributed and private datasets. Major service providers rely on FL to improve services such as text auto-completion, virtual keyboards, and item recommendations. Nonetheless, training models with FL in practice requires significant amount of time (days or even weeks) because FL tasks execute in highly heterogeneous environments where devices only have widespread yet limited computing capabilities and network connectivity conditions.
In this paper, we focus on mitigating the extent of device heterogeneity, which is a main contributing factor to training time in FL. We propose AQFL, a simple and practical approach leveraging adaptive model quantization to homogenize the computing resources of the clients. We evaluate AQFL on five common FL benchmarks. The results show that, in heterogeneous settings, AQFL obtains nearly the same quality and fairness of the model trained in homogeneous settings.

References

[1]
2016. General Data Protection Regulation. https://en.wikipedia.org/wiki/General_Data_Protection_Regulation
[2]
2018. California Consumer Privacy Act. https://en.wikipedia.org/wiki/California_Consumer_Privacy_Act
[3]
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep Learning with Differential Privacy. In CCS.
[4]
Ahmed M. Abdelmoniem and Marco Canini. 2021. DC2: Delay-aware Compression Control for Distributed Machine Learning. In INFOCOM.
[5]
Ahmed M. Abdelmoniem, Chen-Yu Ho, Pantelis Papageorgiou, Muhammad Bilal, and Marco Canini. 2021. On the Impact of Device and Behavioral Heterogeneity in Federated Learning. arXiv:2102.07500 [cs.LG]
[6]
AI Benchmark. 2021. Performance Ranking. https://ai-benchmark.com/ranking.html
[7]
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H. Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. 2019. Towards Federated Learning at Scale: System Design. In MLSys.
[8]
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. 2017. Practical Secure Aggregation for Privacy-Preserving Machine Learning. In CCS.
[9]
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. 2018. LEAF: A Benchmark for Federated Settings. arXiv:1812.01097
[10]
Kapil Chandorikar. 2019. Introduction to Federated Learning and Privacy Preservation. https://towardsdatascience.com/introduction-to-federated-learning-and-privacy-preservation-75644686b559.
[11]
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. 2017. EMNIST: Extending MNIST to handwritten letters. In IJCNN.
[12]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei Fei Li. 2009. ImageNet: a Large-Scale Hierarchical Image Database. In CVPR.
[13]
Kunyuan Du, Ya Zhang, and Haibing Guan. 2020. From Quantized DNNs to Quantizable DNNs. arXiv:2004.05284 [cs.CV]
[14]
Nils Gruschka, Vasileios Mavroeidis, Kamer Vishi, and Meiko Jensen. 2018. Privacy Issues and Data Protection in Big Data: A Case Study Analysis under GDPR. Big Data (2018).
[15]
Song Han, Huizi Mao, and William J. Dally. 2016. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. In ICLR.
[16]
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng. 2014. Deep Speech: Scaling up end-to-end speech recognition.
[17]
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. 2018. Federated Learning for Mobile Keyboard Prediction. arXiv:1811.03604
[18]
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2017. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. Journal of Machine Learning Research 18 (2017), 6869--6898.
[19]
Andrey Ignatov, Radu Timofte, Andrei Kulik, Seungsoo Yang, Ke Wang, Felix Baum, Max Wu, Lirong Xu, and Luc Van Gool. 2019. AI Benchmark: All About Deep Learning on Smartphones in 2019. arXiv:1910.06663 [cs.PF]
[20]
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. In IEEE CVPR.
[21]
Raj Jain, Arjan Durresi, and Gojko Babic. 1999. Throughput fairness index: An explanation. ATM Forum contribution 99, 45 (1999).
[22]
Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtarik, Ananda Theertha Suresh, and Dave Bacon. 2016. Federated Learning: Strategies for Improving Communication Efficiency. In Workshop on Private Multi-Party Machine Learning - NeurIPS.
[23]
Raghuraman Krishnamoorthi. 2018. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv:1806.08342 [cs.LG]
[24]
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated Optimization in Heterogeneous Networks. In MLSys.
[25]
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. 2020. Fair Resource Allocation in Federated Learning. In ICLR.
[26]
Dachao Lin, Peiqin Sun, Guangzeng Xie, Shuchang Zhou, and Zhihua Zhang. 2020. Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond. arXiv:2008.13128 [cs.LG]
[27]
Xiaofan Lin, Cong Zhao, and Wei Pan. 2017. Towards Accurate Binary Convolutional Neural Network. In NeurIPS.
[28]
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep Learning Face Attributes in the Wild. In ICCV.
[29]
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In AISTATS.
[30]
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. 2019. Data-Free Quantization Through Weight Equalization and Bias Correction. In IEEE/CVF ICCV.
[31]
Milad Nasr, Reza Shokri, and Amir Houmansadr. 2019. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In IEEE Symposium on Security and Privacy (SP).
[32]
Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. 2018. Privacy-Preserving Deep Learning via Additively Homomorphic Encryption. IEEE Transactions on Information Forensics and Security (2018). https://doi.org/10.1109/TIFS.2017.2787987
[33]
Pushshift. [n.d.]. Reddit Datasets. https://files.pushshift.io/reddit/
[34]
pytorch.org. 2020. QUANTIZATION. https://pytorch.org/docs/stable/quantization.html
[35]
tensorflow.org. 2020. Module: tf.quantization. https://www.tensorflow.org/api_docs/python/tf/quantization
[36]
tensorflow.org. 2020. TensorFlow Federated: Machine Learning on Decentralized Data. https://www.tensorflow.org/federated
[37]
tensorflow.org. 2020. TensorFlow Lite 8-bit quantization specification. https://www.tensorflow.org/lite/performance/quantization_spec
[38]
Shuang Wu, Guoqi Li, Feng Chen, and Luping Shi. 2018. Training and Inference with Integers in Deep Neural Networks. arXiv:1802.04680 [cs.LG]
[39]
Chengxu Yang, QiPeng Wang, Mengwei Xu, Shangguang Wang, Kaigui Bian, and Xuanzhe Liu. 2020. Heterogeneity-Aware Federated Learning. arXiv:2006.06983
[40]
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Federated Machine Learning: Concept and Applications. ACM Transactions on Intelligent Systems and Technology 10, 2 (2019).
[41]
Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. 2018. Applied Federated Learning: Improving Google Keyboard Query Suggestions. arXiv:1812.02903

Cited By

View all
  • (2024)A survey of energy-efficient strategies for federated learning inmobile edge computing移动边缘计算中联邦学习的能效策略综述Frontiers of Information Technology & Electronic Engineering10.1631/FITEE.230018125:5(645-663)Online publication date: 7-Jun-2024
  • (2024)FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated ClientsProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661880(398-411)Online publication date: 3-Jun-2024
  • (2024)Towards Energy-efficient Federated Learning via INT8-based Training on Mobile DSPsProceedings of the ACM Web Conference 202410.1145/3589334.3645341(2786-2794)Online publication date: 13-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
EuroMLSys '21: Proceedings of the 1st Workshop on Machine Learning and Systems
April 2021
130 pages
ISBN:9781450382984
DOI:10.1145/3437984
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 April 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Federated Learning
  2. Heterogeneity
  3. Model Quantization

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

EuroSys '21
Sponsor:

Acceptance Rates

EuroMLSys '21 Paper Acceptance Rate 18 of 26 submissions, 69%;
Overall Acceptance Rate 18 of 26 submissions, 69%

Upcoming Conference

EuroSys '25
Twentieth European Conference on Computer Systems
March 30 - April 3, 2025
Rotterdam , Netherlands

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)300
  • Downloads (Last 6 weeks)38
Reflects downloads up to 26 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)A survey of energy-efficient strategies for federated learning inmobile edge computing移动边缘计算中联邦学习的能效策略综述Frontiers of Information Technology & Electronic Engineering10.1631/FITEE.230018125:5(645-663)Online publication date: 7-Jun-2024
  • (2024)FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated ClientsProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661880(398-411)Online publication date: 3-Jun-2024
  • (2024)Towards Energy-efficient Federated Learning via INT8-based Training on Mobile DSPsProceedings of the ACM Web Conference 202410.1145/3589334.3645341(2786-2794)Online publication date: 13-May-2024
  • (2024)An Energy Efficient Soft SIMD Microarchitecture and Its Application on Quantized CNNsIEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2024.337579332:6(1018-1031)Online publication date: Jun-2024
  • (2024)Fed-RAC: Resource-Aware Clustering for Tackling Heterogeneity of Participants in Federated LearningIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2024.337993335:7(1207-1220)Online publication date: Jul-2024
  • (2024)Resource-Aware Split Federated Learning for Edge Intelligence2024 IEEE 3rd Workshop on Machine Learning on Edge in Sensor Systems (SenSys-ML)10.1109/SenSys-ML62579.2024.00008(15-20)Online publication date: 13-May-2024
  • (2024)Performance Profiling of Federated Learning Across Heterogeneous Mobile Devices2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C63300.2024.00053(363-372)Online publication date: 1-Jul-2024
  • (2024)Federated Learning for Healthcare ApplicationsIEEE Internet of Things Journal10.1109/JIOT.2023.332582211:5(7339-7358)Online publication date: 1-Mar-2024
  • (2024)Hetero-FedIoT: A Rule-Based Interworking Architecture for Heterogeneous Federated IoT NetworksIEEE Internet of Things Journal10.1109/JIOT.2023.330857911:4(5920-5938)Online publication date: 15-Feb-2024
  • (2024)Mixed-Precision Quantization for Federated Learning on Resource-Constrained Heterogeneous Devices2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00587(6138-6148)Online publication date: 16-Jun-2024
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media