Cited By
View all- Yin KDing ZJi XWang Z(2025)Controlling update distance and enhancing fair trainable prototypes in federated learning under data and model heterogeneityDefence Technology10.1016/j.dt.2024.12.024Online publication date: Jan-2025
How Few Davids Improve One Goliath: Federated Learning in Resource-Skewed Edge Computing Environments
Authors: 
Jiayun Zhang, Shuheng Li, Haiyu Huang, Zihan Wang, Xiaohan Fu, Dezhi Hong, 
Rajesh K. Gupta, 
Jingbo ShangAuthors Info & Claims
Jiayun Zhang, Shuheng Li, Haiyu Huang, Zihan Wang, Xiaohan Fu, Dezhi Hong, Rajesh K. Gupta, Jingbo ShangAuthors Info & ClaimsPages 2976 - 2985
Abstract
Real-world deployment of federated learning requires orchestrating clients with widely varied compute resources, from strong enterprise-grade devices in data centers to weak mobile and Web-of-Things devices. Prior works have attempted to downscale large models for weak devices and aggregate shared parts among heterogeneous models. A typical architectural assumption is that there are equally many strong and weak devices. In reality, however, we often encounter resource skew where a few (1 or 2) strong devices hold substantial data resources, alongside many weak devices. This poses challenges-the unshared portion of the large model rarely receives updates or gains benefits from weak collaborators.
We aim to facilitate reciprocal benefits between strong and weak devices in resource-skewed environments. We propose RecipFL, a novel framework featuring a server-side graph hypernetwork. This hypernetwork is trained to produce parameters for personalized client models adapted to device capacity and unique data distribution. It effectively generalizes knowledge about parameters across different model architectures by encoding computational graphs. Notably, RecipFL is agnostic to model scaling strategies and supports collaboration among arbitrary neural networks. We establish the generalization bound of RecipFL through theoretical analysis and conduct extensive experiments with various model architectures. Results show that RecipFL improves accuracy by 4.5% and 7.4% for strong and weak devices respectively, incentivizing both devices to actively engage in federated learning.
Supplemental Material
MP4 File
video presentation
- Download
- 1465.83 MB
MP4 File
Supplemental video
- Download
- 9.96 MB
References
[1]
Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang. 2022. FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extraction. Advances in Neural Information Processing Systems, Vol. 35 (2022), 29677--29690.
[2]
Jimmy Ba and Rich Caruana. 2014. Do Deep Nets Really Need to be Deep? Advances In Neural Information Processing Systems, Vol. 27 (2014).
[3]
Jonathan Baxter. 2000. A Model of Inductive Bias Learning. Journal of Artificial Intelligence Research, Vol. 12 (2000), 149--198.
[4]
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv preprint arXiv:1406.1078 (2014).
[5]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 (2018).
[6]
Enmao Diao, Jie Ding, and Vahid Tarokh. 2020. HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients. arXiv preprint arXiv:2010.01264 (2020).
[7]
Xiuwen Fang and Mang Ye. 2022. Robust Federated Learning with Noisy and Heterogeneous Clients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10072--10081.
[8]
David Ha, Andrew M. Dai, and Quoc V. Le. 2017. HyperNetworks. In International Conference on Learning Representations.
[9]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognitio. 770--778.
[10]
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531 (2015).
[11]
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-Efficient Transfer Learning for NLP. In International Conference on Machine Learning. PMLR, 2790--2799.
[12]
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. 2019. Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification. arXiv preprint arXiv:1909.06335 (2019).
[13]
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely Connected Convolutional Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4700--4708.
[14]
Zhida Jiang, Yang Xu, Hongli Xu, Zhiyuan Wang, Chunming Qiao, and Yangming Zhao. 2022. FedMP: Federated Learning through Adaptive Model Pruning in Heterogeneous Edge Computing. In 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 767--779.
[15]
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. In International conference on machine learning. PMLR, 5132--5143.
[16]
Minjae Kim, Sangyoon Yu, Suhyun Kim, and Soo-Mook Moon. 2023. DepthFL: Depthwise Federated Learning for Heterogeneous Clients. In The Eleventh International Conference on Learning Representations.
[17]
Boris Knyazev, Michal Drozdzal, Graham W Taylor, and Adriana Romero Soriano. 2021. Parameter Prediction for Unseen Deep Architectures. Advances in Neural Information Processing Systems, Vol. 34 (2021), 29433--29448.
[18]
Jakub Konevc nỳ, Brendan McMahan, and Daniel Ramage. 2015. Federated Optimization: Distributed Optimization Beyond the Datacenter. arXiv preprint arXiv:1511.03575 (2015).
[19]
Jakub Konevc nỳ, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. 2016. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 (2016).
[20]
Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning Multiple Layers of Features from Tiny Images. (2009).
[21]
Fan Lai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury. 2021. Oort: Efficient Federated Learning via Guided Participant Selection. In 15th $$USENIX$$ Symposium on Operating Systems Design and Implementation ($$OSDI$$ 21). 19--35.
[22]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-Based Learning Applied to Document Recognition. Proc. IEEE, Vol. 86, 11 (1998), 2278--2324.
[23]
Ang Li, Jingwei Sun, Pengcheng Li, Yu Pu, Hai Li, and Yiran Chen. 2021a. Hermes: An Efficient Federated Learning Framework for Heterogeneous Mobile Clients. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. 420--437.
[24]
Ang Li, Jingwei Sun, Xiao Zeng, Mi Zhang, Hai Li, and Yiran Chen. 2021b. FedMask: Joint Computation and Communication-Efficient Personalized Federated Learning via Heterogeneous Masking. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems. 42--55.
[25]
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated Optimization in Heterogeneous Networks. Proceedings of Machine Learning and Systems, Vol. 2 (2020), 429--450.
[26]
Xin-Chun Li and De-Chuan Zhan. 2021. FedRS: Federated Learning with Restricted Softmax for Label Distribution Non-IID Data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 995--1005.
[27]
Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. 2015. Gated Graph Sequence Neural Networks. arXiv preprint arXiv:1511.05493 (2015).
[28]
Or Litany, Haggai Maron, David Acuna, Jan Kautz, Gal Chechik, and Sanja Fidler. 2022. Federated Learning with Heterogeneous Architectures using Graph HyperNetworks. arXiv preprint arXiv:2201.08459 (2022).
[29]
Ruixuan Liu, Fangzhao Wu, Chuhan Wu, Yanlin Wang, Lingjuan Lyu, Hong Chen, and Xing Xie. 2022. No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3398--3406.
[30]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Artificial Intelligence and Statistics. PMLR, 1273--1282.
[31]
Chaoyue Niu, Fan Wu, Shaojie Tang, Lifeng Hua, Rongfei Jia, Chengfei Lv, Zhihua Wu, and Guihai Chen. 2020. Billion-Scale Federated Learning on Mobile Clients: A Submodel Design with Tunable Privacy. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. 1--14.
[32]
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019).
[33]
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE Transactions on Neural Networks, Vol. 20, 1 (2008), 61--80.
[34]
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. 2021. Personalized Federated Learning using Hypernetworks. In International Conference on Machine Learning. PMLR, 9489--9502.
[35]
Chunlin Tian, Li Li, Zhan Shi, Jun Wang, and ChengZhong Xu. 2022. HARMONY: Heterogeneity-Aware Hierarchical Management for Federated Learning System. In 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 631--645.
[36]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. Advances In Neural Information Processing Systems, Vol. 30 (2017).
[37]
Johannes Von Oswald, Christian Henning, Benjamin F Grewe, and Jo ao Sacramento. 2019. Continual learning with hypernetworks. arXiv preprint arXiv:1906.00695 (2019).
[38]
Kaibin Wang, Qiang He, Feifei Chen, Chunyang Chen, Faliang Huang, Hai Jin, and Yun Yang. 2023. FlexiFed: Personalized Federated Learning for Edge Clients with Heterogeneous Model Architectures. In Proceedings of the ACM Web Conference 2023. 2979--2990.
[39]
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. arXiv preprint arXiv:1704.05426 (2017).
[40]
Chris Zhang, Mengye Ren, and Raquel Urtasun. 2018. Graph HyperNetworks for Neural Architecture Search. arXiv preprint arXiv:1810.05749 (2018).
[41]
Jie Zhang, Song Guo, Xiaosong Ma, Haozhao Wang, Wenchao Xu, and Feijie Wu. 2021. Parameterized Knowledge Transfer for Personalized Federated Learning. Advances in Neural Information Processing Systems, Vol. 34 (2021), 10092--10104.
[42]
Jiayun Zhang, Xiyuan Zhang, Xinyang Zhang, Dezhi Hong, Rajesh K. Gupta, and Jingbo Shang. 2023. Navigating Alignment for Non-identical Client Class Sets: A Label Name-Anchored Federated Learning Framework. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.
Index Terms
- How Few Davids Improve One Goliath: Federated Learning in Resource-Skewed Edge Computing Environments
Recommendations
Reinforcement Learning for Real-Time Federated Learning for Resource-Constrained Edge Cluster
AbstractFor performing various predictive analytics tasks for real-time mission-critical applications, Federated Learning (FL) have emerged as the go-to machine learning paradigm for its ability to leverage perform machine learning workloads on resource-...
An Analysis about Federated Learning in Low-Powerful Devices
Mid4CC '23: Proceedings of the 1st International Workshop on Middleware for the Computing ContinuumFederated Learning allows us to train Machine Learning models in a distributed way. This improves users' security and privacy and allows the computational load to be distributed. One of the advantages is the application of these models on low-powerful ...
Learning From Your Neighbours: Mobility-Driven Device-Edge-Cloud Federated Learning
ICPP '23: Proceedings of the 52nd International Conference on Parallel ProcessingFederated learning (FL) in large-scale wireless networks is implemented in a hierarchical way by introducing edge servers as relays between the cloud server and devices, where devices are dispersed within multiple clusters coordinated by edges. However, ...
Comments
Information & Contributors
Information
Published In
May 2024
4826 pages
ISBN:9798400701719
DOI:10.1145/3589334
- General Chairs:
- Tat-Seng Chua,
- Chong-Wah Ngo,
- Proceedings Chair:
- Roy Ka-Wei Lee,
- Program Chairs:
- Ravi Kumar,
- Hady W. Lauw
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- NSF (National Science Foundation)
- ACE SRC JUMP 2.0
Conference
WWW '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,899 of 8,196 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 648Total Downloads
- Downloads (Last 12 months)648
- Downloads (Last 6 weeks)51
Reflects downloads up to 03 Mar 2025
Other Metrics
Citations
Cited By
View all- Yin KDing ZJi XWang Z(2025)Controlling update distance and enhancing fair trainable prototypes in federated learning under data and model heterogeneityDefence Technology10.1016/j.dt.2024.12.024Online publication date: Jan-2025
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in