FedNLR: Federated Learning with Neuron-wise Learning Rates
Authors: 
Haozhao Wang, Peirong Zheng, Xingshuo Han, Wenchao Xu, 
Ruixuan Li, Tianwei ZhangAuthors Info & Claims
Haozhao Wang, Peirong Zheng, Xingshuo Han, Wenchao Xu, Ruixuan Li, Tianwei ZhangAuthors Info & ClaimsPages 3069 - 3080
Abstract
Federated Learning (FL) suffers from severe performance degradation due to the data heterogeneity among clients. Some existing work suggests that the fundamental reason is that data heterogeneity can cause local model drift, and therefore proposes to calibrate the direction of local updates to solve this problem. Though effective, existing methods generally take the model as a whole, which lacks a deep understanding of how the neurons within deep classification models evolve during local training to form model drift. In this paper, we bridge this gap by performing an intuitive and theoretical analysis of the activation changes of each neuron during local training. Our analysis shows that the high activation of some neurons on the samples of a certain class will be reduced during local training when these samples are not included in the client, which we call neuron drift, thus leading to the performance reduction of this class. Motivated by this, we propose a novel and simple algorithm called FedNLR, which utilizes <u>N</u>euron-wise <u>L</u>earning <u>R</u>ates during the FL local training process. The principle behind this is to enhance the learning of neurons bound to local classes on local data knowledge while reducing the decay of non-local classes knowledge stored in neurons. Experimental results demonstrate that FedNLR achieves state-of-the-art performance on federated learning with popular deep neural networks.
Supplemental Material
MP4 File - Video for FedNLR: Federated Learning with Neuron-wise Learning Rates
A promote video for paper FedNLR: Federated Learning with Neuron-wise Learning Rates. The video illustrates the core motivation and idea of our paper.
- Download
- 3.42 MB
References
[1]
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, and Venkatesh Saligrama. 2021. Federated Learning Based on Dynamic Regularization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3--7, 2021.
[2]
Muhammad Ammad-ud-din, Elena Ivannikova, Suleiman A. Khan, Were Oyomno, Qiang Fu, Kuan Eeik Tan, and Adrian Flanagan. 2019. Federated Collaborative Filtering for Privacy-Preserving Personalized Recommendation System. CoRR, Vol. abs/1901.09888 (2019).
[3]
Zachary Charles and Jakub Konevcný. 2021. Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning. In The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13--15, 2021, Virtual Event. 2575--2583.
[4]
Luke Nicholas Darlow, Elliot J. Crowley, Antreas Antoniou, and Amos J. Storkey. 2018. CINIC-10 is not ImageNet or CIFAR-10. CoRR, Vol. abs/1810.03505 (2018).
[5]
Canh T. Dinh, Nguyen H. Tran, and Tuan Dung Nguyen. 2020. Personalized Federated Learning with Moreau Envelopes. In Proceedings of Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS.
[6]
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. 2021. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, Vol. 118, 43 (2021), e2103091118.
[7]
Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. 2022. FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18--24. 10102--10111.
[8]
Yonghai Gong, Yichuan Li, and Nikolaos M. Freris. 2022. FedADMM: A Robust Federated Deep Learning Framework with Adaptivity to System Heterogeneity. In 38th IEEE International Conference on Data Engineering, ICDE 2022, Kuala Lumpur, Malaysia, May 9--12, 2022. 2575--2587.
[9]
Pengfei Guo, Puyang Wang, Jinyuan Zhou, Shanshan Jiang, and Vishal M. Patel. [n.,d.]. Multi-Institutional Collaborations for Improving Deep Learning-Based Magnetic Resonance Image Reconstruction Using Federated Learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19--25, 2021. 2423--2432.
[10]
X. Y. Han, Vardan Papyan, and David L. Donoho. 2022. Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25--29, 2022.
[11]
Hangfeng He and Weijie J. Su. 2022. A Law of Data Separation in Deep Learning. CoRR, Vol. abs/2210.17020 (2022).
[12]
Ming Hu, Yue Cao, Anran Li, Zhiming Li, Chengwei Liu, Tianlin Li, Mingsong Chen, and Yang Liu. 2024. FedMut: Generalized Federated Learning via Stochastic Mutation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 12528--12537.
[13]
Ming Hu, Zeke Xia, Dengke Yan, Zhihao Yue, Jun Xia, Yihao Huang, Yang Liu, and Mingsong Chen. 2023. GitFL: Uncertainty-Aware Real-Time Asynchronous Federated Learning Using Version Control. In In Proceedings of IEEE Real-Time Systems Symposium (RTSS). IEEE, 145--157.
[14]
Ming Hu, Peiheng Zhou, Zhihao Yue, Zhiwei Ling, Yihao Huang, Anran Li, Yang Liu, Xiang Lian, and Mingsong Chen. 2024. FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregation. In IEEE International Conference on Data Engineering (ICDE). IEEE, 2137--2150.
[15]
Like Hui, Mikhail Belkin, and Preetum Nakkiran. 2022. Limitations of Neural Collapse for Understanding Generalization in Deep Learning. CoRR, Vol. abs/2202.08384 (2022).
[16]
Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. 2023. FedExP: Speeding Up Federated Averaging via Extrapolation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1--5.
[17]
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. [n.,d.]. SCAFFOLD: Stochastic Controlled Averaging for Federated Learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13--18 July 2020, Virtual Event. 5132--5143.
[18]
Geeho Kim, Jinkyu Kim, and Bohyung Han. 2022. Communication-Efficient Federated Learning with Acceleration of Global Momentum. CoRR, Vol. abs/2201.03172 (2022).
[19]
Jinkyu Kim, Geeho Kim, and Bohyung Han. [n.,d.]. Multi-Level Branched Regularization for Federated Learning. In International Conference on Machine Learning, ICML 2022, 17--23 July 2022, Baltimore, Maryland, USA. 11058--11073.
[20]
Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).
[21]
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. 2022. Federated Learning on Non-IID Data Silos: An Experimental Study. In ICDE. IEEE, 965--978.
[22]
Qinbin Li, Bingsheng He, and Dawn Song. [n.,d.] a. Model-Contrastive Federated Learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19--25, 2021. 10713--10722.
[23]
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated Optimization in Heterogeneous Networks. In Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2--4, 2020.
[24]
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, Vol. 2 (2020), 429--450.
[25]
Xin-Chun Li, Yi-Chu Xu, Shaoming Song, Bingshuai Li, Yinchuan Li, Yunfeng Shao, and De-Chuan Zhan. [n.,d.] b. Federated Learning with Position-Aware Neurons. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18--24, 2022. 10072--10081.
[26]
Xin-Chun Li and De-Chuan Zhan. 2021. FedRS: Federated Learning with Restricted Softmax for Label Distribution Non-IID Data. In KDD '21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14--18, 2021. 995--1005.
[27]
Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. [n.,d.]. FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19--25, 2021. 1013--1023.
[28]
Jianfeng Lu and Stefan Steinerberger. 2020. Neural Collapse with Cross-Entropy Loss. CoRR, Vol. abs/2012.08465 (2020).
[29]
Kangyang Luo, Xiang Li, Yunshi Lan, and Ming Gao. 2023. GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgetting. The IEEE/CVF Conference on Computer Vision and Pattern Recognition, (CVPR 2023), Vancouver, Canada (2023).
[30]
Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. 2021. No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6--14, 2021, virtual. 5972--5984.
[31]
Grigory Malinovskiy, Dmitry Kovalev, Elnur Gasanov, Laurent Condat, and Peter Richtárik. 2020. From Local SGD to Local Fixed-Point Methods for Federated Learning. In Proceedings of the 37th International Conference on Machine Learning, ICML, 13--18 July, Virtual Event. 6692--6701.
[32]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. 1273--1282.
[33]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS.
[34]
Vardan Papyan. [n.,d.]. Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra., Vol. Journal of Machine Learning Research, 2020, 252(21):1--64. ( [n.,d.]).
[35]
Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Franccoise Beaufays. 2019. Federated Learning for Emoji Prediction in a Mobile Keyboard. CoRR, Vol. abs/1906.04329 (2019).
[36]
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konevcný, Sanjiv Kumar, and Hugh Brendan McMahan. 2021. Adaptive Federated Optimization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3--7.
[37]
Yan Sun, Li Shen, Tiansheng Huang, Liang Ding, and Dacheng Tao. 2023. FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracy. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1--5.
[38]
Chunnan Wang, Xiang Chen, Junzhe Wang, and Hongzhi Wang. June 18--24, 2022. ATPFL: Automatic Trajectory Prediction Model Design under Federated Learning Framework. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, New Orleans, LA, USA. 6553--6562.
[39]
Haozhao Wang, Yichen Li, Wenchao Xu, Ruixuan Li, Yufeng Zhan, and Zhigang Zeng. 2023. DaFKD: Domain-aware Federated Knowledge Distillation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17--24, 2023. IEEE, 20412--20421.
[40]
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. 2020. Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6--12, 2020, virtual.
[41]
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael G. Rabbat. 2020. SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26--30, 2020.
[42]
Feijie Wu, Song Guo, Zhihao Qu, Shiqi He, Ziming Liu, and Jing Gao. 2023. Anchor Sampling for Federated Learning with Partial Client Participation. In Proceedings of the 40th International Conference on Machine Learning. 37379--37416.
[43]
Feijie Wu, Song Guo, Haozhao Wang, Haobo Zhang, Zhihao Qu, Jie Zhang, and Ziming Liu. 2023. From Deterioration to Acceleration: A Calibration Approach to Rehabilitating Step Asynchronism in Federated Optimization. IEEE Trans. Parallel Distributed Syst., Vol. 34, 5 (2023), 1548--1559.
[44]
An Xu, Wenqi Li, Pengfei Guo, Dong Yang, Holger Roth, Ali Hatamizadeh, Can Zhao, Daguang Xu, Heng Huang, and Ziyue Xu. [n.,d.]. Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18--24, 2022. 20834--20843.
[45]
Jing Xu, Sen Wang, Liwei Wang, and Andrew Chi-Chih Yao. 2021. FedCM: Federated Learning with Client-level Momentum. CoRR, Vol. abs/2106.10874 (2021).
[46]
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. 2019. Bayesian Nonparametric Federated Learning of Neural Networks. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 7252--7261.
[47]
Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. [n.,d.]. Federated Learning with Label Distribution Skew via Logits Calibration. In International Conference on Machine Learning, ICML 2022, 17--23 July 2022, Baltimore, Maryland, USA. 26311--26329.
Index Terms
- FedNLR: Federated Learning with Neuron-wise Learning Rates
Index terms have been assigned to the content through auto-classification.
Recommendations
Towards federated unsupervised representation learning
EdgeSys '20: Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and NetworkingMaking deep learning models efficient at inferring nowadays requires training with an extensive number of labeled data that are gathered in a centralized system. However, gathering labeled data is an expensive and time-consuming process, centralized ...
Non-IID always Bad? Semi-Supervised Heterogeneous Federated Learning with Local Knowledge Enhancement
CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge ManagementFederated learning (FL) is important for privacy-preserving services by training models without collecting raw user data. Most FL algorithms assume all data is annotated, which is impractical due to the high cost of labeling data in real applications. To ...
Fast deep autoencoder for federated learning
Highlights- A fast and privacy-preserving training algorithm for deep autoencoders is proposed.
- The encoder block of the autoencoder is trained using Singular Value Decomposition.
- The decoder block of the autoencoder is trained layer by layer.
AbstractThis paper presents a novel, fast and privacy preserving implementation of deep autoencoders. DAEF (Deep AutoEncoder for Federated learning), unlike traditional neural networks, trains a deep autoencoder network in a non-iterative way, which ...
Comments
Information & Contributors
Information
Published In
August 2024
6901 pages
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 August 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
KDD '24: The 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
August 25 - 29, 2024
Barcelona, Spain
Acceptance Rates
Overall Acceptance Rate 1,133 of 8,635 submissions, 13%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 288Total Downloads
- Downloads (Last 12 months)288
- Downloads (Last 6 weeks)57
Reflects downloads up to 01 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in