Skip to main content

Advertisement

Stabilizing and improving federated learning with highly non-iid data and client dropout

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The label distribution skew has been shown to be a significant obstacle that limits the model performance in federated learning (FL). This challenge could be more serious when the participating clients are in unstable network circumstances and drop out frequently. Previous works have demonstrated that the classifier head is particularly sensitive to the label skew. Therefore, maintaining a balanced classifier head is of significant importance for building a good and unbiased global model. To this end, we propose a simple yet effective framework by introducing a calibrated softmax function with smoothed prior for computing the cross-entropy loss, and a prototype-based feature augmentation scheme to re-balance the local training, which provide a new perspective on tackling the label distribution skew in FL and are lightweight for edge devices and can facilitate the global model aggregation. With extensive experiments on two benchmark classification tasks of Fashion-MNIST and CIFAR-10, our numerical results demonstrate that our proposed method can consistently outperform the baselines, 2 8% of accuracy over FedAvg in the presence of severe label skew and client dropout.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

The used datasets can be publicly accessed from the Internet with details provided in reference.

Materials Availability

Not applicable

Code Availability

Code is available at https://github.com/JianXu95/ReBaFL

References

  1. Hinton G, Deng L, Yu D, Dahl GE, Mohamed A, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN, Kingsbury B (2012) Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Process Magaz 29(6):82–97

    Article  Google Scholar 

  2. Hayyolalam V, Aloqaily M, Özkasap Ö, Guizani M (2022) Edge-assisted solutions for iot-based connected healthcare systems: A literature review. IEEE Internet Things J. 9(12):9419–9443

    Article  Google Scholar 

  3. Ullah Z, Al-Turjman F, Mostarda L, Gagliardi R (2020) Applications of artificial intelligence and machine learning in smart cities. Comput Commun 154:313–323

    Article  Google Scholar 

  4. Sharma S (2019) Data privacy and gdpr handbook

  5. McMahan B, Moore E, Ramage D, Hampson S, Arcas BA (2017) Communication-efficient learning of deep networks from decentralized data. In: International Conference on Artificial Intelligence and Statistics, AISTATS

  6. Yang Q, Liu Y, Chen T, Tong Y (2019) Federated machine learning: Concept and applications. ACM Trans Intell Syst Technol 10(2)

  7. Li T, Sahu AK, Talwalkar A, Smith V (2020) Federated learning: Challenges, methods, and future directions. IEEE Signal Process Mag 37(3):50–60

    Article  Google Scholar 

  8. Li T, Sahu AK, Zaheer M, Sanjabi M, Talwalkar A, Smith V (2020) Federated optimization in heterogeneous networks. In: Proceedings of Machine Learning and Systems 2020, MLSys

  9. Wahab OA, Mourad A, Otrok H, Taleb T (2021) Federated machine learning: Survey, multi-level classification, desirable criteria and future directions in communication and networking systems. IEEE Commun Surv Tutorials 23(2):1342–1397

    Article  Google Scholar 

  10. Kairouz P, McMahan HB, Avent B, Bellet A, Bennis M (2021) al: Advances and open problems in federated learning. Found Trends Mach Learn 14(1–2):1–210

    Article  Google Scholar 

  11. Wang S, Tuor T, Salonidis T, Leung KK, Makaya C, He T, Chan K (2019) Adaptive federated learning in resource constrained edge computing systems. IEEE J Sel Areas Commun 37(6):1205–1221

    Article  Google Scholar 

  12. Karimireddy SP, Kale S, Mohri M, Reddi SJ, Stich SU, Suresh AT (2020) SCAFFOLD: stochastic controlled averaging for federated learning. In: Proceedings of the 37th International Conference on Machine Learning, ICML

  13. Acar DAE, Zhao Y, Navarro RM, Mattina M, Whatmough PN, Saligrama V (2021) Federated learning based on dynamic regularization. In: 9th International Conference on Learning Representations, ICLR

  14. Li Q, He B, Song D (2021) Model-contrastive federated learning. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR

  15. Li Q, Diao Y, Chen Q, He B (2022) Federated learning on non-iid data silos: An experimental study. In: 2022 IEEE 38th International Conference on Data Engineering (ICDE), pp 965–978

  16. Ren J, Yu C, Sheng S, Ma X, Zhao H, Yi S, Li H (2020) Balanced meta-softmax for long-tailed visual recognition. In: Annual Conference on Neural Information Processing Systems, NeurIPS

  17. Luo M, Chen F, Hu D, Zhang Y, Liang J, Feng J (2021) No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Adv Neural Inf Process Syst 34:5972–5984

    Google Scholar 

  18. Zhang X, Hong M, Dhople S, Yin W, Liu Y (2021) Fedpd: A federated learning framework with adaptivity to non-iid data. IEEE Trans Signal Process 69:6055–6070

    Article  MathSciNet  Google Scholar 

  19. Ozfatura E, Ozfatura K, Gündüz D (2021) Fedadc: Accelerated federated learning with drift control. In: IEEE International Symposium on Information Theory, ISIT

  20. Hsu TH, Qi H, Brown M (2020) Federated visual classification with real-world data distribution. In: Computer Vision - ECCV 2020 - 16th European Conference

  21. Wang L, Xu S, Wang X, Zhu Q (2021) Addressing class imbalance in federated learning. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI

  22. Yoon T, Shin S, Hwang SJ, Yang E (2021) Fedmix: Approximation of mixup under mean augmented federated learning. In: 9th International Conference on Learning Representations, ICLR

  23. Wang H, Kaplan Z, Niu D, Li B (2020) Optimizing federated learning on non-iid data with reinforcement learning. In: IEEE Conference on Computer Communications, INFOCOM

  24. Tang M, Ning X, Wang Y, Sun J, Wang Y, Li H, Chen Y (2022) Fedcor: Correlation-based active client selection strategy for heterogeneous federated learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 10102–10111

  25. Fraboni Y, Vidal R, Kameni L, Lorenzi M (2021) Clustered sampling: Low-variance and improved representativity for clients selection in federated learning. In: Proceedings of the 38th International Conference on Machine Learning, ICML

  26. Chen H, Chao W (2021) Fedbe: Making bayesian model ensemble applicable to federated learning. In: International Conference on Learning Representations, ICLR

  27. Lin T, Kong L, Stich SU, Jaggi M (2020) Ensemble distillation for robust model fusion in federated learning. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS

  28. Zhu Z, Hong J, Zhou J (2021) Data-free knowledge distillation for heterogeneous federated learning. In: Proceedings of the 38th International Conference on Machine Learning, ICML

  29. Zhang L, Shen L, Ding L, Tao D, Duan L (2022) Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp 10164–10173

  30. Kulkarni V, Kulkarni M, Pant A (2020) Survey of personalization techniques for federated learning. In: Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4)

  31. Dinh CT, Tran NH, Nguyen TD (2020) Personalized federated learning with moreau envelopes. In: Conference on Neural Information Processing Systems, NeurIPS

  32. Fallah A, Mokhtari A, Ozdaglar AE (2020) Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In: Advances in Neural Information Processing Systems, NeurIPS

  33. Collins L, Hassani H, Mokhtari A, Shakkottai S (2021) Exploiting shared representations for personalized federated learning. In: Proceedings of the 38th International Conference on Machine Learning, ICML

  34. Sattler F, Müller K, Samek W (2021) Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Trans Neural Networks Learn Syst 32(8):3710–3722

    Article  MathSciNet  Google Scholar 

  35. Ghosh A, Chung J, Yin D, Ramchandran K (2022) An efficient framework for clustered federated learning. IEEE Trans Inf Theory 68(12):8076–8091

    Article  MathSciNet  Google Scholar 

  36. Chen Y, Ning Y, Slawski M, Rangwala H (2020) Asynchronous online federated learning for edge devices with non-iid data. In: IEEE International Conference on Big Data (BigData)

  37. Wu W, He L, Lin W, Mao R, Maple C, Jarvis SA (2021) SAFA: A semi-asynchronous protocol for fast federated learning with low overhead. IEEE Trans Comput 70(5):655–668

    Article  MathSciNet  Google Scholar 

  38. Gu X, Huang K, Zhang J, Huang L (2021) Fast federated learning in the presence of arbitrary device unavailability. In: Annual Conference on Neural Information Processing Systems, NeurIPS

  39. Zhu L, Lin H, Lu Y, Lin Y, Han S (2021) Delayed gradient averaging: Tolerate the communication latency for federated learning. In: Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems, NeurIPS

  40. Reisizadeh A, Tziotis I, Hassani H, Mokhtari A, Pedarsani R (2022) Straggler-resilient federated learning: Leveraging the interplay between statistical accuracy and system heterogeneity. IEEE J Sel Areas Inf Theory 3(2):197–205

    Article  Google Scholar 

  41. Luo B, Xiao W, Wang S, Huang J, Tassiulas L (2022) Tackling system and statistical heterogeneity for federated learning with adaptive client sampling. In: IEEE INFOCOM 2022 - IEEE Conference on Computer Communications

  42. Ma Z, Xu Y, Xu H, Meng Z, Huang L, Xue Y (2021) Adaptive batch size for federated learning in resource-constrained edge computing. IEEE Trans Mobile Comput 22(1):37–53

    Article  Google Scholar 

  43. Li L, Duan M, Liu D, Zhang Y, Ren A, Chen X, Tan Y, Wang C (2021) Fedsae: A novel self-adaptive federated learning framework in heterogeneous systems. In: International Joint Conference on Neural Networks, IJCNN

  44. Jhunjhunwala D, Sharma P, Nagarkatti A, Joshi G (2022) Fedvarp: Tackling the variance due to partial client participation in federated learning. In: Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, UAI

  45. Wang H, Xu J (2024) Friends to help: Saving federated learning from client dropout. In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 8896–8900 . IEEE

  46. Tian J, Liu Y, Glaser N, Hsu Y, Kira Z (2020) Posterior re-calibration for imbalanced datasets. In: Annual Conference on Neural Information Processing System, NeurIPS

  47. Li X-C, Zhan D-C (2021) FedRS: Federated learning with restricted softmax for label distribution non-iid data. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp 995–1005

  48. Yin X, Yu X, Sohn K, Liu X, Chandraker M (2019) Feature transfer learning for face recognition with under-represented data. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR

  49. Xiao H, Rasul K, Vollgraf R (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR arXiv:1708.07747

  50. Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. University of Toronto

  51. Oh J, Kim S, Yun S (2022) Fedbabu: Toward enhanced representation for federated image classification. In: The Tenth International Conference on Learning Representations, ICLR

  52. Chen H, Chao W (2022) On bridging generic and personalized federated learning for image classification. In: The Tenth International Conference on Learning Representations, ICLR

  53. Zhang J, Li Z, Li B, Xu J, Wu S, Ding S, Wu C (2022) Federated learning with label distribution skew via logits calibration. In: International Conference on Machine Learning, ICML

  54. Smith SL, Elsen E, De S (2020) On the generalization benefit of noise in stochastic gradient descent. In: Proceedings of the 37th International Conference on Machine Learning, ICML

  55. Dosovitskiy A, Brox T (2016) Inverting visual representations with convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR

  56. Kingma DP, Welling M (2014) Auto-encoding variational bayes. In: 2nd International Conference on Learning Representations, ICLR

  57. Ren H, Deng J, Xie X (2022) GRNN: generative regression neural network - A data leakage attack for federated learning. ACM Trans Intell Syst Technol 13(4):65–16524

    Article  Google Scholar 

  58. Liu R, Cao Y, Chen H, Guo R, Yoshikawa M (2021) FLAME: differentially private federated learning in the shuffle model. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI

  59. Girgis AM, Data D, Diggavi SN, Kairouz P, Suresh AT (2021) Shuffled model of differential privacy in federated learning. In: The 24th International Conference on Artificial Intelligence and Statistics, AISTATS

Download references

Acknowledgements

The research of Shao-Lun Huang is supported in part by National Key R&D Program of China under Grant 2021YFA0715202, Shenzhen Key Laboratory of Ubiquitous Data Enabling under Grant ZDSYS20220527171406015 and the Shenzhen Science and Technology Program under Grant KQTD20170810150821146.

Funding

The research of Shao-Lun Huang is supported in part by National Key R&D Program of China under Grant 2021YFA0715202?Shenzhen Key Laboratory of Ubiquitous Data Enabling under Grant ZDSYS20220527171406015 and the Shenzhen Science and Technology Program under Grant KQTD20170810150821146.

Author information

Authors and Affiliations

Authors

Contributions

All authors contributed to the study conception and design. Experiment execution and paper writing were performed by Jian Xu and Meilin Yang. The first draft of the manuscript was written by Jian Xu and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript. This paper has a pre-printed version at https://export.arxiv.org/abs/2303.06314.

Corresponding author

Correspondence to Shao-Lun Huang.

Ethics declarations

Conflict of Interest/Competing Interests

the authors have no competing interests to declare that are relevant to the content of this article.

Ethics Approval and Consent to Participate

Not applicable

Consent for Publication

Not applicable

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, J., Yang, M., Ding, W. et al. Stabilizing and improving federated learning with highly non-iid data and client dropout. Appl Intell 55, 216 (2025). https://doi.org/10.1007/s10489-024-05956-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-05956-3

Keywords