Skip to main content
Springer Nature Link
Log in

Personalized federated learning with global information fusion and local knowledge inheritance collaboration

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Traditional federated learning has shown mediocre performance on heterogeneous data, thus sparking increasing interest in personalized federated learning. Unlike traditional federated learning, which trains a single global consensual model, personalized federated learning allows for the provision of distinct models to different clients. However, existing federated learning algorithms solely optimize either unidirectionally at the server or client side, leading to a dilemma: “Should we prioritize the learned model’s generic performance or its personalized performance?” In this paper, we demonstrate the feasibility of simultaneously addressing both aspects. Concretely, we propose a novel dual-duty framework. On the client side, personalized models are utilized to retain local knowledge and integrate global information, minimizing risks associated with each client’s experience. On the server side, virtual sample generation approximates second-order gradients, embedding local class structures into the global model to enhance its generalization capability. Utilizing a dual optimization framework termed FedCo, we achieve parallelism of global universality and personalized performance. Finally, theoretical analysis and extensive experiments validate that FedCo surpasses previous solutions, achieving state-of-the-art performance for both general and personalized performance in a variety of heterogeneous data scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Algorithm 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

No datasets were generated or analysed during the current study.

References

  1. McMahan B, Moore E, Ramage D, Hampson S, Arcas BA (2017) Communication-efficient learning of deep networks from decentralized data, pp 1273–1282. PMLR

  2. Kairouz P, McMahan HB, Avent B, Bellet A, Bennis M, Bhagoji AN, Bonawitz K, Charles Z, Cormode G, Cummings R et al (2021) Advances and open problems in federated learning. Found Trends® Mach Learn 14(1–2):1–210

    Article  Google Scholar 

  3. Huang W, Ye M, Shi Z, Wan G, Li H, Du B, Yang Q (2024) Federated learning for generalization, robustness, fairness: A survey and benchmark. In: IEEE Transactions on Pattern Analysis and Machine Intelligence

  4. Huang H, Shang F, Liu Y, Liu H (2021) Behavior mimics distribution: Combining individual and group behaviors for federated learning. arXiv preprint arXiv:2106.12300

  5. Huang W, Ye M, Shi Z, Du B (2023) Generalizable heterogeneous federated cross-correlation and instance similarity learning. In: IEEE Transactions on Pattern Analysis and Machine Intelligence

  6. Tan AZ, Yu H, Cui L, Yang Q (2022) Towards personalized federated learning. In: IEEE Transactions on Neural Networks and Learning Systems

  7. Fallah A, Mokhtari A, Ozdaglar A (2020) Personalized federated learning with theoretical guarantees: a model-agnostic meta-learning approach. Adv Neural Inf Process Syst 33:3557–3568

    Google Scholar 

  8. Jiang Y, Konečnỳ J, Rush K, Kannan S (2019) Improving federated learning personalization via model agnostic meta learning. arXiv preprint arXiv:1909.12488

  9. Lin S, Yang G, Zhang J (2020) Real-time edge intelligence in the making: a collaborative learning framework via federated meta-learning. arXiv preprint arXiv:2001.03229

  10. Li D, Wang J (2019) Fedmd: heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581

  11. Zhang J, Guo S, Ma X, Wang H, Xu W, Wu F (2021) Parameterized knowledge transfer for personalized federated learning. Adv Neural Inf Process Syst 34:10092–10104

    Google Scholar 

  12. Chen Y, Lu W, Qin X, Wang J, Xie X (2023) Metafed: federated learning among federations with cyclic knowledge distillation for personalized healthcare. In: IEEE Transactions on Neural Networks and Learning Systems

  13. Jin H, Bai D, Yao D, Dai Y, Gu L, Yu C, Sun L (2022) Personalized edge intelligence via federated self-knowledge distillation. IEEE Trans Parallel Distrib Syst 34(2):567–580

    Article  Google Scholar 

  14. Zhang J, Hua Y, Wang H, Song T, Xue Z, Ma R, Guan H (2023) Fedcp: Separating feature information for personalized federated learning via conditional policy. In: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp 3249–3261

  15. Zhi M, Bi Y, Xu W, Wang H, Xiang T (2024) Knowl-aware Parameter Coach Personal Fed Learn. In: Personalized Federated Learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 38, No. 15, pp 17069–17077

  16. Li X-C, Zhan D-C, Shao Y, Li B, Song S (2021) Fedphp: Federated personalization with inherited private models. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Cham: Springer International Publishing, pp 587–602

  17. Luo J, Mendieta M, Chen C, Wu S (2023) Pgfed: personalize each client’s global objective for federated learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 3946–3956

  18. Lindsay BG (1995) Mixture models: theory, geometry, and applications. IMS

  19. Luo M, Chen F, Hu D, Zhang Y, Liang J, Feng J (2021) No fear of heterogeneity: classifier calibration for federated learning with non-iid data. Adv Neural Inf Process Syst 34:5972–5984

    Google Scholar 

  20. Hamer J, Mohri M, Suresh AT (2020) Fedboost: a communication-efficient algorithm for federated learning. In: International Conference on Machine Learning. PMLR, pp 3973–3983

  21. Fallah A, Mokhtari A, Ozdaglar A (2020) Personalized federated learning with theoretical guarantees: a model-agnostic meta-learning approach. Adv Neural Inf Process Syst 33:3557–3568

    Google Scholar 

  22. Tinh C, Tran N, Nguyen J (2020) Personalized federated learning with Moreau envelopes. Adv Neural Inf Process Syst 33:21394–21405

    Google Scholar 

  23. Li X, Huang K, Yang W, Wang S, Zhang Z (2019) On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189

  24. Hanzely F, Hanzely S, Horváth S, Richtárik P (2020) Lower bounds and optimal algorithms for personalized federated learning. Adv Neural Inf Process Syst 33:2304–2315

    Google Scholar 

  25. Li T, Sahu AK, Zaheer M, Sanjabi M, Talwalkar A, Smith V (2020) Federated optimization in heterogeneous networks. Proc Mach Learn Syst 2:429–450

    Google Scholar 

  26. Karimireddy SP, Kale S, Mohri M, Reddi S, Stich S, Suresh AT (2020) Scaffold: stochastic controlled averaging for federated learning. In: International Conference on Machine Learning. PMLR, pp 5132–5143

  27. Li X, Jiang M, Zhang X, Kamp M, Dou Q (2021) Fedbn: federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623

  28. Wu C, Wu F, Qi T, Huang Y, Xie X (2022) Fedcl: federated contrastive learning for privacy-preserving recommendation. arXiv preprint arXiv:2204.09850

  29. Li Q, He B, Song D (2021) Model-contrastive federated learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 10713–10722

  30. Chen F, Luo M, Dong Z, Li Z, He X (2018) Federated meta-learning with fast convergence and efficient communication. arXiv preprint arXiv:1802.07876

  31. Jiang Y, Konečnỳ J, Rush K, Kannan S (2019) Improving federated learning personalization via model agnostic meta learning. arXiv preprint arXiv:1909.12488

  32. Nichol A, Schulman J (2018) Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999 2(3), 4

  33. Fallah A, Mokhtari A, Ozdaglar A (2020) Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Adv Neural Inf Process Syst 33:3557–3568

    Google Scholar 

  34. Tinh C, Tran N, Nguyen J (2020) Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems 33:21394–21405

    Google Scholar 

  35. Nichol A, Achiam J, Schulman J (2018) On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999

  36. Finn C, Abbeel P, Levine S (2017) Model-agnostic meta-learning for fast adaptation of deep networks. In: International Conference on Machine Learning. PMLR, pp 1126–1135

  37. Li D, Wang J (2019) Fedmd: heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581

  38. Lin T, Kong L, Stich SU, Jaggi M (2020) Ensemble distillation for robust model fusion in federated learning. Adv Neural Inf Process Syst 33:2351–2363

    Google Scholar 

  39. Zhu Z, Hong J, Zhou J (2021) Data-free knowledge distillation for heterogeneous federated learning. In: International Conference on Machine Learning. PMLR, pp 12878–12889

  40. Zhang L, Shen L, Ding L, Tao D, Duan L.-Y (2022) Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 10174–10183

  41. He C, Annavaram M, Avestimehr S (2020) Group knowledge transfer: federated learning of large cnns at the edge. Adv Neural Inf Process Syst 33:14068–14080

    Google Scholar 

  42. Jin H, Bai D, Yao D, Dai Y, Gu L, Yu C, Sun L (2022) Personalized edge intelligence via federated self-knowledge distillation. IEEE Trans Parallel Distrib Syst 34(2):567–580

    Article  Google Scholar 

  43. Bistritz I, Mann A, Bambos N (2020) Distributed distillation for on-device learning. Adv Neural Inf Process Syst 33:22593–22604

    Google Scholar 

  44. Chen Y, Lu W, Qin X, Wang J, Xie X (2023) Metafed: federated learning among federations with cyclic knowledge distillation for personalized healthcare. IEEE Transactions on Neural Networks and Learning Systems

  45. Xie M, Long G, Shen T, Zhou T, Wang X, Jiang J, Zhang C (2021) Multi-center federated learning. arXiv e-prints, p 2108

  46. Briggs C, Fan Z, Andras P (2020) Federated learning with hierarchical clustering of local updates to improve training on non-iid data. In: 2020 International Joint Conference on Neural Networks (IJCNN), IEEE, pp 1–9

  47. Sattler F, Müller K-R, Samek W (2020) Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Trans Neural Netw Learn Syst 32(8):3710–3722

    Article  MathSciNet  Google Scholar 

  48. Duan M, Liu D, Ji X, Liu R, Liang L, Chen X, Tan Y (2020) Fedgroup: efficient clustered federated learning via decomposed data-driven measure. arXiv preprint arXiv:2010.06870

  49. Blumenberg L, Ruggles KV (2020) Hypercluster: a flexible tool for parallelized unsupervised clustering optimization. BMC Bioinform 21:1–7

    Article  Google Scholar 

  50. Ghosh A, Chung J, Yin D, Ramchandran K (2020) An efficient framework for clustered federated learning. Adv Neural Inf Process Syst 33:19586–19597

    Google Scholar 

  51. Song X, Gao W, Yang Y, Choromanski K, Pacchiano A, Tang Y (2019) Es-maml: Simple hessian-free meta learning. arXiv preprint arXiv:1910.01215

  52. Fallah A, Mokhtari A, Ozdaglar A (2020) On the convergence theory of gradient-based model-agnostic meta-learning algorithms. In: International Conference on Artificial Intelligence and Statistics. PMLR, pp 1082–1092

  53. Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

  54. Shen T, Zhang J, Jia X, Zhang F, Huang G, Zhou P, Kuang K, Wu F, Wu C (2020) Federated mutual learning. arXiv preprint arXiv:2006.16765

  55. Gretton A, Borgwardt K, Rasch M, Schölkopf B, Smola A (2006) A kernel method for the two-sample-problem. Advances in neural information processing systems, vol 19

  56. Long M, Cao Y, Wang J, Jordan M (2015) Learning transferable features with deep adaptation networks. In: International Conference on Machine Learning, PMLR, pp 97–105

  57. Hsu T-MH, Qi H, Brown M (2019) Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335

  58. Husnoo MA, Anwar A, Hosseinzadeh N, Islam SN, Mahmood AN, Doss R (2022) Fedrep: towards horizontal federated load forecasting for retail energy providers. In: 2022 IEEE PES 14th Asia-Pacific Power and Energy Engineering Conference (APPEEC). IEEE, pp 1–6

  59. Chen H-Y, Chao W-L (2021) On bridging generic and personalized federated learning for image classification. arXiv preprint arXiv:2107.00778

  60. Shi J, Zheng S, Yin X, Lu Y, Xie Y, Qu Y (2023) Clip-guided federated learning on heterogeneous and long-tailed data. arXiv preprint arXiv:2312.08648

Download references

Author information

Authors and Affiliations

  1. Department of computer science and engineering, Shanghai university of Electric Power, No. 1851, Hucheng Ring Road, Pudong New District, Shanghai, 200120, China

    Hongjiao Li, Jiayi Xu, Ming Jin & Anyang Yin

Authors
  1. Hongjiao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiayi Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ming Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anyang Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Hongjiao Li: Conceptualization, Project administration, Writing –review & editing, Supervision, Funding acquisition. Jiayi Xu: Conceptualization, Writing – original draft& editing, Methodology, Software, Validation. Ming Jin: Data curation, Formal analysis, Software. Anyang Yin: Visualization, Software, Resources.

Corresponding author

Correspondence to Jiayi Xu.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, H., Xu, J., Jin, M. et al. Personalized federated learning with global information fusion and local knowledge inheritance collaboration. J Supercomput 81, 158 (2025). https://doi.org/10.1007/s11227-024-06529-4

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11227-024-06529-4

Keywords