Skip to main content
SpringerLink
Log in

Layer-Wise Personalized Federated Learning with Hypernetwork

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Federated learning is a machine learning paradigm in which decentralized client devices collaboratively train shared model under the coordinating of a central server without sharing local data. Data heterogeneity is one of the major challenges faced by federated learning, which leading to lagged convergence and decreased accuracy of the global model. Moreover, existing personalized federated learning (PFL) methods suffer from limited personalization capabilities and slow model convergence. To address these limitations, we propose a novel Hypernetwork based layer-wise PFL algorithm, pFedLHN. The Hypernetwork located on the server is collaboratively trained by the clients to generate unique personalized models based on their local data distribution. Unlike previous PFL algorithms that focus on optimizing and personalizing the whole shared model, we implement layer-wise modules with parameter sharing mechanisms in the Hypernetwork. This approach allows the Hypernetwork to generate parameters independently for each layer of the client’s network, enabling the layer-wise modules to focus on learning the different utilities in the network. Consequently, this enhances the model’s personalized ability and training efficiency. We conducted several experiments on various PFL benchmark datasets, demonstrating that pFedLHN outperforms previous methods and has better generalization ability on clients that did not participate in the training phase.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5

Similar content being viewed by others

Code availability

Code available on: https://github.com/tianyuzeke/pFedLHN

References

  1. McMahan B, Moore E, Ramage D, Hampson S, Arcas BA (2017) Communication-efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics. PMLR, pp 1273–1282

  2. Kairouz P, McMahan HB, Avent B, Bellet A, Bennis M, Bhagoji AN, Bonawitz K, Charles Z, Cormode G, Cummings R et al (2021) Advances and open problems in federated learning. Found Trends® Mach Learn 14(1–2):1–210

    Article  Google Scholar 

  3. Mothukuri V, Parizi RM, Pouriyeh S, Huang Y, Dehghantanha A, Srivastava G (2021) A survey on security and privacy of federated learning. Futur Gener Comput Syst 115:619–640

    Article  Google Scholar 

  4. Zhao Y, Li M, Lai L, Suda N, Civin D, Chandra V (2018) Federated learning with non-iid data. arXiv preprint arXiv:1806.00582

  5. Shaheen M, Farooq MS, Umer T, Kim B-S (2022) Applications of federated learning; taxonomy, challenges, and research trends. Electronics 11(4):670

    Article  Google Scholar 

  6. Tan AZ, Yu H, Cui L, Yang Q (2022) Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems

  7. Hsu T-MH, Qi H, Brown M (2019) Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335

  8. Wang J, Charles Z, Xu Z, Joshi G, McMahan HB, Al-Shedivat M, Andrew G, Avestimehr S, Daly K, Data D et al (2021) A field guide to federated optimization. arXiv preprint arXiv:2107.06917

  9. Caldas S, Duddu SMK, Wu P, Li T, Konečnỳ J, McMahan HB, Smith V, Talwalkar A (2018) Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097

  10. Chen X, Chen T, Sun H, Wu SZ, Hong M (2020) Distributed training with heterogeneous data: bridging median-and mean-based algorithms. Adv Neural Inf Process Syst 33:21616–21626

    Google Scholar 

  11. Liang PP, Liu T, Ziyin L, Allen NB, Auerbach RP, Brent D, Salakhutdinov R, Morency L-P (2020) Think locally, act globally: federated learning with local and global representations. arXiv preprint arXiv:2001.01523

  12. Yao X, Sun L (2020) Continual local training for better initialization of federated models. In: 2020 IEEE International conference on image processing (ICIP). IEEE, pp 1736–1740

  13. Shoham N, Avidor T, Keren A, Israel N, Benditkis D, Mor-Yosef L, Zeitak I (2019) Overcoming forgetting in federated learning on non-IID data. arXiv preprint arXiv:1910.07796

  14. Shamsian A, Navon A, Fetaya E, Chechik G (2021) Personalized federated learning using hypernetworks. In: International conference on machine learning. PMLR, pp 9489–9502

  15. Wang K, Mathews R, Kiddon C, Eichner H, Beaufays F, Ramage D (2019) Federated evaluation of on-device personalization. arXiv preprint arXiv:1910.10252

  16. Chen F, Luo M, Dong Z, Li Z, He X (2018) Federated meta-learning with fast convergence and efficient communication. arXiv preprint arXiv:1802.07876

  17. Zhang M, Sapra K, Fidler S, Yeung S, Alvarez JM (2020) Personalized federated learning with first order model optimization. arXiv preprint arXiv:2012.08565

  18. Yang H, He H, Zhang W, Cao X (2020) FedSteg: a federated transfer learning framework for secure image steganalysis. IEEE Trans Netw Sci Eng 8(2):1084–1094

    Article  Google Scholar 

  19. Chen Y, Qin X, Wang J, Yu C, Gao W (2020) FedHealth: a federated transfer learning framework for wearable healthcare. IEEE Intell Syst 35(4):83–93

    Article  Google Scholar 

  20. Huang Y, Chu L, Zhou Z, Wang L, Liu J, Pei J, Zhang Y (2021) Personalized cross-silo federated learning on non-IID data. In: Proceedings of the AAAI conference on artificial intelligence, vol 35. pp 7865–7873

  21. Li T, Hu S, Beirami A, Smith V (2021) Ditto: fair and robust federated learning through personalization. In: International conference on machine learning. PMLR, pp 6357–6368

  22. Kopparapu K, Lin E, Zhao J (2020) FedCD: Improving performance in non-IID federated learning. arXiv preprint arXiv:2006.09637

  23. Collins L, Hassani H, Mokhtari A, Shakkottai S (2021) Exploiting shared representations for personalized federated learning. In: International conference on machine learning. PMLR, pp 2089–2099

  24. Bui D, Malik K, Goetz J, Liu H, Moon S, Kumar A, Shin KG (2019) Federated user representation learning. arXiv preprint arXiv:1909.12535

  25. Yu T, Bagdasaryan E, Shmatikov V (2020) Salvaging federated learning by local adaptation. arXiv preprint arXiv:2002.04758

  26. Mansour Y, Mohri M, Ro J, Suresh AT (2020) Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619

  27. Sharma S, Xing C, Liu Y, Kang Y (2019) Secure and efficient federated transfer learning. In: 2019 IEEE International conference on big data (Big Data). IEEE, pp 2569–2576

  28. Li D, Wang J (2019) Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581

  29. Finn C, Abbeel P, Levine S (2017) Model-agnostic meta-learning for fast adaptation of deep networks. In: International conference on machine learning. PMLR, pp 1126–1135

  30. Fallah A, Mokhtari A, Ozdaglar A (2020) Personalized federated learning with theoretical guarantees: a model-agnostic meta-learning approach. Adv Neural Inf Process Syst 33:3557–3568

    Google Scholar 

  31. T Dinh C, Tran N, Nguyen J (2020) Personalized federated learning with moreau envelopes. Adv Neural Inf Process Syst 33:21394–21405

    Google Scholar 

  32. Ghosh A, Chung J, Yin D, Ramchandran K (2020) An efficient framework for clustered federated learning. Adv Neural Inf Process Syst 33:19586–19597

    Google Scholar 

  33. Duan M, Liu D, Ji X, Liu R, Liang L, Chen X, Tan Y (2021) Fedgroup: efficient federated learning via decomposed similarity-based clustering. In: 2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom), IEEE. pp 228–237

  34. Arivazhagan MG, Aggarwal V, Singh AK, Choudhary S (2019) Federated learning with personalization layers. arXiv preprint arXiv:1912.00818

  35. Marfoq O, Neglia G, Bellet A, Kameni L, Vidal R (2021) Federated multi-task learning under a mixture of distributions. Adv Neural Inf Process Syst 34:15434–15447

    Google Scholar 

  36. Smith V, Chiang C-K, Sanjabi M, Talwalkar AS (2017) Federated multi-task learning. In: Advances in neural information processing systems, vol. 30

  37. Ha D, Dai A, Le QV (2016) Hypernetworks. arXiv preprint arXiv:1609.09106

  38. Sendera M, Przewięźlikowski M, Karanowski K, Zięba M, Tabor J, Spurek P (2023) Hypershot: few-shot learning by kernel hypernetworks. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 2469–2478

  39. Ma X, Zhang J, Guo S, Xu W (2022) Layer-wised model aggregation for personalized federated learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10092–10101

  40. Li T, Sahu AK, Zaheer M, Sanjabi M, Talwalkar A, Smith V (2020) Federated optimization in heterogeneous networks. Proc Mach Learn Syst 2:429–450

    Google Scholar 

  41. Karimireddy SP, Kale S, Mohri M, Reddi S, Stich S, Suresh AT (2020) Scaffold: stochastic controlled averaging for federated learning. In: International conference on machine learning. PMLR, pp 5132–5143

  42. Lee S, Zhang T, He C, Avestimehr S (2021) Layer-wise adaptive model aggregation for scalable federated learning. arXiv preprint arXiv:2110.10302

  43. Yu F, Wang D, Shelhamer E, Darrell T (2018) Deep layer aggregation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2403–2412

Download references

Acknowledgements

This work was supported by the Natural Science Foundation of Heilongjiang Province (Grant No. LH2021F032) and the Key Research and Development Projects of Heilongjiang Province (Grant No. 2022ZX01A34).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Harbin University of Science and Technology, Harbin, 150080, China

    Suxia Zhu, Tianyu Liu & Guanglu Sun

  2. Research Center of Information Security and Intelligent Technology, Harbin University of Science and Technology, Harbin, 150080, China

    Suxia Zhu, Tianyu Liu & Guanglu Sun

Authors
  1. Suxia Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tianyu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guanglu Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Suxia Zhu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest to report regarding the present study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, S., Liu, T. & Sun, G. Layer-Wise Personalized Federated Learning with Hypernetwork. Neural Process Lett 55, 12273–12287 (2023). https://doi.org/10.1007/s11063-023-11420-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11420-z

Keywords

Search

Navigation