Skip to main content
SpringerLink
Log in

Personalization in Federated Learning

  • Chapter
  • First Online:
Federated Learning

Abstract

Typical federated learning (FL) problem formulation requires learning a single model suitable for all parties while prohibiting parties from sharing their data with the aggregator. However, it may not be possible to learn a common single model that is suitable for all parties. For example, consider a sentence completion problem: “I live in the state of …” The answer clearly depends on the party, and no single model is appropriate here. To handle such situations, various personalization strategies have been proposed in the recent literature. In particular, the problem appears to have a close connection to meta-learning. We review recent FL personalization techniques categorizing them into eight groups and summarize three strategies and corresponding datasets for benchmarking personalization in federated learning. We provide an overview of the statistical challenges of personalization in federated learning. At a high level, personalization leads to an increase in the model complexity, which in turn increases the hardness of the federated learning task. We study when too much personalization can prevent standard approaches to personalized federated learning from learning the common parts of the parties and present alternative approaches that overcome such issues.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://yann.lecun.com/exdb/mnist/

  2. 2.

    http://yann.lecun.com/exdb/mnist/

  3. 3.

    http://www.gutenberg.org/ebooks/100

  4. 4.

    https://github.com/cvdfoundation/google-landmark

References

  1. Amir S, Wallace BC, Lyu H, Silva PCMJ (2016) Modelling context with user embeddings for sarcasm detection in social media. arXiv preprint arXiv:160700976

    Google Scholar 

  2. Ammad-Ud-Din M, Ivannikova E, Khan SA, Oyomno W, Fu Q, Tan KE, Flanagan A (2019) Federated collaborative filtering for privacy-preserving personalized recommendation system. arXiv preprint arXiv:190109888

    Google Scholar 

  3. Arivazhagan MG, Aggarwal V, Singh AK, Choudhary S (2019) Federated learning with personalization layers. arXiv preprint arXiv:191200818

    Google Scholar 

  4. Baxter J (2000) A model of inductive bias learning. J Artif Intell Res 12:149–198

    Article  MathSciNet  Google Scholar 

  5. Bui D, Malik K, Goetz J, Liu H, Moon S, Kumar A, Shin KG (2019) Federated user representation learning. arXiv preprint arXiv:190912535

    Google Scholar 

  6. Caldas S, Duddu SMK, Wu P, Li T, Konečnỳ J, McMahan HB, Smith V, Talwalkar A (2018) Leaf: a benchmark for federated settings. arXiv preprint arXiv:181201097

    Google Scholar 

  7. Caruana R (1997) Multitask learning. Mach Learn 28(1):41–75

    Article  MathSciNet  Google Scholar 

  8. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) Smote: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357

    Article  Google Scholar 

  9. Chen F, Luo M, Dong Z, Li Z, He X (2018) Federated meta-learning with fast convergence and efficient communication. arXiv preprint arXiv:180207876

    Google Scholar 

  10. Chen M, Suresh AT, Mathews R, Wong A, Allauzen C, Beaufays F, Riley M (2019) Federated learning of n-gram language models. In: Proceedings of the 23rd conference on computational natural language learning (CoNLL), pp 121–130

    Google Scholar 

  11. Cheng Y, Wang D, Zhou P, Zhang T (2017) A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:171009282

    Google Scholar 

  12. Chernozhukov V, Chetverikov D, Demirer M, Duflo E, Hansen C, Newey W, Robins J (2017) Double/debiased machine learning for treatment and causal parameters. arXiv:160800060 [econ, stat] 1608.00060

    Google Scholar 

  13. Cohen G, Afshar S, Tapson J, Van Schaik A (2017) EMNIST: extending MNIST to handwritten letters. In: 2017 international joint conference on neural networks (IJCNN). IEEE, pp 2921–2926

    Google Scholar 

  14. Corinzia L, Beuret, A, Buhmann JM (2019) Variational federated multi-task learning. arXiv preprint arXiv:190606268

    Google Scholar 

  15. Deng Y, Kamani MM, Mahdavi M (2020) Adaptive personalized federated learning. arXiv preprint arXiv:200313461

    Google Scholar 

  16. Dinh CT, Tran NH, Nguyen TD (2020) Personalized federated learning with Moreau envelopes. arXiv preprint arXiv:200608848

    Google Scholar 

  17. Fallah A, Mokhtari A, Ozdaglar A (2020) Personalized federated learning: a meta-learning approach. arXiv preprint arXiv:200207948

    Google Scholar 

  18. Finn C, Abbeel P, Levine S (2017) Model-agnostic meta-learning for fast adaptation of deep networks. In: International conference on machine learning. PMLR, pp 1126–1135

    Google Scholar 

  19. Ghosh A, Hong J, Yin D, Ramchandran K (2019) Robust federated learning in a heterogeneous environment. arXiv preprint arXiv:190606629

    Google Scholar 

  20. Ghosh A, Chung J, Yin D, Ramchandran K (2020) An efficient framework for clustered federated learning. arXiv preprint arXiv:200604088

    Google Scholar 

  21. Gou J, Yu B, Maybank SJ, Tao D (2020) Knowledge distillation: a survey. arXiv preprint arXiv:200605525

    Google Scholar 

  22. Grbovic M, Cheng H (2018) Real-time personalization using embeddings for search ranking at Airbnb. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp 311–320

    Google Scholar 

  23. Hanzely F, Richtárik P (2020) Federated learning of a mixture of global and local models. arXiv preprint arXiv:200205516

    Google Scholar 

  24. Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:150302531

    Google Scholar 

  25. Hospedales T, Antoniou A, Micaelli P, Storkey A (2020) Meta-learning in neural networks: a survey. arXiv preprint arXiv:200405439

    Google Scholar 

  26. Huang Y, Chu L, Zhou Z, Wang L, Liu J, Pei J, Zhang Y (2021) Personalized cross-silo federated learning on non-IID data. In: Proceedings of the AAAI conference on artificial intelligence, vol 35, pp 7865–7873

    Google Scholar 

  27. Jaech A, Ostendorf M (2018) Personalized language model for query auto-completion. arXiv preprint arXiv:180409661

    Google Scholar 

  28. Jeong E, Oh S, Kim H, Park J, Bennis M, Kim SL (2018) Communication-efficient on-device machine learning: federated distillation and augmentation under non-IID private data. arXiv preprint arXiv:181111479

    Google Scholar 

  29. Jiang Y, Konečnỳ J, Rush K, Kannan S (2019) Improving federated learning personalization via model agnostic meta learning. arXiv preprint arXiv:190912488

    Google Scholar 

  30. Johnson AE, Pollard TJ, Shen L, Li-Wei HL, Feng M, Ghassemi M, Moody B, Szolovits P, Celi LA, Mark RG (2016) MIMIC-III, a freely accessible critical care database. Sci Data 3(1):1–9

    Article  Google Scholar 

  31. Khodak M, Balcan MF, Talwalkar A (2019) Adaptive gradient-based meta-learning methods. arXiv preprint arXiv:190602717

    Google Scholar 

  32. Krizhevsky A, Hinton G et al (2009) Learning multiple layers of features from tiny images

    Google Scholar 

  33. Lancaster T (2000) The incidental parameter problem since 1948. J Econ 95(2):391–413. https://doi.org/10.1016/S0304-4076(99)00044-5

    Article  MathSciNet  Google Scholar 

  34. Li D, Wang J (2019) FedMD: Heterogenous federated learning via model distillation. arXiv preprint arXiv:191003581

    Google Scholar 

  35. Liang PP, Liu T, Ziyin L, Allen NB, Auerbach RP, Brent D, Salakhutdinov R, Morency LP (2020) Think locally, act globally: federated learning with local and global representations. arXiv preprint arXiv:200101523

    Google Scholar 

  36. Mansour Y, Mohri M, Ro J, Suresh AT (2020) Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:200210619

    Google Scholar 

  37. Mariani G, Scheidegger F, Istrate R, Bekas C, Malossi C (2018) BAGAN: data augmentation with balancing GAN. arXiv preprint arXiv:180309655

    Google Scholar 

  38. McGraw I, Prabhavalkar R, Alvarez R, Arenas MG, Rao K, Rybach D, Alsharif O, Sak H, Gruenstein A, Beaufays F et al (2016) Personalized speech recognition on mobile devices. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 5955–5959

    Google Scholar 

  39. McMahan B, Moore E, Ramage D, Hampson S, y Arcas BA (2017) Communication-efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics. PMLR, pp 1273–1282

    Google Scholar 

  40. Mikołajczyk A, Grochowski M (2018) Data augmentation for improving deep learning in image classification problem. In: 2018 international interdisciplinary PhD workshop (IIPhDW). IEEE, pp 117–122

    Google Scholar 

  41. Minka TP (2013) Expectation propagation for approximate Bayesian inference. arXiv preprint arXiv:13012294

    Google Scholar 

  42. Neyman J (1979) C(α) tests and their use. Sankhyā: Indian J Stat Ser A (1961–2002) 41(1/2):1–21

    MathSciNet  MATH  Google Scholar 

  43. Nichol A, Achiam J, Schulman J (2018) On first-order meta-learning algorithms. arXiv:180302999 [cs] 1803.02999

    Google Scholar 

  44. Peterson D, Kanani P, Marathe VJ (2019) Private federated learning with domain adaptation. arXiv preprint arXiv:191206733

    Google Scholar 

  45. Raghu A, Raghu M, Bengio S, Vinyals O (2019) Rapid learning or feature reuse? Towards understanding the effectiveness of MAML. In: International conference on learning representations

    Google Scholar 

  46. Ruder S (2017) An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:170605098

    Google Scholar 

  47. Sattler F, Müller KR, Samek W (2020) Clustered federated learning: model-agnostic distributed multitask optimization under privacy constraints. IEEE Trans Neural Netw Learn Syst 1–13. https://doi.org/10.1109/TNNLS.2020.3015958

  48. Schafer JB, Frankowski D, Herlocker J, Sen S (2007) Collaborative filtering recommender systems. In: Brusilovsky P, Kobsa A, Nejdl W (eds) The adaptive web: methods and strategies of web personalization. Springer, pp 291–324. https://doi.org/10.1007/978-3-540-72079-9_9

  49. Shamir O, Srebro N, Zhang T (2014) Communication-efficient distributed optimization using an approximate Newton-type method. In: International conference on machine learning. PMLR, pp 1000–1008

    Google Scholar 

  50. Shen T, Zhang J, Jia X, Zhang F, Huang G, Zhou P, Wu F, Wu C (2020) Federated mutual learning. arXiv preprint arXiv:200616765

    Google Scholar 

  51. Shin M, Hwang C, Kim J, Park J, Bennis M, Kim SL (2020) XOR mixup: privacy-preserving data augmentation for one-shot federated learning. arXiv preprint arXiv:200605148

    Google Scholar 

  52. Shorten C, Khoshgoftaar TM (2019) A survey on image data augmentation for deep learning. J Big Data 6(1):1–48

    Article  Google Scholar 

  53. Smith V, Chiang CK, Sanjabi M, Talwalkar A (2017) Federated multi-task learning. arXiv preprint arXiv:170510467

    Google Scholar 

  54. Thrun S (1996) Is learning the n-th thing any easier than learning the first? In: Advances in neural information processing systems. Morgan Kaufmann Publishers, San Mateo, pp 640–646

    Google Scholar 

  55. Thrun S (1998) Lifelong learning algorithms. In: Learning to learn. Springer, Boston pp 181–209

    Google Scholar 

  56. Vanschoren J (2018) Meta-learning: a survey. arXiv preprint arXiv:181003548

    Google Scholar 

  57. Vilalta R, Drissi Y (2002) A perspective view and survey of meta-learning. Artif Intell Rev 18(2):77–95

    Article  Google Scholar 

  58. Vu T, Nguyen DQ, Johnson M, Song D, Willis A (2017) Search personalization with embeddings. In: European conference on information retrieval. Springer, pp 598–604

    Google Scholar 

  59. Weng L (2018) Meta-learning: learning to learn fast. lilianwenggithubio/lil-log. http://lilianweng.github.io/lil-log/2018/11/29/meta-learning.html

  60. Weyand T, Araujo A, Cao B, Sim J (2020) Google Landmarks Dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2575–2584

    Google Scholar 

  61. Yu P, Kundu A, Wynter L, Lim SH (2021) Fed+: a unified approach to robust personalized federated learning. 2009.06303

    Google Scholar 

  62. Yu T, Bagdasaryan E, Shmatikov V (2020) Salvaging federated learning by local adaptation. arXiv preprint arXiv:200204758

    Google Scholar 

  63. Yuksel SE, Wilson JN, Gader PD (2012) Twenty years of mixture of experts. IEEE Trans Neural Netw Learn Syst 23(8):1177–1193

    Article  Google Scholar 

  64. Yurochkin M, Agarwal M, Ghosh S, Greenewald K, Hoang N, Khazaeni Y (2019) Bayesian nonparametric federated learning of neural networks. In: International conference on machine learning. PMLR, pp 7252–7261

    Google Scholar 

  65. Zhang Y, Yang Q (2017) A survey on multi-task learning. arXiv preprint arXiv:170708114

    Google Scholar 

  66. Zhao Y, Li M, Lai L, Suda N, Civin D, Chandra V (2018) Federated learning with non-IID data. arXiv preprint arXiv:180600582

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. IBM Research, Cambridge, MA, USA

    Mayank Agarwal

  2. MIT-IBM Watson AI Lab, IBM Research, Cambridge, MA, USA

    Mikhail Yurochkin

  3. Department of Statistics, University of Michigan, Ann Arbor, MI, USA

    Yuekai Sun

Authors
  1. Mayank Agarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mikhail Yurochkin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuekai Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mayank Agarwal .

Editor information

Editors and Affiliations

  1. IBM Research – Almaden, San Jose, CA, USA

    Heiko Ludwig

  2. IBM Research -- Almaden, San Jose, CA, USA

    Nathalie Baracaldo

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Agarwal, M., Yurochkin, M., Sun, Y. (2022). Personalization in Federated Learning. In: Ludwig, H., Baracaldo, N. (eds) Federated Learning. Springer, Cham. https://doi.org/10.1007/978-3-030-96896-0_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-96896-0_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-96895-3

  • Online ISBN: 978-3-030-96896-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation