Abstract
Federated learning (FL) enables clients to collaboratively train a model, while keeping their local training data decentralized. However, high communication costs, data heterogeneity across clients, and lack of personalization techniques hinder the development of FL. In this paper, we propose FedLTN, a novel approach motivated by the well-known Lottery Ticket Hypothesis to learn sparse and personalized lottery ticket networks (LTNs) for communication-efficient and personalized FL under non-identically and independently distributed (non-IID) data settings. Preserving batch-norm statistics of local clients, postpruning without rewinding, and aggregation of LTNs using server momentum ensures that our approach significantly outperforms existing state-of-the-art solutions. Experiments on CIFAR-10 and Tiny ImageNet datasets show the efficacy of our approach in learning personalized models while significantly reducing communication costs.
V. Mugunthan and E. Lin—Equal contributions.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Alistarh, D., Grubic, D., Li, J., Tomioka, R., Vojnovic, M.: QSGD: communication-efficient SGD via gradient quantization and encoding. In: Advances in Neural Information Processing Systems 30 (2017)
Barnes, L.P., Inan, H.A., Isik, B., Özgür, A.: rTop-k: a statistical estimation approach to distributed SGD. IEEE J. Sel. Areas Inf. Theory 1(3), 897–907 (2020)
Chen, Y., Lu, W., Wang, J., Qin, X.: FedHealth 2: weighted federated transfer learning via batch normalization for personalized healthcare. arXiv preprint arXiv:2106.01009 (2021)
Chen, Y., Lu, W., Wang, J., Qin, X., Qin, T.: Federated learning with adaptive batchnorm for personalized healthcare. arXiv preprint arXiv:2112.00734 (2021)
Fallah, A., Mokhtari, A., Ozdaglar, A.: Personalized federated learning: a meta-learning approach. arXiv preprint arXiv:2002.07948 (2020)
Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 (2018)
Frankle, J., Dziugaite, G.K., Roy, D.M., Carbin, M.: Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.01611 (2019)
Hamer, J., Mohri, M., Suresh, A.T.: FedBoost: a communication-efficient algorithm for federated learning. In: International Conference on Machine Learning, pp. 3973–3983. PMLR (2020)
Hsu, T.M.H., Qi, H., Brown, M.: Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 (2019)
Idrissi, M.J., Berrada, I., Noubir, G.: FEDBS: learning on Non-IID data in federated learning using batch normalization. In: 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI), pp. 861–867. IEEE (2021)
Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S., Stich, S., Suresh, A.T.: Scaffold: stochastic controlled averaging for federated learning. In: International Conference on Machine Learning, pp. 5132–5143. PMLR (2020)
Kim, G., Kim, J., Han, B.: Communication-efficient federated learning with acceleration of global momentum. arXiv preprint arXiv:2201.03172 (2022)
Konečnỳ, J., McMahan, H.B., Yu, F.X., Richtárik, P., Suresh, A.T., Bacon, D.: Federated learning: strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 (2016)
Kopparapu, K., Lin, E.: FedFMC: sequential efficient federated learning on Non-IID data. arXiv preprint arXiv:2006.10937 (2020)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Kulkarni, V., Kulkarni, M., Pant, A.: Survey of personalization techniques for federated learning. In: 2020 Fourth World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4), pp. 794–797. IEEE (2020)
Le, Y., Yang, X.: Tiny ImageNet visual recognition challenge. CS 231N 7(7), 3 (2015)
Li, A., Sun, J., Wang, B., Duan, L., Li, S., Chen, Y., Li, H.: LotteryFL: personalized and communication-efficient federated learning with lottery ticket hypothesis on Non-IID datasets. arXiv preprint arXiv:2008.03371 (2020)
Li, X., Jiang, M., Zhang, X., Kamp, M., Dou, Q.: FedBN: federated learning on Non-IID features via local batch normalization. arXiv preprint arXiv:2102.07623 (2021)
Mansour, Y., Mohri, M., Ro, J., Suresh, A.T.: Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619 (2020)
McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282. PMLR (2017)
Morcos, A., Yu, H., Paganini, M., Tian, Y.: One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. In: Advances in Neural Information Processing Systems 32 (2019)
Ozfatura, E., Ozfatura, K., Gündüz, D.: FedADC: accelerated federated learning with drift control. In: 2021 IEEE International Symposium on Information Theory (ISIT), pp. 467–472. IEEE (2021)
Reddi, S., et al.: Adaptive federated optimization. arXiv preprint arXiv:2003.00295 (2020)
Reyes, J., Di Jorio, L., Low-Kam, C., Kersten-Oertel, M.: Precision-weighted federated learning. arXiv preprint arXiv:2107.09627 (2021)
Smith, V., Chiang, C.K., Sanjabi, M., Talwalkar, A.S.: Federated multi-task learning. In: Advances in Neural Information Processing Systems 30 (2017)
Suresh, A.T., Felix, X.Y., Kumar, S., McMahan, H.B.: Distributed mean estimation with limited communication. In: International Conference on Machine Learning, pp. 3329–3337. PMLR (2017)
Tenison, I., Sreeramadas, S.A., Mugunthan, V., Oyallon, E., Belilovsky, E., Rish, I.: Gradient masked averaging for federated learning. arXiv preprint arXiv:2201.11986 (2022)
Wang, K., Mathews, R., Kiddon, C., Eichner, H., Beaufays, F., Ramage, D.: Federated evaluation of on-device personalization. arXiv preprint arXiv:1910.10252 (2019)
Xu, A., Huang, H.: Double momentum SGD for federated learning. arXiv preprint arXiv:2102.03970 (2021)
Xu, J., Wang, S., Wang, L., Yao, A.C.C.: FedCM: federated learning with client-level momentum. arXiv preprint arXiv:2106.10874 (2021)
Yeganeh, Y., Farshad, A., Navab, N., Albarqouni, S.: Inverse distance aggregation for federated learning with Non-IID data. In: Albarqouni, S., et al. (eds.) DART/DCL -2020. LNCS, vol. 12444, pp. 150–159. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60548-3_15
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Mugunthan, V., Lin, E., Gokul, V., Lau, C., Kagal, L., Pieper, S. (2022). FedLTN: Federated Learning for Sparse and Personalized Lottery Ticket Networks. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13672. Springer, Cham. https://doi.org/10.1007/978-3-031-19775-8_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-19775-8_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-19774-1
Online ISBN: 978-3-031-19775-8
eBook Packages: Computer ScienceComputer Science (R0)