Abstract
Clustered Federated Learning (CFL) leverages the differences among data distributions on clients to partition all clients into several clusters for personalized federated training. Compared with the conventional federated algorithms such as FedAvg, existing methods for CFL require either more communication costs or multi-stage computation overheads. In this paper, we propose an iterative CFL framework with almost the same communication cost as FedAvg in each round based on a novel model distance. Specifically, the model distance measures the discrepancy between the client model and the cluster model so that we can estimate the cluster identities for clients on the server side. The proposed model distance considers class-wise model dissimilarity, which enables us to apply it to multi-class classification even when the labels are non-iid across clients. To calculate the proposed model distance, we introduce two sampling methods which generate samples from feature distributions approximately without accessing the raw dataset. Experimental results show that our method can achieve superior and comparable performance on non-iid and iid data respectively with less communication cost compared with the baselines.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availibility
The datasets used in the study are publicly available from their corresponding authors.
Code availability
The code for this study are not publicly available until the paper is published.
References
Chen, H., Tino, P., Rodan, A., et al. (2013). Learning in the model space for cognitive fault diagnosis. IEEE Transactions on Neural Networks and Learning Systems, 25(1), 124–136.
Fu, Y., Liu, X., & Tang, S., et.al. (2021) Cic-fl: Enabling class imbalance-aware clustered federated learning over shifted distributions. In International conference on database systems for advanced applications (pp. 37–52). Springer.
Ghosh, A., Hong, J., & Yin, D. (2019). Robust federated learning in a heterogeneous environment. arXiv preprint arXiv:1906.06629
Ghosh, A., Chung, J., & Yin, D., et al. (2020). An efficient framework for clustered federated learning. arXiv preprint arXiv:2006.04088
Huang, Y., Chu, L., Zhou, Z., et al. (2021). Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI conference on artificial intelligence (pp. 7865–7873).
Kairouz, P., McMahan, H. B., Avent, B., et al. (2019). Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977
Karimireddy, S. P., Kale, S., & Mohri, M. (2020). Scaffold: Stochastic controlled averaging for federated learning. In ICML, PMLR (pp. 5132–5143).
Kingma, D. P., & Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114
Li, T., & Sahu, A. K. (2020). Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3), 50–60.
Li, X., Huang, K., & Yang, W., et.al. (2019). On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189
Lin, T., Kong, L., Stich, S. U., et al. (2020). Ensemble distillation for robust model fusion in federated learning. NIPS, 33, 2351–2363.
Liu, B., Guo, Y., & Chen, X. (2021). Pfa: Privacy-preserving federated adaptation for effective model personalization. In Proceedings of the web conference (vol. 2021, pp. 923–934).
Mavi, A. (2020) A new dataset and proposed convolutional neural network architecture for classification of american sign language digits. arXiv preprint arXiv:2011.08927
McMahan, B., Moore, E., & Ramage, D. (2017). Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, PMLR (pp. 1273–1282).
Sattler, F., & Müller, K. R. (2020). Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 32(8), 3710–3722.
Smith, V., Chiang, C. K., Sanjabi, M., et al. (2017). Federated multi-task learning. arXiv preprint arXiv:1705.10467
Wang, H., Yurochkin, M. (2020). Federated learning with matched averaging. arXiv preprint arXiv:2002.06440
Wang, K., Mathews, R., Kiddon, C., et al. (2019). Federated evaluation of on-device personalization. arXiv preprint arXiv:1910.10252
Zhang, M., Sapra, K., Fidler, S., et.al. (2020). Personalized federated learning with first order model optimization. arXiv preprint arXiv:2012.08565
Zhu, Z., Hong, J., Zhou, J. (2021). Data-free knowledge distillation for heterogeneous federated learning. In ICML, PMLR (pp. 12878–12889).
Funding
This research was supported by the National Natural Science Foundation of China (Grant no. 62276245), and Anhui Provincial Natural Science Foundation (Grant no. 2008085J31).
Author information
Authors and Affiliations
Contributions
MZ is first author. LX is corresponding author. TZ, YC CB, HC and DJ are co-authors.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethics approval
Not applicable.
Consent to participate
The authors agree to participate.
Consent for publication
The authors agree to the publication of the data and images in this paper.
Additional information
Editor: Vu Nguyen, Dani Yogatama.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhang, M., Zhang, T., Cheng, Y. et al. Communication-efficient clustered federated learning via model distance. Mach Learn 113, 3869–3888 (2024). https://doi.org/10.1007/s10994-023-06443-5
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10994-023-06443-5