Abstract
Federated learning (FL) is a novel technique in deep learning that enables clients to collaboratively train a shared model while retaining their decentralized data. However, researchers working on FL face several unique challenges, especially in the context of heterogeneity. Heterogeneity in data distributions, computational capabilities, and scenarios among clients necessitates the development of customized models and objectives in FL. Unfortunately, existing works such as FedAvg may not effectively accommodate the specific needs of each client. To address the challenges arising from heterogeneity in FL, we provide an overview of the heterogeneities in data, model, and objective (DMO). Furthermore, we propose a novel framework called federated mutual learning (FML), which enables each client to train a personalized model that accounts for the data heterogeneity (DH). A “meme model” serves as an intermediary between the personalized and global models to address model heterogeneity (MH). We introduce a knowledge distillation technique called deep mutual learning (DML) to transfer knowledge between these two models on local data. To overcome objective heterogeneity (OH), we design a shared global model that includes only certain parts, and the personalized model is task-specific and enhanced through mutual learning with the meme model. We evaluate the performance of FML in addressing DMO heterogeneities through experiments and compare it with other commonly used FL methods in similar scenarios. The results demonstrate that FML outperforms other methods and effectively addresses the DMO challenges encountered in the FL setting.
摘要
联邦学习(FL)是深度学习中的一种新技术, 可以让客户端在保留各自隐私数据的情况下协同训练模型. 然而, 由于每个客户端的数据分布、 算力和场景都不同, 联邦学习面临客户端异构环境的挑战. 现有方法(如FedAvg)无法有效满足每个客户的定制化需求. 为解决联邦学习中的异构挑战, 本文首先详述了数据、模型和目标(DMO)这3个主要异构来源, 然后提出一种新的联邦相互学习(FML)框架. 该框架使得每个客户端都能训练一个考虑到数据异构(DH)的个性化模型. 在模型异构(MH)问题上, 引入一种“模因模型”作为个性化模型与全局模型之间的中介, 并且采用深度相互学习(DML)的知识蒸馏技术在两个异构模型之间传递知识. 针对目标异构(OH)问题, 通过共享部分模型参数, 设计针对特定任务的个性化模型, 同时, 利用模因模型进行相互学习. 本研究通过实验评估了FML在应对DMO异构性方面的表现, 并与其他常见FL方法在相似场景下进行对比. 实验结果表明, FML在处理FL环境中的DMO问题的表现卓越, 优于其他方法.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data that support the findings of this study are openly available in public repositories. The MNIST dataset used in this study is publicly available and can be downloaded from the MNIST website (http://yann.lecun.com/exdb/mnist/). The CIFAR-10/100 datasets used in this study are also publicly available and can be downloaded from the CIFAR website (https://www.cs.toronto.edu/∼kriz/cifar.html).
References
Alam S, Liu LY, Yan M, et al., 2023. FedRolex: model-heterogeneous federated learning with rolling sub-model extraction. https://arxiv.org/abs/2212.01548
Chen HT, Wang YH, Xu C, et al., 2019. Data-free learning of student networks. IEEE/CVF Int Conf on Computer Vision, p.3513–3521. https://doi.org/10.1109/ICCV.2019.00361
Chen HY, Chao WL, 2022. On bridging generic and personalized federated learning for image classification. https://arxiv.org/abs/2107.00778
Corchado JM, Li WG, Bajo J, et al., 2016. Special issue on distributed computing and artificial intelligence. Front Inform Technol Electron Eng, 17(4):281–282. https://doi.org/10.1631/FITEE.DCAI2015
Gao DS, Ju C, Wei XG, et al., 2020. HHHFL: hierarchical heterogeneous horizontal federated learning for electroencephalography. https://arxiv.org/abs/1909.05784
Gao JQ, Li JQ, Shan HM, et al., 2023. Forget less, count better: a domain-incremental self-distillation learning benchmark for lifelong crowd counting. Front Inform Technol Electron Eng, 24(2):187–202. https://doi.org/10.1631/FITEE.2200380
He CY, Annavaram M, Avestimehr S, et al., 2021. FedNAS: federated deep learning via neural architecture search. https://arxiv.org/abs/2004.08546v1
Hinton G, Vinyals O, Dean J, 2015. Distilling the knowledge in a neural network. https://arxiv.org/abs/1503.02531
Jiang YH, Konečný J, Rush K, et al., 2023. Improving federated learning personalization via model agnostic meta learning. https://arxiv.org/abs/1909.12488
Kairouz P, McMahan HB, Avent B, et al., 2021. Advances and open problems in federated learning. Found Trends® Mach Learn, 14(1–2):1–210. https://doi.org/10.1561/2200000083
Khodak M, Balcan MF, Talwalkar A, 2019. Adaptive gradient-based meta-learning methods. https://arxiv.org/abs/1906.02717
Krizhevsky A, 2009. Learning Multiple Layers of Features from Tiny Images. Master Thesis, Department of Computer Science, University of Toronto, Canada.
LeCun Y, Boser B, Denker J, et al., 1989. Handwritten digit recognition with a back-propagation network. Proc 2nd Int Conf on Neural Information Processing Systems, p.396–404.
LeCun Y, Bottou L, Bengio Y, et al., 1998. Gradient-based learning applied to document recognition. Proc IEEE, 86(11):2278–2324. https://doi.org/10.1109/5.726791
Li DL, Wang JP, 2019. FedMD: heterogenous federated learning via model distillation. https://arxiv.org/abs/1910.03581
Li JH, 2018. Cyber security meets artificial intelligence: a survey. Front Inform Technol Electron Eng, 19(12):1462–1474. https://doi.org/10.1631/FITEE.1800573
Li T, Sahu AK, Zaheer M, et al., 2020. Federated optimization in heterogeneous networks. https://arxiv.org/abs/1812.06127v5
Li WH, Bilen H, 2020. Knowledge distillation for multi-task learning. Proc European Conf on Computer Vision, p.163–176. https://doi.org/10.1007/978-3-030-65414-6_13
Li X, Huang KX, Yang WH, et al., 2019. On the convergence of FedAvg on non-IID data. https://arxiv.org/abs/1907.02189
Li X, Yang WH, Wang SS, et al., 2021. Communication efficient decentralized training with multiple local updates. https://arxiv.org/abs/1910.09126v1
Lian XR, Zhang C, Zhang H, et al., 2017. Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. Proc 31st Int Conf on Neural Information Processing Systems, p.5336–5346.
Liang PP, Liu T, Liu ZY, et al., 2020. Think locally, act globally: federated learning with local and global representations. https://arxiv.org/abs/2001.01523
Lim WYB, Luong NC, Hoang DT, et al., 2020 Federated learning in mobile edge networks: a comprehensive survey. IEEE Commun Surv Tutor, 22(3):2031–2063. https://doi.org/10.1109/COMST.2020.2986024
Liu FL, Wu X, Ge S, et al., 2020. Federated learning for vision-and-language grounding problems. Proc AAAI Conf Artif Intell, 34(7):11572–11579. https://doi.org/10.1609/aaai.v34i07.6824
Liu PX, Jiang JM, Zhu GX, et al., 2022. Training time minimization for federated edge learning with optimized gradient quantization and bandwidth allocation. Front Inform Technol Electron Eng, 23(8):1247–1263. https://doi.org/10.1631/FITEE.2100538
McMahan B, Moore E, Ramage D, et al., 2017. Communication-efficient learning of deep networks from decentralized data. Proc 20th Int Conf on Artificial Intelligence and Statistics, p.1273–1282.
Padhya M, Jinwala DC, 2019. MULKASE: a novel approach for key-aggregate searchable encryption for multi-owner data. Front Inform Technol Electron Eng, 20(12):1717–1748. https://doi.org/10.1631/FITEE.1800192
Pan YH, 2017. Special issue on artificial intelligence 2.0. Front Inform Technol Electron Eng, 18(1):1–2. https://doi.org/10.1631/FITEE.1710000
Pan YH, 2018. 2018 special issue on artificial intelligence 2.0: theories and applications. Front Inform Technol Electron Eng, 19(1):1–2. https://doi.org/10.1631/FITEE.1810000
Smith V, Chiang CK, Sanjabi M, et al., 2017. Federated multi-task learning. Proc 31st Int Conf on Neural Information Processing Systems, p.4427–4437.
Wang J, Li R, Wang J, et al., 2020. Artificial intelligence and wireless communications. Front Inform Technol Electron Eng, 21(10):1413–1425. https://doi.org/10.1631/FITEE.1900527
Wang TZ, Zhu JY, Torralba A, et al., 2020. Dataset distillation. https://arxiv.org/abs/1811.10959
Wu BC, Dai XL, Zhang PZ, et al., 2019. FBNet: hardware-aware efficient ConvNet design via differentiable neural architecture search. IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.10726–10734. https://doi.org/10.1109/CVPR.2019.01099
Wu JX, Li JH, Ji XS, 2018. Security for cyberspace: challenges and opportunities. Front Inform Technol Electron Eng, 19(12):1459–1461. https://doi.org/10.1631/FITEE.1840000
Yang Q, Liu Y, Cheng Y, et al., 2019. Federated Learning. Springer, Cham, Switzerland, p.1–207.
Yu T, Bagdasaryan E, Shmatikov V, 2022. Salvaging federated learning by local adaptation. https://arxiv.org/abs/2002.04758
Zhang X, Li YC, Li WP, et al., 2022. Personalized federated learning via variational Bayesian inference. Proc Int Conf on Machine Learning, p.26293–26310.
Zhang Y, Xiang T, Hospedales TM, et al., 2018. Deep mutual learning. IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.4320–4328. https://doi.org/10.1109/CVPR.2018.00454
Zhao Y, Li M, Lai LZ, et al., 2022. Federated learning with non-IID data. https://arxiv.org/abs/1806.00582
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Tao SHEN, Fengda ZHANG, and Chao WU proposed the motivation of the study. Tao SHEN, Jie ZHANG, and Xinkang JIA designed the method. Tao SHEN, Jie ZHANG, and Zheqi LV performed the experiments. Tao SHEN drafted the paper. All authors commented on previous versions of the paper. Kun KUANG, Chao WU, and Fei WU revised the paper. All authors read and approved the final paper.
Corresponding authors
Ethics declarations
Fei WU is an editorial board member of Frontiers of Information Technology & Electronic Engineering. Tao SHEN, Jie ZHANG, Xinkang JIA, Fengda ZHANG, Zheqi LV, Kun KUANG, Chao WU, and Fei WU declare that they have no conflict of interest.
Additional information
Project supported by the National Natural Science Foundation of China (Nos. U20A20387, 62006207, and 62037001), the Young Elite Scientists Sponsorship Program by China Association for Science and Technology (No. 2021QNRC001), the Zhejiang Provincial Natural Science Foundation, China (No. LQ21F020020), the Project by Shanghai AI Laboratory, China (No. P22KS00111), the Program of Zhejiang Province Science and Technology (No. 2022C01044), the StarryNight Science Fund of Zhejiang University Shanghai Institute for Advanced Study, China (No. SN-ZJU-SIAS-0010), and the Fundamental Research Funds for the Central Universities, China (Nos. 226-2022-00142 and 226-2022-00051)
Rights and permissions
About this article
Cite this article
Shen, T., Zhang, J., Jia, X. et al. Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives. Front Inform Technol Electron Eng 24, 1390–1402 (2023). https://doi.org/10.1631/FITEE.2300098
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2300098