Lei Yang
Abstract
Personalized federated learning (PFL) has emerged as a paradigm to provide a personalized model that can fit the local data distribution of each client. One natural choice for PFL is to leverage the fast adaptation capability of meta-learning, where it first obtains a single global model, and each client achieves a personalized model by fine-tuning the global one with its local data. However, existing meta-learning-based approaches implicitly assume that the data distribution among different clients is similar, which may not be applicable due to the property of data heterogeneity in federated learning. In this work, we propose a Group-based Federated Meta-Learning framework, called G-FML, which adaptively divides the clients into groups based on the similarity of their data distribution, and the personalized models are obtained with meta-learning within each group. In particular, we develop a simple yet effective grouping mechanism to adaptively partition the clients into multiple groups. Our mechanism ensures that each group is formed by the clients with similar data distribution such that the group-wise meta-model can achieve “personalization” at large. By doing so, our framework can be generalized to a highly heterogeneous environment. We evaluate the effectiveness of our proposed G-FML framework on three heterogeneous benchmarking datasets. The experimental results show that our framework improves the model accuracy by up to 13.15% relative to the state-of-the-art federated meta-learning.
- [1] . 2017. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. 1273–1282.Google Scholar
- [2] . 2018. Federated learning with non-IID data. arXiv:1806.00582. Retrieved from https://arxiv.org/abs/1806.00582.Google Scholar
- [3] . 2020. On the convergence of FedAvg on non-IID data. In Proceedings of the 8th International Conference on Learning Representations.Google Scholar
- [4] . 2021. FedGNN: Federated graph neural network for privacy-preserving recommendation. arXiv:2102.04925. Retrieved from https://arxiv.org/abs/2102.04925.Google Scholar
- [5] . 2017. Federated multi-task learning. In Proceedings of the Advances in Neural Information Processing Systems. 4424–4434.Google Scholar
- [6] . 2020. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Proceedings of the Advances in Neural Information Processing Systems. 3557–3568.Google Scholar
- [7] . 2020. Adaptive personalized federated learning. arXiv:2003.13461. Retrieved from https://arxiv.org/abs/2003.13461.Google Scholar
- [8] . 2020. Personalized federated learning with moreau envelopes. In Proceedings of the Advances in Neural Information Processing Systems. 21394–21405.Google Scholar
- [9] . 2021. Personalized cross-silo federated learning on non-IID data. In Proceedings of the AAAI Conference on Artificial Intelligence. 7865–7873.Google ScholarCross Ref
- [10] . 2018. Federated meta-learning for recommendation. arXiv:1802.07876. Retrieved from https://arxiv.org/abs/1802.07876.Google Scholar
- [11] . 2019. Improving federated learning personalization via model agnostic meta learning. arXiv:1909.12488. Retrieved from https://arxiv.org/abs/1909.12488.Google Scholar
- [12] . 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning. JMLR.org, 1126–1135.Google ScholarDigital Library
- [13] . 2018. On first-order meta-learning algorithms. arXiv:1803.02999. Retrieved from https://arxiv.org/abs/1803.02999.Google Scholar
- [14] . 2019. Federated learning with personalization layers. arXiv:1912.00818. Retrieved from https://arxiv.org/abs/1912.00818.Google Scholar
- [15] . 2021. Personalized federated learning using hypernetworks. In Proceedings of the 38th International Conference on Machine Learning. PMLR, 9489–9502.Google Scholar
- [16] . 2021. Exploiting shared representations for personalized federated learning. In Proceedings of the 38th International Conference on Machine Learning. PMLR, 2089–2099.Google Scholar
- [17] . 2019. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology 10, 2 (2019), 1–19.Google ScholarDigital Library
- [18] . 2020. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine 37, 3 (2020), 50–60.Google ScholarCross Ref
- [19] . 2021. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning 14, 1–2 (2021), 1–210.Google ScholarDigital Library
- [20] . 2021. A survey on security and privacy of federated learning. Future Generation Computer Systems 115 (2021), 619–640.Google ScholarCross Ref
- [21] . 2020. Survey of personalization techniques for federated learning. In Proceedings of the 2020 4th World Conference on Smart Trends in Systems, Security, and Sustainability. IEEE, 794–797.Google ScholarCross Ref
- [22] . 2022. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems (2022).Google ScholarCross Ref
- [23] . 2018. Communication-efficient on-device machine learning: Federated distillation and augmentation under non-IID private data. arXiv:1811.11479. Retrieved from https://arxiv.org/abs/1811.11479.Google Scholar
- [24] . 2021. Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems 32, 1 (2021), 59–71.Google ScholarDigital Library
- [25] . 2020. Optimizing federated learning on non-IID data with reinforcement learning. In Proceedings of the IEEE INFOCOM. IEEE, 1698–1707.Google ScholarDigital Library
- [26] . 2021. Federated learning with class imbalance reduction. In Proceedings of the 29th European Signal Processing Conference. IEEE, 2174–2178.Google ScholarCross Ref
- [27] . 2021. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems 32, 8 (2021), 3710–3722.Google ScholarCross Ref
- [28] . 2020. An efficient framework for clustered federated learning. In Proceedings of the Advances in Neural Information Processing Systems. 19586–19597.Google Scholar
- [29] . 2020. Federated learning with hierarchical clustering of local updates to improve training on non-IID data. In Proceedings of the 2020 International Joint Conference on Neural Networks. IEEE, 1–9.Google ScholarCross Ref
- [30] . 2021. FedGroup: Efficient federated learning via decomposed similarity-based clustering. In Proceedings of the IEEE International Conference on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking.228–237.Google Scholar
- [31] . 2019. FedMD: Heterogenous federated learning via model distillation. arXiv:1910.03581. Retrieved from https://arxiv.org/abs/1910.03581.Google Scholar
- [32] . 2020. FedHealth: A federated transfer learning framework for wearable healthcare. IEEE Intelligent Systems 35, 4 (2020), 83–93.Google ScholarCross Ref
- [33] . 2021. Multi-center federated learning. arXiv:2108.08647. Retrieved from https://arxiv.org/abs/2108.08647.Google Scholar
- [34] . 2019. Federated user representation learning. arXiv:1909.12535. Retrieved from https://arxiv.org/abs/1909.12535.Google Scholar
- [35] . 2020. Think locally, act globally: Federated learning with local and global representations. arXiv:2001.01523. Retrieved from https://arxiv.org/abs/2001.01523.Google Scholar
- [36] . 2020. Meta-learning in neural networks: A survey. arXiv:2004.05439. Retrieved from https://arxiv.org/abs/2004.05439.Google Scholar
- [37] . 2020. Learning from very few samples: A survey. arXiv:2009.02653. Retrieved from https://arxiv.org/abs/2009.02653.Google Scholar
- [38] . 2020. A collaborative learning framework via federated meta-learning. In Proceedings of the 40th IEEE International Conference on Distributed Computing Systems. IEEE, 289–299.Google ScholarCross Ref
- [39] . 2019. Adaptive gradient-based meta-learning methods. In Proceedings of the Advances in Neural Information Processing Systems. 5915–5926.Google Scholar
- [40] . 2019. Multimodal model-agnostic meta-learning via task-aware modulation. In Proceedings of the Advances in Neural Information Processing Systems. 1–12.Google Scholar
- [41] . 2019. Hierarchically structured meta-learning. In Proceedings of the 36th International Conference on Machine Learning. PMLR, 7045–7054.Google Scholar
- [42] . 2014. Distributed representations of sentences and documents. In Proceedings of the 31th International Conference on Machine Learning. JMLR.org, 1188–1196.Google ScholarDigital Library
- [43] . 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, 3980–3990.Google ScholarCross Ref
- [44] . 2018. LEAF: A benchmark for federated settings. arXiv:1812.01097. Retrieved from https://arxiv.org/abs/1812.01097.Google Scholar
- [45] . 2017. EMNIST: Extending MNIST to handwritten letters. In Proceedings of the 2017 International Joint Conference on Neural Networks. IEEE, 2921–2926.Google ScholarCross Ref
- [46] . 2020. Federated optimization in heterogeneous networks. In Proceedings of the Machine Learning and Systems. mlsys.org.Google Scholar
- [47] . 2019. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the Advances in Neural Information Processing Systems. 8024–8035.Google Scholar
- [48] . 2008. Visualizing data using t-SNE.Journal of Machine Learning Research 9, 11 (2008).Google Scholar
Index Terms
- Personalized Federated Learning on Non-IID Data via Group-based Meta-learning
Recommendations
Configure Your Federation: Hierarchical Attention-enhanced Meta-Learning Network for Personalized Federated Learning
Federated learning, as a distributed machine learning framework, enables clients to conduct model training without transmitting their data to the server, which is used to solve the dilemma of data silos and data privacy. It can work well on clients having ...
Memory-based optimization methods for model-agnostic meta-learning and personalized federated learning
In recent years, model-agnostic meta-learning (MAML) has become a popular research area. However, the stochastic optimization of MAML is still underdeveloped. Existing MAML algorithms rely on the "episode" idea by sampling a few tasks and data points to ...
FedDRL: Deep Reinforcement Learning-based Adaptive Aggregation for Non-IID Data in Federated Learning
ICPP '22: Proceedings of the 51st International Conference on Parallel ProcessingThe uneven distribution of local data across different edge devices (clients) results in slow model training and accuracy reduction in federated learning. Naive federated learning (FL) strategy and most alternative solutions attempted to achieve more ...
Comments