Abstract
Federated learning allows multiple clients to train a global model without data exchanging. But in real world, the global model is not suitable for all clients because they may hold heterogenous data and have personalized and individual demands, which will directly weaken the motivation of them to participate in federated learning. To make each client benefits from federated learning, researchers propose to train personalized models from global model using local data. However, the lack of raw data in the model retraining process will lead to the challenge of forgetting, which can deprive the personalized model of the benefits gained from federated learning. In extreme cases (e.g., the client lacks certain classes of data), the ability to recognize the lacked data may even be completely forgotten. To this end, we propose a local adaptation method to overcome forgetting, which add the generator synthetic data to local adaptation to realize model updating incrementally. We test our method on real-world datasets, and the results show that when adopting the proposed method on local adaptation, the clients can get flexible adaption ability to new data as well as keep the original recognition capability of the global model even in extreme cases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mcmahan, H.B., Moore, E., Ramage, D., Hampson, S., Arcas, B.: Communication-efficient learning of deep networks from decentralized data. In: PMLR, pp. 1273–1282 (2017)
Kulkarni, V., Kulkarni, M., Pant, A.: Survey of personalization techniques for federated learning. In: World S4, pp. 794–797 (2020)
Yu, T., Bagdasaryan, E., Shmatikov, V.: Salvaging federated learning by local adaptation. arXiv preprint arXiv:2002.04758 (2020)
Zhang, W., Wang, X., Zhou, P., Wu, W., Zhang, X.: Client selection for federated learning with non-IID data in mobile edge computing. IEEE Access 99, 1 (2021)
Smith, V., Chiang, C-K., Sanjabi, M., Talwalkar, A.: Federated multi-task learning. In: NIPS, pp. 4427–4437 (2017)
Fallah, A., Mokhtari, A., Ozdaglar, A.: Personalized federated learning: a meta-learning approach. arXiv preprint arXiv:2002.07948 (2020)
Cheng, G., Chadha, K., Duchi, J.: Fine-tuning is fine in federated learning. arXiv preprint arXiv:2108.07313 (2021)
Kairouz, P., Mcmahan, B., Avent, B., Bellet, A., Bennis, M., et al.: Advances and open problems in federated learning. arXiv e-prints arXiv:1912.04977 (2019)
McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: the sequential learning problem. Psychol. Learn. Motiv. 24, 109–165 (1989)
Chen, Y., Qin, X., Wang, J., Yu, C., Gao, W.: Fedhealth: a Federated transfer learning framework for wearable healthcare. IEEE Intell. Syst. 35(04), 83–93 (2020)
Arivazhagan, M.G., Aggarwal, V., Singh, A.K., Choudhary, S.: Federated learning with personalization layers. arXiv preprint arXiv:1912.00818 (2019)
Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. In: NIPS, pp. 2994–3003 (2017)
Yin, H., Molchanov, P., Alvarez, J.M., Li, Z., Kautz, J.: Dreaming to distill: data-free knowledge transfer via deepinversion. In: CVPR, pp. 8712–8721 (2020)
Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Bing, X., Bengio, Y.: Generative adversarial nets. In: NIPS, pp. 2672–2680 (2014)
Yoo, J., Cho, M., Kim, T., Kang, U.: Knowledge extraction with no observable data. In: NIPS, pp. 2705–2714 (2019)
Fan, C., Liu, P.: Federated generative adversarial learning. In: Peng, Y. (ed.) PRCV 2020. LNCS, vol. 12307, pp. 3–15. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60636-7_1
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40(12), 2935–2947 (2018)
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Granada, 12–17 December 2011, 5 (2011)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Belouadah, E,. Popescu, A,. Kanellos, I.: Initial classifier weights replay for memoryless class incremental learning. arXiv preprint arXiv:2008.13710 (2020)
Acknowledgements
This work is supported by NSF China (Nos. 61971139, 61601126), Foundation of Fujian Province (No. 2021J01576), and Research Fund of Fuzhou University (No. GXRC-21012).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, S., Feng, X., Zheng, H. (2022). Overcoming Forgetting in Local Adaptation of Federated Learning Model. In: Gama, J., Li, T., Yu, Y., Chen, E., Zheng, Y., Teng, F. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2022. Lecture Notes in Computer Science(), vol 13280. Springer, Cham. https://doi.org/10.1007/978-3-031-05933-9_48
Download citation
DOI: https://doi.org/10.1007/978-3-031-05933-9_48
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-05932-2
Online ISBN: 978-3-031-05933-9
eBook Packages: Computer ScienceComputer Science (R0)