Abstract
Federated Learning(FL) is a popular privacy-preserving machine learning paradigm that enables the creation of a robust centralized model without sacrificing the privacy of clients’ data. FL has a wide range of applications, but it does not integrate the idea of independent model design for each client, which is imperative in the areas like healthcare, banking sector, or AI as service (AIaaS), due to the paramount importance of data and heterogeneous nature of tasks. In this work, we propose a Robust Multi-Model FL (RMMFL) framework, an extension to FedMD, which under the same set of assumptions significantly improves the results of each individual model. RMMFL adapted two changes in the FedMD training process: First, a high entropy aggregation method is introduced to soften the output predictions. Second, a weighted ensemble technique is used to weigh the predictions of each client model in line with their performance. Extensive experiments are performed on heterogeneous models using CIFAR/MNIST as benchmark datasets, the simulations results obtained from our study shows that RMMFL exceeds the accuracy by over \(5\%\) compare to the baseline method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Voigt, P., von dem Bussche, A.: Introduction and ‘checklist’. The EU General Data Protection Regulation (GDPR). https://doi.org/10.1007/978-3-319-57959-7
Annas, G.J.: HIPAA regulations - a new era of medical-record privacy? N. Engl. J. Med. 348, 1486 (2003). https://doi.org/10.1056/nejmlim035027. ISSN 0028-4793
Shokri, R., Shmatikov, V.: Privacy-preserving deep learning. In: 2015 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015 (2016). https://doi.org/10.1109/ALLERTON.2015.7447103. ISBN 9781509018239
Yang, Q., Liu, Y., Chen, T., Tong, Y.: Federated machine learning: concept and applications. ACM Trans. Intell. Syst. Technol. 10, 1–9 (2019). https://doi.org/10.1145/3298981, arXiv: 1902.04885. ISSN 21576912
McMahan, H.B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.Y.: Communication-efficient learning of deep networks from decentralized data. arXiv:1602.05629 [cs.LG] (2017)
Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: challenges, methods, and future directions. IEEE Signal Process. Mag. 37(3), 50–60 (2020). https://doi.org/10.1109/MSP.2020.2975749, arXiv:1908.07873. ISSN 15580792
Nishio, T., Yonetani, R.: Client selection for federated learning with heterogeneous resources in mobile edge. In: IEEE International Conference on Communications (2019). https://doi.org/10.1109/ICC.2019.8761315, arXiv:1804.08333. ISBN 9781538680889
Sahu, A.K., Li, T., Sanjabi, M., Zaheer, M., Talwalkar, A., Smith, V.: On the convergence of federated optimization in heterogeneous networks. arXiv:1812.06127 [cs, stat] (2018)
Smith, V., Chiang, C.K., Sanjabi, M., Talwalkar, A.: Federated multi-task learning. In: Advances in Neural Information Processing Systems (2017)
Wu, Q., He, K., Chen, X.: Personalized federated learning for intelligent IoT applications: a cloud-edge based framework. IEEE Open J. Comput. Soc. 1, 35–44 (2020). https://doi.org/10.1109/OJCS.2020.2993259, arXiv:2002.10671. ISSN 15581756
Li, D., Wang, J.: FedMD: heterogenous federated learning via model distillation. arXiv:1910.03581 (2019)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv:1503.02531, March 2015
Liu, Y., Yuan, X., Xiong, Z., Kang, J., Wang, X., Niyato, D.: Federated learning for 6G communications: challenges, methods, and future directions. CoRR arXiv:2006.02931 (2020)
Kairouz, P., et al.: Advances and open problems in federated learning. CoRR arXiv:1912.04977 (2019)
Lim, W.Y.B., et al.: Federated learning in mobile edge networks: a comprehensive survey. IEEE Commun. Surv. Tutor. 22(3), 2031–2063 (2020). https://doi.org/10.1109/COMST.2020.2986024
Lin, T., Stich, S.U., Patel, K.K., Jaggi, M.: Don’t use large mini-batches, use local SGD. arXiv:1808.07217 [cs.LG] (2020)
Sattler, F., Müller, K.-R., Samek, W.: Clustered federated learning: model-agnostic distributed multi-task optimization under privacy constraints. arXiv:1910.01991 [cs.LG] (2019)
Hanzely, F., Richtárik, P.: Federated learning of a mixture of global and local models. arXiv:2002.05516 [cs.LG] (2021)
Diao, E., Ding, J., Tarokh, V.: HeteroFL: Computation and communication efficient federated learning for heterogeneous clients. arXiv:2010.01264 [cs.LG] (2021)
Yang, H., He, H., Zhang, W., Cao, X.: FedSteg: a federated transfer learning framework for secure image steganalysis. IEEE Trans. Netw. Sci. Eng. 8(2), 1084–1094 (2021). https://doi.org/10.1109/TNSE.2020.2996612
Chen, Y., et al.: FedHealth: a federated transfer learning framework for wearable healthcare. arXiv: 1907.09173 [cs.LG] (2021)
Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: FitNets: hints for thin deep nets. arXiv:1412.6550, December 2014
Chang, H., Shejwalkar, V., Shokri, R., Houmansadr, A.: Cronus: robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv:1912.11279, December 2019
Hashem, S., Schmeiser, B.: Approximating a function and its derivatives using MSE-optimal linear combinations of trained feedforward neural networks. In: Proceedings of the Joint Conference on Neural Networks, pp. 617–620 (1993)
Yuksel, S.E., Wilson, J.N., Gader, P.D.: Twenty years of mixture of experts. IEEE Trans. Neural Netw. Learn. Syst. 23(8), 1177–1193 (2012). https://doi.org/10.1109/TNNLS.2012.2200299
Schapire, R.E.: A brief introduction to boosting. In: Proceedings of the 16th International Joint Conference on Artificial Intelligence - Volume 2, IJCAI’99, San Francisco, CA, USA, pp. 1401–1406. Morgan Kaufmann Publishers Inc. (1999)
Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation and active learning. In: Proceedings of the 7th International Conference on Neural Information Processing Systems, NIPS’94, Cambridge, MA, USA, pp. 231–238. MIT Press (1994)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Muhammad, A., Lin, K., Gao, J., Chen, B. (2022). Robust Multi-model Personalized Federated Learning via Model Distillation. In: Lai, Y., Wang, T., Jiang, M., Xu, G., Liang, W., Castiglione, A. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2021. Lecture Notes in Computer Science(), vol 13157. Springer, Cham. https://doi.org/10.1007/978-3-030-95391-1_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-95391-1_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-95390-4
Online ISBN: 978-3-030-95391-1
eBook Packages: Computer ScienceComputer Science (R0)