Abstract
Today’s deep learning systems rely on large amounts of useful data to make accurate predictions. Often such data is private and thus not readily available due to rising privacy concerns. Federated learning (FL) tackles this problem by training a shared model locally on devices to aid learning in a privacy-preserving manner. Unfortunately, FL’s effectiveness degrades when model training involves clients with heterogeneous devices; a common case especially in developing countries. Slow clients are dropped in FL, which not only limits learning but also systematically excludes slow clients thereby potentially biasing results. We propose Hasaas; a system that tackles this challenge by adapting the model size for slow clients based on their hardware resources. By doing so, Hasaas obviates the need to drop slow clients, which improves model accuracy and fairness. To improve robustness in the presence of statistical heterogeneity, Hasaas uses insights from the Central Limit Theorem to estimate model parameters in every round. Experimental evaluation involving large-scale simulations and a small-scale real testbed shows that Hasaas provides robust performance in terms of test accuracy, fairness, and convergence times compared to state-of-the-art schemes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Private data includes any personal, personally identifiable, financial or sensitive user information [14].
- 2.
In the Urdu language, Hasaas means sensitive.
- 3.
Small models can reduce accuracy, whereas large models lead to slow convergence.
- 4.
In case of CNN models, it prunes filters too.
- 5.
Thus, in every training round, we estimate the mean and variance of each model parameter, which together uniquely identifies a Normal distribution.
- 6.
- 7.
With multi-model AFD, a different subset model is used by each client, all of the same size. However, training with a small fraction of clients in each round – a typical scenario in FL – makes the algorithm behave randomly, just like the FD scheme [6].
- 8.
Moreover, if the slow clients are unable to run the large model due to resource constraints, they cannot participate in the training process.
- 9.
In 2018, \(\sim \)300 million Android phones shipped globally had 1 GB or less RAM [1].
- 10.
CDR is the fraction of slow clients in the system. With FedAvg, such clients are dropped from the training process.
- 11.
Similar to Li et al. [22], we capture model fairness using the variance of test loss across clients. Thus, the more uniform the loss distribution is, the fairer the model.
- 12.
The memory specifications of these devices represent a wide range of smartphones. In developing regions, phones with 1 GB or less RAM had a market share of 57% compared to 20% in developed regions. Phones \(\ge \) 3 GB RAM had less than 25% market share in developing regions and over 50% share in developed countries [30].
- 13.
Some clients may not be able to run large models at all due to memory constraints.
- 14.
While the sample mean is an unbiased estimator of the population mean, the variance of the sample mean depends on the sample size (i.e., the number of clients).
- 15.
If each client’s model weights follow a different distribution, one can use generalizations of CLT, such as the Lyapunov CLT and Lindeberg CLT [15].
- 16.
Figures related to the ablation experiments are in Appendix A.1, which is available on our GitHub repository.
References
Build for Android (Go edition): optimize your app for global markets (Google I/O ’18). https://bit.ly/2UKLQDl
Abdullah, M., Qazi, Z.A., Qazi, I.A.: Causal impact of android go on mobile web performance. In: Proceedings of the 22nd ACM Internet Measurement Conference. IMC 2022, New York, NY, USA, pp. 113–129. Association for Computing Machinery (2022). https://doi.org/10.1145/3517745.3561456
Ahmad, S., Haamid, A.L., Qazi, Z.A., Zhou, Z., Benson, T., Qazi, I.A.: A view from the other side: understanding mobile phone characteristics in the developing world. In: Proceedings of the 2016 Internet Measurement Conference. IMC 2016, pp. 319–325 (2016). https://doi.org/10.1145/2987443.2987470
Bonawitz, K., Kairouz, P., McMahan, B., Ramage, D.: Federated learning and privacy: building privacy-preserving systems for machine learning and data science on decentralized data. Queue 19(5), 87–114 (2021). https://doi.org/10.1145/3494834.3500240
Bonawitz, K., et al.: Towards federated learning at scale: System design (2019). http://arxiv.org/abs/1902.01046
Bouacida, N., Hou, J., Zang, H., Liu, X.: Adaptive federated dropout: improving communication efficiency and generalization for federated learning. In: IEEE INFOCOM 2021 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), pp. 1–6 (2021). https://doi.org/10.1109/INFOCOMWKSHPS51825.2021.9484526
Bukaty, P.: The California Consumer Privacy Act (CCPA): An Implementation Guide. IT Governance Publishing (2019). http://www.jstor.org/stable/j.ctvjghvnn
Caldas, S., Konečný, J., McMahan, H.B., Talwalkar, A.: Expanding the reach of federated learning by reducing client resource requirements. CoRR abs/1812.07210 (2018). http://arxiv.org/abs/1812.07210
Caldas, S., et al.: LEAF: a benchmark for federated settings. CoRR abs/1812.01097 (2018). http://arxiv.org/abs/1812.01097
Chou, L., Liu, Z., Wang, Z., Shrivastava, A.: Efficient and less centralized federated learning. In: Oliver, N., Pérez-Cruz, F., Kramer, S., Read, J., Lozano, J.A. (eds.) ECML PKDD 2021. LNCS (LNAI), vol. 12975, pp. 772–787. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86486-6_47
Cohen, G., Afshar, S., Tapson, J., van Schaik, A.: EMNIST: an extension of MNIST to handwritten letters. CoRR abs/1702.05373 (2017). http://arxiv.org/abs/1702.05373
Diao, E., Ding, J., Tarokh, V.: Heterofl: computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264 (2020)
Duan, J.-H., Li, W., Lu, S.: FedDNA: federated learning with decoupled normalization-layer aggregation for non-iid data. In: Oliver, N., Pérez-Cruz, F., Kramer, S., Read, J., Lozano, J.A. (eds.) ECML PKDD 2021. LNCS (LNAI), vol. 12975, pp. 722–737. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86486-6_44
Dwork, C., McSherry, F., Nissim, K., Smith, A.: Calibrating noise to sensitivity in private data analysis. In: Halevi, S., Rabin, T. (eds.) TCC 2006. LNCS, vol. 3876, pp. 265–284. Springer, Heidelberg (2006). https://doi.org/10.1007/11681878_14
Feller, W.: An Introduction to Probability Theory and Its Applications, vol. 1. Wiley (1968). http://www.amazon.ca/exec/obidos/redirect?tag=citeulike04-20 &path=ASIN/0471257087
Google: Firebase services. https://firebase.google.com
Hashimoto, T.B., Srivastava, M., Namkoong, H., Liang, P.: Fairness without demographics in repeated loss minimization (2018)
Hu, Z., Shaloudegi, K., Zhang, G., Yu, Y.: Fedmgda+: federated learning meets multi-objective optimization. CoRR abs/2006.11489 (2020). https://arxiv.org/abs/2006.11489
Jiang, Y., et al.: Model pruning enables efficient federated learning on edge devices. IEEE Trans. Neural Networks Learn. Syst. 1–13 (2022). https://doi.org/10.1109/TNNLS.2022.3166101
Kairouz, P., McMahan, H.B., Avent, B., Bellet, A., et al.: Advances and open problems in federated learning. CoRR abs/1912.04977 (2019). http://arxiv.org/abs/1912.04977
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015). https://doi.org/10.1038/nature14539
Li, T., Hu, S., Beirami, A., Smith, V.: Ditto: fair and robust federated learning through personalization. CoRR abs/2012.04221 (2020). https://arxiv.org/abs/2012.04221
Li, T., Sahu, A.K., Sanjabi, M., Zaheer, M., Talwalkar, A.S., Smith, V.: Federated optimization in heterogeneous networks. In: MLSys (2020)
Li, T., Sanjabi, M., Smith, V.: Fair resource allocation in federated learning. CoRR abs/1905.10497 (2019). http://arxiv.org/abs/1905.10497
Li, X.-C., Zhan, D.-C., Shao, Y., Li, B., Song, S.: FedPHP: Federated Personalization with Inherited Private Models. In: Oliver, N., Pérez-Cruz, F., Kramer, S., Read, J., Lozano, J.A. (eds.) ECML PKDD 2021. LNCS (LNAI), vol. 12975, pp. 587–602. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86486-6_36https://2021.ecmlpkdd.org/wp-content/uploads/2021/07/sub_654.pdf
Ma, Z., L, Y., Li, W., Cui, S.: Beyond random selection: a perspective from model inversion in personalized federated learning. In: Amini, MR., Canu, S., Fischer, A., Guns, T., Kralj Novak, P., Tsoumakas, G. (eds.) ECML PKDD 2022. LNCS, vol. 13716, pp. 572–586. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-26412-2_35, https://2022.ecmlpkdd.org/wp-content/uploads/2022/09/sub_801.PDF
McMahan, B., Moore, E., Ramage, D., Hampson, S., Arcas, B.A.V.: Communication-efficient learning of deep networks from decentralized data. In: Singh, A., Zhu, J. (eds.) Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, 20–22 April 2017, vol. 54, pp. 1273–1282. PMLR (2017). https://proceedings.mlr.press/v54/mcmahan17a.html
Mohri, M., Sivek, G., Suresh, A.T.: Agnostic federated learning. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, 09–15 Jun 2019, vol. 97, pp. 4615–4625. PMLR (2019). https://proceedings.mlr.press/v97/mohri19a.html
Naseer, U., Benson, T.A., Netravali, R.: Webmedic: disentangling the memory-functionality tension for the next billion mobile web users. In: Proceedings of the 22nd International Workshop on Mobile Computing Systems and Applications. HotMobile 2021, pp. 71–77. Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3446382.3448652
Naseer, U., Benson, T.A., Netravali, R.: Webmedic: disentangling the memory-functionality tension for the next billion mobile web users. In: Proceedings of the 22nd International Workshop on Mobile Computing Systems and Applications. HotMobile 2021, New York, NY, USA, pp. 71–77. Association for Computing Machinery (2021). https://doi.org/10.1145/3446382.3448652
OpenMined: Kotlinsyft. https://github.com/OpenMined/KotlinSyft/
OpenMined: Pysyft. https://github.com/OpenMined/PySyft/
Qazi, I.A., et al.: Mobile web browsing under memory pressure. SIGCOMM Comput. Commun. Rev. 50(4), 35–48 (2020). https://doi.org/10.1145/3431832.3431837
Smith, V., Chiang, C.K., Sanjabi, M., Talwalkar, A.: Federated multi-task learning. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS 2017, Red Hook, NY, USA, pp. 4427–4437. Curran Associates Inc. (2017)
Tan, C.M.J., Motani, M.: DropNet: reducing neural network complexity via iterative pruning. In: III, H.D., Singh, A. (eds.) Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, 13–18 July 2020, vol. 119, pp. 9356–9366. PMLR (2020). https://proceedings.mlr.press/v119/tan20a.html
Voigt, P., Bussche, A.V.D.: The EU General Data Protection Regulation (GDPR): A Practical Guide, 1st edn. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-57959-7
Waheed, T., Qazi, I.A., Akhtar, Z., Qazi, Z.A.: Coal not diamonds: how memory pressure falters mobile video QOE. In: Proceedings of the 18th International Conference on Emerging Networking EXperiments and Technologies. CoNEXT 2022, New York, NY, USA, pp. 307–320. Association for Computing Machinery (2022). https://doi.org/10.1145/3555050.3569120
Wang, J., Charles, Z., Xu, Z., Joshi, G., et al.: A field guide to federated optimization. CoRR abs/2107.06917 (2021).https://arxiv.org/abs/2107.06917
Xu, C., Qu, Y., Xiang, Y., Gao, L.: Asynchronous federated learning on heterogeneous devices: a survey (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Munir, M.T., Saeed, M.M., Ali, M., Qazi, Z.A., Raza, A.A., Qazi, I.A. (2023). Learning Fast and Slow: Towards Inclusive Federated Learning. In: Koutra, D., Plant, C., Gomez Rodriguez, M., Baralis, E., Bonchi, F. (eds) Machine Learning and Knowledge Discovery in Databases: Research Track. ECML PKDD 2023. Lecture Notes in Computer Science(), vol 14170. Springer, Cham. https://doi.org/10.1007/978-3-031-43415-0_23
Download citation
DOI: https://doi.org/10.1007/978-3-031-43415-0_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43414-3
Online ISBN: 978-3-031-43415-0
eBook Packages: Computer ScienceComputer Science (R0)
