Skip to main content
Springer Nature Link
Log in

Federated Learning with Flexible Architectures

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases. Research Track (ECML PKDD 2024)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14942))

  • 869 Accesses

Abstract

Traditional federated learning (FL) methods have limited support for clients with varying computational and communication abilities, leading to inefficiencies and potential inaccuracies in model training. This limitation hinders the widespread adoption of FL in diverse and resource-constrained environments, such as those with client devices ranging from powerful servers to mobile devices. To address this need, this paper introduces Federated Learning with Flexible Architectures (FedFA), an FL training algorithm that allows clients to train models of different widths and depths. Each client can select a network architecture suitable for its resources, with shallower and thinner networks requiring fewer computing resources for training. Unlike prior work in this area, FedFA incorporates the layer grafting technique to align clients’ local architectures with the largest network architecture in the FL system during model aggregation. Layer grafting ensures that all client contributions are uniformly integrated into the global model, thereby minimizing the risk of any individual client’s data skewing the model’s parameters disproportionately and introducing security benefits. Moreover, FedFA introduces the scalable aggregation method to manage scale variations in weights among different network architectures. Experimentally, FedFA outperforms previous width and depth flexible aggregation strategies. Specifically, FedFA’s testing accuracy matches (1.00 times) or is up to 1.16 times higher globally for IID settings, 0.98 to 1.13 times locally, and 0.95 times to 1.20 times higher globally for non-IID settings compared to earlier strategies. Furthermore, FedFA demonstrates increased robustness against performance degradation in backdoor attack scenarios compared to earlier strategies. Earlier strategies exhibit more drops in testing accuracy under attacks-for IID data by 1.01 to 2.11 times globally, and for non-IID data by 0.89 to 3.31 times locally, and 1.11 to 1.74 times globally, compared to FedFA.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abad, G., Paguada, S., Ersoy, O., Picek, S., Ramírez-Durán, V.J., Urbieta, A.: Sniper backdoor: single client targeted backdoor attack in federated learning. In: 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 377–391. IEEE (2023)

    Google Scholar 

  2. Antunes, R.S., André da Costa, C., Küderle, A., Yari, I.A., Eskofier, B.: Federated learning for healthcare: systematic review and architecture proposal. ACM Trans. Intell. Syst. Technol. (TIST) 13(4), 1–23 (2022)

    Google Scholar 

  3. Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., Shmatikov, V.: How to backdoor federated learning. In: International Conference on Artificial Intelligence and Statistics, pp. 2938–2948. PMLR (2020)

    Google Scholar 

  4. Demertzis, K., Kikiras, P., Skianis, C., Rantos, K., Iliadis, L., Stamoulis, G.: Federated auto-meta-ensemble learning framework for AI-enabled military operations. Electronics 12(2), 430 (2023)

    Article  Google Scholar 

  5. Deng, Y., et al.: Tailorfl: dual-personalized federated learning under system and data heterogeneity. In: Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, pp. 592–606 (2022)

    Google Scholar 

  6. Diao, E., Ding, J., Tarokh, V.: Heterofl: computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264 (2020)

  7. Duan, Q., Hu, S., Deng, R., Lu, Z.: Combined federated and split learning in edge computing for ubiquitous intelligence in internet of things: state-of-the-art and future directions. Sensors 22(16), 5983 (2022)

    Article  Google Scholar 

  8. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256. JMLR Workshop and Conference Proceedings (2010)

    Google Scholar 

  9. Greff, K., Srivastava, R.K., Schmidhuber, J.: Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:1612.07771 (2016)

  10. Han, D.J., Bhatti, H.I., Lee, J., Moon, J.: Accelerating federated learning with split learning on locally generated losses. In: ICML 2021 Workshop on Federated Learning for User Privacy and Data Confidentiality. ICML Board (2021)

    Google Scholar 

  11. Hanin, B.: Which neural net architectures give rise to exploding and vanishing gradients? In: Advances in Neural Information Processing Systems, vol. 31 (2018)

    Google Scholar 

  12. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)

    Google Scholar 

  13. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  14. Howard, A., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324 (2019)

    Google Scholar 

  15. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456. PMLR (2015)

    Google Scholar 

  16. Kang, H., Cha, S., Shin, J., Lee, J., Kang, J.: NeFL: nested federated learning for heterogeneous clients. arXiv preprint arXiv:2308.07761 (2023)

  17. Kumar, A., Yin, B., Shaikh, A.M., Ali, M., Wei, W.: CorrNet: pearson correlation based pruning for efficient convolutional neural networks. Int. J. Mach. Learn. Cybern. 13(12), 3773–3783 (2022)

    Article  Google Scholar 

  18. Li, G., Yang, Y., Bhardwaj, K., Marculescu, R.: Zico: zero-shot NAS via inverse coefficient of variation on gradients. arXiv preprint arXiv:2301.11300 (2023)

  19. Li, L., Fan, Y., Tse, M., Lin, K.Y.: A review of applications in federated learning. Comput. Ind. Eng. 149, 106854 (2020)

    Article  Google Scholar 

  20. Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: challenges, methods, and future directions. IEEE Signal Process. Mag. 37(3), 50–60 (2020)

    Article  Google Scholar 

  21. Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogeneous networks. Proc. Mach. Learn. Syst. 2, 429–450 (2020)

    Google Scholar 

  22. Liu, F., Ren, X., Zhang, Z., Sun, X., Zou, Y.: Rethinking skip connection with layer normalization. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 3586–3598 (2020)

    Google Scholar 

  23. Lyu, L., Yu, H., Yang, Q.: Threats to federated learning: a survey. arXiv preprint arXiv:2003.02133 (2020)

  24. McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282. PMLR (2017)

    Google Scholar 

  25. Mhaskar, H.N., Poggio, T.: Deep vs. shallow networks: an approximation theory perspective. Anal. Appl. 14(06), 829–848 (2016)

    Google Scholar 

  26. Murshed, M.S., Murphy, C., Hou, D., Khan, N., Ananthanarayanan, G., Hussain, F.: Machine learning at the network edge: a survey. ACM Comput. Surv. (CSUR) 54(8), 1–37 (2021)

    Article  Google Scholar 

  27. Nguyen, D.C., Ding, M., Pathirana, P.N., Seneviratne, A., Li, J., Poor, H.V.: Federated learning for internet of things: a comprehensive survey. IEEE Commun. Surv. Tutor. 23(3), 1622–1658 (2021)

    Article  Google Scholar 

  28. Noci, L., et al.: The shaped transformer: attention models in the infinite depth-and-width limit. arXiv preprint arXiv:2306.17759 (2023)

  29. Oh, S., et al.: LocFedMix-SL: Localize, federate, and mix for improved scalability, convergence, and latency in split learning. In: Proceedings of the ACM Web Conference 2022, pp. 3347–3357 (2022)

    Google Scholar 

  30. Park, J., Yoon, D., Yeo, S., Oh, S.: Amble: adjusting mini-batch and local epoch for federated learning with heterogeneous devices. J. Parallel Distrib. Comput. 170, 13–23 (2022)

    Article  Google Scholar 

  31. Pfeiffer, K., Rapp, M., Khalili, R., Henkel, J.: Federated learning for computationally-constrained heterogeneous devices: a survey. ACM Comput. Surv. 55, 1–27 (2023)

    Article  Google Scholar 

  32. Ribero, M., Vikalo, H.: Communication-efficient federated learning via optimal client sampling. arXiv preprint arXiv:2007.15197 (2020)

  33. Rodríguez-Barroso, N., Jiménez-López, D., Luzón, M.V., Herrera, F., Martínez-Cámara, E.: Survey on federated learning threats: concepts, taxonomy on attacks and defences, experimental study and challenges. Inf. Fusion 90, 148–173 (2023)

    Article  Google Scholar 

  34. Ruan, Y., Zhang, X., Liang, S.C., Joe-Wong, C.: Towards flexible device participation in federated learning. In: International Conference on Artificial Intelligence and Statistics, pp. 3403–3411. PMLR (2021)

    Google Scholar 

  35. Samikwa, E., Di Maio, A., Braun, T.: Ares: adaptive resource-aware split learning for internet of things. Comput. Netw. 218, 109380 (2022)

    Article  Google Scholar 

  36. Tan, M., Le, Q.: Efficientnetv2: smaller models and faster training. In: International Conference on Machine Learning, pp. 10096–10106. PMLR (2021)

    Google Scholar 

  37. Tolpegin, V., Truex, S., Gursoy, M.E., Liu, L.: Data poisoning attacks against federated learning systems. In: Chen, L., Li, N., Liang, K., Schneider, S. (eds.) ESORICS 2020. LNCS, vol. 12308, pp. 480–501. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58951-6_24

    Chapter  Google Scholar 

  38. Turina, V., Zhang, Z., Esposito, F., Matta, I.: Federated or split? A performance and privacy analysis of hybrid split and federated learning architectures. In: 2021 IEEE 14th International Conference on Cloud Computing (CLOUD), pp. 250–260. IEEE (2021)

    Google Scholar 

  39. Vahidian, S., Morafah, M., Lin, B.: Personalized federated learning by structured and unstructured pruning under data heterogeneity. In: 2021 IEEE 41st International Conference on Distributed Computing Systems Workshops (ICDCSW), pp. 27–34. IEEE (2021)

    Google Scholar 

  40. Veit, A., Wilber, M.J., Belongie, S.: Residual networks behave like ensembles of relatively shallow networks. In: Advances in Neural Information Processing Systems, vol. 29 (2016)

    Google Scholar 

  41. Wang, K., et al.: Flexifed: personalized federated learning for edge clients with heterogeneous model architectures. In: Proceedings of the ACM Web Conference 2023, pp. 2979–2990 (2023)

    Google Scholar 

Download references

Acknowledgements

We thank A. Datta and P. Mardziel for access to computing resources for completing our experiments. This work was partially supported by the National Science Foundation under grants CNS-1751075, CNS-2106891, and CNS-2312761.

Author information

Authors and Affiliations

  1. Carnegie Mellon University, Pittsburgh, PA, 15213, USA

    Jong-Ik Park & Carlee Joe-Wong

Authors
  1. Jong-Ik Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carlee Joe-Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carlee Joe-Wong .

Editor information

Editors and Affiliations

  1. LTCI, Télécom Paris, Palaiseau Cedex, France

    Albert Bifet

  2. KU Leuven, Leuven, Belgium

    Jesse Davis

  3. Faculty of Informatics, Vytautas Magnus University, Akademija, Lithuania

    Tomas Krilavičius

  4. Institute of Computer Science, University of Tartu, Tartu, Estonia

    Meelis Kull

  5. Department of Computer Science, Bundeswehr University Munich, Munich, Germany

    Eirini Ntoutsi

  6. Department of Computer Science, University of Helsinki, Helsinki, Finland

    Indrė Žliobaitė

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1306 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Park, JI., Joe-Wong, C. (2024). Federated Learning with Flexible Architectures. In: Bifet, A., Davis, J., Krilavičius, T., Kull, M., Ntoutsi, E., Žliobaitė, I. (eds) Machine Learning and Knowledge Discovery in Databases. Research Track. ECML PKDD 2024. Lecture Notes in Computer Science(), vol 14942. Springer, Cham. https://doi.org/10.1007/978-3-031-70344-7_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-70344-7_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-70343-0

  • Online ISBN: 978-3-031-70344-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships