Skip to main content
SpringerLink
Log in

Neuron Pruning-Based Federated Learning for Communication-Efficient Distributed Training

  • Conference paper
  • First Online:
Algorithms and Architectures for Parallel Processing (ICA3PP 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14490))

  • 115 Accesses

Abstract

Efficient and flexible cloud computing is widely used in distributed systems. However, in the Internet of Things (IoT) environment with heterogeneous capabilities, the performance of cloud computing may decline due to limited communication resources. As located closer to the end, edge computing is used to replace cloud computing to provide timely and stable services. To accomplish distributed system and privacy preserving, Federated Learning (FL) has been combined with edge computing. However, due to the large number of clients, the amount of data transmitted will also grow exponentially. How to reduce the communication overhead in FL is still a big problem. As a major method to reduce the communication overhead, compressing the transmission parameters can effectively reduce the communication overhead. However, the existing methods do not consider the possible internal relationship between neurons. In this paper, we propose Neuron Pruning-Based FL for communication-efficient distributed training, which is a model pruning method to compress model parameters transmitted in FL. In contrast to the previous methods, we use dimensionality reduction method as the importance factor of neurons, and take advantage of the correlation between them to carry out model pruning. Our analysis results show that NPBFL can reduce communication overhead while maintaining classification accuracy.

This research was supported by the National Key R &D Program of China under Grant No. 2022YFB3102304 and in part by National Natural Science Foundation of China Grants (6222510562001057).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Peng, C., Hu, Q., Chen, J., Kang, K., Li, F., Zou, X.: Energy-efficient device selection in federated edge learning. In: 30th International Conference on Computer Communications and Networks, ICCCN 2021, Athens, Greece, 19–22 July 2021, pp. 1–9. IEEE (2021). https://doi.org/10.1109/ICCCN52240.2021.9522303

  2. Yang, D., Cheng, Z., Zhang, W., Zhang, H., Shen, X.: Burst-aware time-triggered flow scheduling with enhanced multi-CQF in time-sensitive networks. IEEE/ACM Trans. Networking 31(6), 2809–2824 (2023). https://doi.org/10.1109/TNET.2023.3264583

  3. Tange, K., Donno, M.D., Fafoutis, X., Dragoni, N.: A systematic survey of industrial Internet of Things security: requirements and fog computing opportunities. IEEE Commun. Surv. Tutorials 22(4), 2489–2520 (2020). https://doi.org/10.1109/COMST.2020.3011208

  4. Al-Garadi, M.A., Mohamed, A., Al-Ali, A.K., Du, X., Ali, I., Guizani, M.: A survey of machine and deep learning methods for Internet of Things (IoT) security. IEEE Commun. Surv. Tutorials 22(3), 1646–1685 (2020). https://doi.org/10.1109/COMST.2020.2988293

  5. Sha, J., Basara, N., Freedman, J., Xu, H.: FLOR: a federated learning-based music recommendation engine. In: 31st International Conference on Computer Communications and Networks, ICCCN 2022, Honolulu, HI, USA, 25–28 July 2022, pp. 1–2. IEEE (2022). https://doi.org/10.1109/ICCCN54977.2022.9868921

  6. Zhang, W., et al.: Optimizing federated learning in distributed industrial IoT: a multi-agent approach. IEEE J. Sel. Areas Commun. 39(12), 3688–3703 (2021). https://doi.org/10.1109/JSAC.2021.3118352

    Article  Google Scholar 

  7. Yang, D., et al.: DetFed: dynamic resource scheduling for deterministic federated learning over time-sensitive networks. IEEE Trans. Mobile Comput. (2023). Early Access, https://doi.org/10.1109/TMC.2023.3303017

  8. Mo, Z., Gao, Z., Zhao, C., Lin, Y.: FedDQ: a communication-efficient federated learning approach for internet of vehicles. J. Syst. Archit. 131, 102690 (2022). https://doi.org/10.1016/j.sysarc.2022.102690

  9. Peng, K., Zhang, H., Zhao, B., Liu, P.: Edge-cloud collaborative computation offloading for federated learning in smart city. In: IEEE International Conference on Dependable, Autonomic and Secure Computing, International Conference on Pervasive Intelligence and Computing, International Conference on Cloud and Big Data Computing, International Conference on Cyber Science and Technology Congress, DASC/PiCom/CBDCom/CyberSciTech 2022, Falerna, Italy, 12–15 September 2022, pp. 1–7. IEEE (2022). https://doi.org/10.1109/DASC/PiCom/CBDCom/Cy55231.2022.9927848

  10. Otoum, Y., Wan, Y., Nayak, A.: Federated transfer learning-based IDS for the internet of medical things (IoMT). In: IEEE Globecom 2021 Workshops, Madrid, Spain, 7–11 December 2021, pp. 1–6. IEEE (2021). https://doi.org/10.1109/GCWkshps52748.2021.9682118

  11. Deng, Y., et al.: SHARE: shaping data distribution at edge for communication-efficient hierarchical federated learning. In: 41st IEEE International Conference on Distributed Computing Systems, ICDCS 2021, Washington DC, USA, 7–10 July 2021, pp. 24–34. IEEE (2021). https://doi.org/10.1109/ICDCS51616.2021.00012

  12. Luo, S., Fan, P., Xing, H., Luo, L., Yu, H.: Eliminating communication bottlenecks in cross-device federated learning with in-network processing at the edge. In: IEEE International Conference on Communications, ICC 2022, Seoul, Korea, 16–20 May 2022, pp. 4601–4606. IEEE (2022). https://doi.org/10.1109/ICC45855.2022.9838381

  13. Liu, L., Zhang, J., Song, S., Letaief, K.B.: Communication-efficient federated distillation with active data sampling. In: IEEE International Conference on Communications, ICC 2022, Seoul, Korea, 16–20 May 2022, pp. 201–206. IEEE (2022). https://doi.org/10.1109/ICC45855.2022.9839214

  14. Qu, L., Song, S., Tsui, C.: FedDQ: communication-efficient federated learning with descending quantization. In: IEEE Global Communications Conference, GLOBECOM 2022, Rio de Janeiro, Brazil, 4–8 December 2022, pp. 281–286. IEEE (2022). https://doi.org/10.1109/GLOBECOM48099.2022.10001205

  15. Lian, Z., Wang, W., Su, C.: COFEL: communication-efficient and optimized federated learning with local differential privacy. In: ICC 2021 - IEEE International Conference on Communications, Montreal, QC, Canada, 14–23 June 2021, pp. 1–6. IEEE (2021). https://doi.org/10.1109/ICC42927.2021.9500632

  16. Vahidian, S., Morafah, M., Lin, B.: Personalized federated learning by structured and unstructured pruning under data heterogeneity. In: 41st IEEE International Conference on Distributed Computing Systems Workshops, ICDCSW 2021, Washington, DC, USA, 7–10 July 2021, pp. 27–34. IEEE (2021). https://doi.org/10.1109/ICDCSW53096.2021.00012

  17. Magnússon, S., Ghadikolaei, H.S., Li, N.: On maintaining linear convergence of distributed learning and optimization under limited communication. IEEE Trans. Signal Process. 68, 6101–6116 (2020). https://doi.org/10.1109/TSP.2020.3031073

  18. Li, Y., Gu, S., Mayer, C., Gool, L.V., Timofte, R.: Group sparsity: the hinge between filter pruning and decomposition for network compression. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 8015–8024. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00804, https://openaccess.thecvf.com/content_CVPR_2020/html/Li_Group_Sparsity_The_Hinge_Between_Filter_Pruning_and_Decomposition_for_CVPR_2020_paper.html

  19. Zhao, K., Zhang, X., Han, Q., Cheng, M.: Dependency aware filter pruning. CoRR abs/2005.02634 (2020). https://arxiv.org/abs/2005.02634

  20. Louizos, C., Welling, M., Kingma, D.P.: Learning sparse neural networks through l\({}_{{0}}\) regularization. CoRR abs/1712.01312 (2017). http://arxiv.org/abs/1712.01312

  21. Lin, M., et al.: HRank: filter pruning using high-rank feature map. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, 13–19 June 2020, pp. 1526–1535. Computer Vision Foundation/IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00160, https://openaccess.thecvf.com/content_CVPR_2020/html/Lin_HRank_Filter_Pruning_Using_High-Rank_Feature_Map_CVPR_2020_paper.html

  22. Chen, Y., Dai, X., Liu, M., Chen, D., Yuan, L., Liu, Z.: Dynamic convolution: attention over convolution kernels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    Google Scholar 

  23. Wang, W., Zhu, M., Zeng, X., Ye, X., Sheng, Y.: Malware traffic classification using convolutional neural network for representation learning. In: 2017 International Conference on Information Networking (ICOIN), pp. 712–717 (2017). https://doi.org/10.1109/ICOIN.2017.7899588

Download references

Author information

Authors and Affiliations

  1. School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing, 100876, China

    Jianfeng Guan, Pengcheng Wang & Jing Zhang

  2. Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, 100876, China

    Jianfeng Guan

  3. Department of Computer Science and Technology, Tsinghua University, Beijing, 100084, China

    Su Yao

Authors
  1. Jianfeng Guan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pengcheng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Su Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianfeng Guan .

Editor information

Editors and Affiliations

  1. Royal Melbourne Institute of Technology, Melbourne, VIC, Australia

    Zahir Tari

  2. Tianjin University, Tianjin, China

    Keqiu Li

  3. University of Arizona, Tucson, AZ, USA

    Hongyi Wu

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Guan, J., Wang, P., Yao, S., Zhang, J. (2024). Neuron Pruning-Based Federated Learning for Communication-Efficient Distributed Training. In: Tari, Z., Li, K., Wu, H. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2023. Lecture Notes in Computer Science, vol 14490. Springer, Singapore. https://doi.org/10.1007/978-981-97-0859-8_4

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-0859-8_4

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-0858-1

  • Online ISBN: 978-981-97-0859-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation