Skip to main content
Springer Nature Link
Log in

Towards Heterogeneous Federated Learning

  • Conference paper
  • First Online:
Computer Supported Cooperative Work and Social Computing (ChineseCSCW 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1681))

  • 715 Accesses

Abstract

Federated Learning (FL), a novel distributed machine learning framework, made it possible to model collaboratively without risking participants’ privacy. All components of FL, including devices, networks, data, and models, are heterogeneous because of the dispersed feature. These heterogeneity issues impeded FL’s performance. HFL (Heterogeneous Federated Learning) offers a viable solution to these issues.

HFL has become an emerging research topic. We have conducted detailed research into the unique characteristics and challenges of HFL in the paper. And summaries methods of HFL at different levels. We reviewed the evaluation methods for HFL and provided an outlook on the future direction of HFL by analyzing the strengths and limits of the existing study.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abdellatif, A.A., et al.: Communication-efficient hierarchical federated learning for IoT heterogeneous systems with imbalanced data. Fut. Gener. Comput. Syst. 128, 406–419 (2022)

    Article  Google Scholar 

  2. Abdelmoniem, A.M., Ho, C.Y., Papageorgiou, P., Canini, M.: Empirical analysis of federated learning in heterogeneous environments. In: Proceedings of the 2nd European Workshop on Machine Learning and Systems, pp. 1–9 (2022)

    Google Scholar 

  3. Anguita, D., Ghio, A., Oneto, L., Parra Perez, X., Reyes Ortiz, J.L.: A public domain dataset for human activity recognition using smartphones. In: Proceedings of the 21th International European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, pp. 437–442 (2013)

    Google Scholar 

  4. Baumgartner, J., Zannettou, S., Keegan, B., Squire, M., Blackburn, J.: The pushshift reddit dataset. In: Proceedings of the International AAAI Conference on Web and Social Media, vol. 14, pp. 830–839 (2020)

    Google Scholar 

  5. Bonawitz, K., et al.: Practical secure aggregation for privacy-preserving machine learning. In: proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175–1191 (2017)

    Google Scholar 

  6. Cao, X., Li, Z., Yu, H., Sun, G.: COFED: cross-silo heterogeneous federated multi-task learning via co-training. arXiv preprint arXiv:2202.08603 (2022)

  7. Chen, L.Y., Chiu, T.C., Pang, A.C., Cheng, L.C.: Fedequal: defending model poisoning attacks in heterogeneous federated learning. In: 2021 IEEE Global Communications Conference (GLOBECOM), pp. 1–6. IEEE (2021)

    Google Scholar 

  8. Cho, Y.J., Manoel, A., Joshi, G., Sim, R., Dimitriadis, D.: Heterogeneous ensemble knowledge transfer for training large models in federated learning. arXiv preprint arXiv:2204.12703 (2022)

  9. Cohen, G., Afshar, S., Tapson, J., van Schaik, A.: Emnist: extending mnist to handwritten letters. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 2921–2926 (2017). https://doi.org/10.1109/IJCNN.2017.7966217

  10. Cui, Y., Cao, K., Zhou, J., Wei, T.: HELCFL: high-efficiency and low-cost federated learning in heterogeneous mobile-edge computing. In: 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1227–1232. IEEE (2022)

    Google Scholar 

  11. De Campos, T.E., Babu, B.R., Varma, M.: Character recognition in natural images. VISAPP 7(2), 2 (2009)

    Google Scholar 

  12. Duan, M., et al.: Fedgroup: efficient federated learning via decomposed similarity-based clustering. In: 2021 IEEE International Conference on Parallel & Distributed Processing with Applications, pp. 228–237. IEEE (2021)

    Google Scholar 

  13. Elkordy, A.R., Avestimehr, A.S.: Heterosag: secure aggregation with heterogeneous quantization in federated learning. IEEE Trans. Commun. 70(4), 2372–2386 (2022)

    Article  Google Scholar 

  14. Gao, D., Liu, Y., Huang, A., Ju, C., Yu, H., Yang, Q.: Privacy-preserving heterogeneous federated transfer learning. In: 2019 IEEE International Conference on Big Data (Big Data), pp. 2552–2559. IEEE (2019)

    Google Scholar 

  15. Gao, Z., Duan, Y., Yang, Y., Rui, L., Zhao, C.: Fedsec: a robust differential private federated learning framework in heterogeneous networks. In: 2022 IEEE Wireless Communications and Networking Conference (WCNC), pp. 1868–1873 (2022). https://doi.org/10.1109/WCNC51071.2022.9771929

  16. Go, A., Bhayani, R., Huang, L.: Twitter sentiment classification using distant supervision. Processing 150 (2009)

    Google Scholar 

  17. Guo, K., Chen, Z., Yang, H.H., Quek, T.Q.: Dynamic scheduling for heterogeneous federated learning in private 5g edge networks. IEEE J. Sel. Topics Signal Process. 16, 26–40 (2021)

    Article  Google Scholar 

  18. Guo, Y., Wang, Q., Ji, T., Wang, X., Li, P.: Resisting distributed backdoor attacks in federated learning: a dynamic norm clipping approach. In: 2021 IEEE International Conference on Big Data (Big Data), pp. 1172–1182. IEEE (2021)

    Google Scholar 

  19. Hin, C.Y., Edith, N.: Fedhe: heterogeneous models and communication-efficient federated learning. arXiv preprint arXiv:2110.09910 (2021)

  20. Hu, M., Wu, D., Zhou, Y., Chen, X., Chen, M.: Incentive-aware autonomous client participation in federated learning. IEEE Trans. Parallel Distrib. Syst. 33(10), 2612–2627 (2022)

    Article  Google Scholar 

  21. Kachuee, M., Fazeli, S., Sarrafzadeh, M.: ECG heartbeat classification: a deep transferable representation. In: 2018 IEEE International Conference on Healthcare Informatics (ICHI), pp. 443–444. IEEE (2018)

    Google Scholar 

  22. Kanaparthy, S., Padala, M., Damle, S., Gujar, S.: Fair federated learning for heterogeneous face data. arXiv preprint arXiv:2109.02351 (2021)

  23. Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S., Stich, S., Suresh, A.T.: Scaffold: stochastic controlled averaging for federated learning. In: International Conference on Machine Learning, pp. 5132–5143. PMLR (2020)

    Google Scholar 

  24. Karkkainen, K., Joo, J.: Fairface: face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1548–1558 (2021)

    Google Scholar 

  25. Kinnunen, T., Kamarainen, J.K., Lensu, L., Lankinen, J., Käviäinen, H.: Making visual object categorization more challenging: randomized caltech-101 data set. In: 2010 20th International Conference on Pattern Recognition, pp. 476–479. IEEE (2010)

    Google Scholar 

  26. Kohavi, R.: Scaling up the accuracy of naive-bayes classifiers: a decision-tree hybrid. In: KDD, vol. 96, pp. 202–207 (1996)

    Google Scholar 

  27. Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. University of Toronto (2009)

    Google Scholar 

  28. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  29. Li, A., Set al.: Lotteryfl: personalized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets. arXiv preprint arXiv:2008.03371 (2020)

  30. Li, A., Sun, J., Zeng, X., Zhang, M., Li, H., Chen, Y.: Fedmask: joint computation and communication-efficient personalized federated learning via heterogeneous masking. In: Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pp. 42–55 (2021)

    Google Scholar 

  31. Li, D., Wang, J.: FEDMD: heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581 (2019)

  32. Li, D., Yang, Y., Song, Y.Z., Hospedales, T.M.: Deeper, broader and artier domain generalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5542–5550 (2017)

    Google Scholar 

  33. Li, L., et al.: Fedsae: a novel self-adaptive federated learning framework in heterogeneous systems. In: 2021 International Joint Conference on Neural Networks (IJCNN) (2021)

    Google Scholar 

  34. Li, L., Shi, D., Hou, R., Li, H., Pan, M., Han, Z.: To talk or to work: flexible communication compression for energy efficient federated learning over heterogeneous mobile edge devices. In: IEEE INFOCOM 2021-IEEE Conference on Computer Communications, pp. 1–10. IEEE (2021)

    Google Scholar 

  35. Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: challenges, methods, and future directions. IEEE Signal Process. Maga. 37(3), 50–60 (2020)

    Article  Google Scholar 

  36. Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V.: Federated optimization in heterogeneous networks. Proc. Mach. Learn. Syst. 2, 429–450 (2020)

    Google Scholar 

  37. Li, X., Li, Y., Li, S., Zhou, Y., Chen, C., Zheng, Z.: A unified federated DNNs framework for heterogeneous mobile devices. IEEE Internet Things J. 9(3), 1737–1748 (2021)

    Article  Google Scholar 

  38. Li, Y., Zhou, W., Wang, H., Mi, H., Hospedales, T.M.: Fedh2l: federated learning with model and statistical heterogeneity. arXiv preprint arXiv:2101.11296 (2021)

  39. Liu, Y., Zhang, L., Ge, N., Li, G.: A systematic literature review on federated learning: from a model quality perspective. arXiv preprint arXiv:2012.01973 (2020)

  40. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)

    Google Scholar 

  41. Lu, X., Liao, Y., Liu, C., Lio, P., Hui, P.: Heterogeneous model fusion federated learning mechanism based on model mapping. IEEE Internet Things J. 9, 6058–6068 (2021)

    Article  Google Scholar 

  42. Luo, J., Yang, J., Ye, X., Guo, X., Zhao, W.: Fedskel: efficient federated learning on heterogeneous systems with skeleton gradients update. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 3283–3287 (2021)

    Google Scholar 

  43. Ma, C., et al.: On safeguarding privacy and security in the framework of federated learning. IEEE Netw. 34(4), 242–248 (2020)

    Article  Google Scholar 

  44. Ma, Q., Xu, Y., Xu, H., Jiang, Z., Huang, L., Huang, H.: FEDSA: a semi-asynchronous federated learning mechanism in heterogeneous edge computing. IEEE J. Sel. Areas Commun. 39(12), 3654–3672 (2021)

    Article  Google Scholar 

  45. Ma, X., Zhou, Y., Wang, L., Miao, M.: Privacy-preserving byzantine-robust federated learning. Comput. Stand. Interfaces 80, 103561 (2022)

    Article  Google Scholar 

  46. McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282. PMLR (2017)

    Google Scholar 

  47. Melis, L., Song, C., De Cristofaro, E., Shmatikov, V.: Exploiting unintended feature leakage in collaborative learning. In: 2019 IEEE Symposium on Security and Privacy (SP), pp. 691–706 (2019). https://doi.org/10.1109/SP.2019.00029

  48. Shakespeare, W.: The Complete Works of William Shakespeare. Race Point Publishing (2014)

    Google Scholar 

  49. Shamir, O., Srebro, N., Zhang, T.: Communication-efficient distributed optimization using an approximate newton-type method. In: International Conference on Machine Learning, pp. 1000–1008. PMLR (2014)

    Google Scholar 

  50. Shang, X., Lu, Y., Huang, G., Wang, H.: Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features. arXiv preprint arXiv:2204.13399 (2022)

  51. Shin, J., Li, Y., Liu, Y., Lee, S.J.: Sample selection with deadline control for efficient federated learning on heterogeneous clients. arXiv preprint arXiv:2201.01601 (2022)

  52. Stevenson, N.J., Tapani, K., Lauronen, L., Vanhatalo, S.: A dataset of neonatal EEG recordings with seizure annotations. Sci. Data 6(1), 1–8 (2019)

    Article  Google Scholar 

  53. Sun, C., Jiang, T., Zonouz, S., Pompili, D.: Fed2kd: heterogeneous federated learning for pandemic risk assessment via two-way knowledge distillation. In: 2022 17th Wireless On-Demand Network Systems and Services Conference (WONS), pp. 1–8. IEEE (2022)

    Google Scholar 

  54. Tan, Y., et al.: Fedproto: federated prototype learning across heterogeneous clients. In: AAAI Conference on Artificial Intelligence, vol. 1 (2022)

    Google Scholar 

  55. Timofte, R., Zimmermann, K., Van Gool, L.: Multi-view traffic sign detection, recognition, and 3d localisation. Mach. Vision Appl. 25(3), 633–647 (2014)

    Article  Google Scholar 

  56. Van Oord, A., Kalchbrenner, N., Kavukcuoglu, K.: Pixel recurrent neural networks. In: International Conference on Machine Learning, pp. 1747–1756. PMLR (2016)

    Google Scholar 

  57. Venkateswara, H., Eusebio, J., Chakraborty, S., Panchanathan, S.: Deep hashing network for unsupervised domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5018–5027 (2017)

    Google Scholar 

  58. Wang, D., et al.: CFL-HC: a coded federated learning framework for heterogeneous computing scenarios. In: 2021 IEEE Global Communications Conference (GLOBECOM), pp. 1–6. IEEE (2021)

    Google Scholar 

  59. Wang, J., Liu, Q., Liang, H., Joshi, G., Poor, H.V.: A novel framework for the analysis and design of heterogeneous federated learning. IEEE Trans. Signal Process. 69, 5234–5249 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  60. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)

  61. Xu, C., Qu, Y., Xiang, Y., Gao, L.: Asynchronous federated learning on heterogeneous devices: a survey. arXiv preprint arXiv:2109.04269 (2021)

  62. Xu, G., Li, H., Liu, S., Yang, K., Lin, X.: Verifynet: secure and verifiable federated learning. IEEE Trans. Inf. For. Secur. 15, 911–926 (2019)

    Article  Google Scholar 

  63. Zaccone, R., Rizzardi, A., Caldarola, D., Ciccone, M., Caputo, B.: Speeding up heterogeneous federated learning with sequentially trained superclients. arXiv preprint arXiv:2201.10899 (2022)

  64. Zeng, H., Zhou, T., Guo, Y., Cai, Z., Liu, F.: Fedcav: contribution-aware model aggregation on distributed heterogeneous data in federated learning. In: 50th International Conference on Parallel Processing, pp. 1–10 (2021)

    Google Scholar 

  65. Zhang, H., Kim, J.: Towards a federated learning framework for heterogeneous devices of internet of things. arXiv preprint arXiv:2105.14675 (2021)

  66. Zhang, X., Li, F., Zhang, Z., Li, Q., Wang, C., Wu, J.: Enabling execution assurance of federated learning at untrusted participants. In: IEEE INFOCOM 2020-IEEE Conference on Computer Communications, pp. 1877–1886. IEEE (2020)

    Google Scholar 

  67. Zhang, Z., Song, Y., Qi, H.: Age progression/regression by conditional adversarial autoencoder. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5810–5818 (2017)

    Google Scholar 

  68. Zhou, X., et al.: Two-layer federated learning with heterogeneous model aggregation for 6G supported internet of vehicles. IEEE Trans. Veh. Technol. 70(6), 5308–5317 (2021)

    Article  Google Scholar 

Download references

Acknowledgments

This paper is partly supported by the National Research and Development Plan under Grant No. 2021YFF0704102, the National Social Science Fund under Grant No. 20BJY131, the major Science and Technology Innovation of Shandong Province under Grant Nos. 2021CXGC010108, the China-Singapore International Joint Research Project under Grant No. 206-A021002, the Industrial Experts Program of Spring City, the Fundamental Research Funds of Shandong University.

Author information

Authors and Affiliations

  1. College of Software, Shandong University, Jinan, 250101, China

    Yue Huang, Lanju Kong, Qingzhong Li & Lizhen Cui

  2. Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University, Jinan, 250101, China

    Yonghui Xu & Lizhen Cui

  3. China-Singapore International Joint Research Institute, Guangzhou, 510663, China

    Yonghui Xu

  4. Dareway Software Co., Ltd., Jinan, 250200, China

    Lanju Kong & Qingzhong Li

Authors
  1. Yue Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yonghui Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lanju Kong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qingzhong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lizhen Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qingzhong Li .

Editor information

Editors and Affiliations

  1. Shandong University, Jinan, China

    Yuqing Sun

  2. Fudan University, Shanghai, China

    Tun Lu

  3. Taiyuan University of Science and Technology, Taiyuan, China

    Yinzhang Guo

  4. Shanxi Datong University, Datong, China

    Xiaoxia Song

  5. Tongji University, Shanghai, China

    Hongfei Fan

  6. Guangdong University of Technology, Guangzhou, China

    Dongning Liu

  7. University of Shanghai for Science and Technology, Shanghai, China

    Liping Gao

  8. Tongji University, Shanghai, China

    Bowen Du

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Huang, Y., Xu, Y., Kong, L., Li, Q., Cui, L. (2023). Towards Heterogeneous Federated Learning. In: Sun, Y., et al. Computer Supported Cooperative Work and Social Computing. ChineseCSCW 2022. Communications in Computer and Information Science, vol 1681. Springer, Singapore. https://doi.org/10.1007/978-981-99-2356-4_31

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-2356-4_31

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-2355-7

  • Online ISBN: 978-981-99-2356-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships