Skip to main content

Advertisement

Springer Nature Link
Log in

Cluster knowledge-driven vertical federated learning

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

In industrial scenarios, cross-departmental collaboration is necessary to achieve quality traceability. However, data cannot be shared due to privacy concerns. Vertical Federated Learning (VFL) enables heterogeneous industrial sectors to jointly train models while preserving product privacy. However, existing traditional VFL algorithms only focus on aligning feature benefits and suffer from high communication costs and poor performance. This paper proposes a “Cluster Knowledge-Driven Vertical Federated Learning” (Cluster-VFL) algorithm, which integrates cluster intelligence to optimize heterogeneous distributed environments. In Cluster-VFL, each participant engages in training as an individual within the cluster, taking into account the utilization of non-aligned features. Cluster-VFL promotes model updates by identifying global optimal individuals and transferring global optimal knowledge. Subsequently, this knowledge is merged with the individual optimal knowledge obtained from local training of each participant. We conducted extensive experiments using an open-source diagnostic dataset and a proprietary dataset from Company A. The results unequivocally demonstrate that this algorithm enhances participants’ learning abilities, improves their communication efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Algorithm 3
Algorithm 4
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Jinying Li MS, Maiti Ananda, Gray T (2020) Blockchain for supply chain quality management: challenges and opportunities in context of open manufacturing and industrial internet of things. Int J Comput Integr Manuf 33(12):1321–1355. https://doi.org/10.1080/0951192X.2020.1815853

    Article  Google Scholar 

  2. Lavelli V, Beccalli MP (2022) Cheese whey recycling in the perspective of the circular economy: modeling processes and the supply chain to design the involvement of the small and medium enterprises. Trends Food Sci Technol 126:86–98. https://doi.org/10.1016/j.tifs.2022.06.013

    Article  Google Scholar 

  3. Kwon S, Monnier LV, Barbau R, Bernstein WZ (2020) Enriching standards-based digital thread by fusing as-designed and as-inspected data using knowledge graphs. Adv Eng Inform 46:101102. https://doi.org/10.1016/j.aei.2020.101102

    Article  Google Scholar 

  4. Amalfitano D, De Simone V, Maietta RR, Scala S, Fasolino AR (2019) Using tool integration for improving traceability management testing processes: An automotive industrial experience. J Softw Evolut Process 31(6):2171

    Article  Google Scholar 

  5. Song C, Wu Z, Gray J, Meng Z (2024) An rfid-powered multisensing fusion industrial iot system for food quality assessment and sensing. IEEE Trans Ind Inform 20(1):337–348. https://doi.org/10.1109/TII.2023.3262197

    Article  Google Scholar 

  6. Mulyasari D, Wahyuningtyas R, Alamsyah A (2023) Blockchain technology for privacy protection in healthcare industry. In: 2023 IEEE International Biomedical Instrumentation and Technology Conference (IBITeC), pp. 86–91. https://doi.org/10.1109/IBITeC59006.2023.10390975

  7. Liu Y, Kang Y, Zou T, Pu Y, He Y, Ye X, Ouyang Y, Zhang Y-Q, Yang Q (2024) Vertical federated learning: concepts, advances, and challenges. IEEE Trans Knowl Data Eng. https://doi.org/10.1109/TKDE.2024.3352628

    Article  Google Scholar 

  8. Huang L, Li Z, Sun J, Zhao H (2024) Coresets for vertical federated learning: regularized linear regression and k-means clustering. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. NIPS ’22. Curran Associates Inc., Red Hook, NY, USA

  9. Chen S, Yang J, Wang G, Wang Z, Yin H, Feng Y (2024) Clfldp: communication-efficient layer clipping federated learning with local differential privacy. J Syst Arch 148:103067. https://doi.org/10.1016/j.sysarc.2024.103067

    Article  Google Scholar 

  10. Ribero M, Vikalo H (2024) Reducing communication in federated learning via efficient client sampling. Patt Recogn 148:110122. https://doi.org/10.1016/j.patcog.2023.110122

    Article  Google Scholar 

  11. Liu P, Zhu G, Jiang W, Luo W, Xu J, Cui S (2022) Vertical federated edge learning with distributed integrated sensing and communication. IEEE Commun Lett 26(9):2091–2095. https://doi.org/10.1109/LCOMM.2022.3181612

    Article  Google Scholar 

  12. Reisizadeh A, Mokhtari A, Hassani H, Jadbabaie A, Pedarsani R (2020) Fedpaq: a communication-efficient federated learning method with periodic averaging and quantization. Proc Mach Learn Res 108:2021–2031

    Google Scholar 

  13. Lin Y, Han S, Mao H, Wang Y, Dally WJ (2018) Deep Gradient Compression: Reducing the communication bandwidth for distributed training. In: The International Conference on Learning Representations

  14. Tao Z, Li Q (2018) eSGD: communication efficient distributed deep learning on the edge. In: USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18). USENIX Association, Boston, MA. https://www.usenix.org/conference/hotedge18/presentation/tao

  15. Li W, Wu Z, Chen T, Li L, Ling Q (2022) Communication-censored distributed stochastic gradient descent. IEEE Trans Neural Netw Learn Syst 33(11):6831–6843. https://doi.org/10.1109/TNNLS.2021.3083655

    Article  MathSciNet  Google Scholar 

  16. Fletcher PT, Venkatasubramanian S, Joshi S (2008) Robust statistics on riemannian manifolds via the geometric median. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. DOI:https://doi.org/10.1109/CVPR.2008.4587747

  17. Yang Z, Sun Q (2022) Communication-efficient federated learning with cooperative filter selection. In: 2022 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 2172–2176. DOI:https://doi.org/10.1109/ISCAS48785.2022.9937667

  18. Dorigo M, Theraulaz G, Trianni V (2021) Swarm robotics: past, present, and future [point of view]. Proc IEEE 109(7):1152–1165. https://doi.org/10.1109/JPROC.2021.3072740

    Article  Google Scholar 

  19. Park S, Suh Y, Lee J (2021) Fedpso: federated learning using particle swarm optimization to reduce communication costs. Sensors 21(2):600. https://doi.org/10.3390/s21020600

    Article  Google Scholar 

  20. Neto HNC, Dusparic I, Mattos DMF, Fernandes NC (2022) Fedsa: Accelerating intrusion detection in collaborative environments with federated simulated annealing. 2022 IEEE 8th International Conference on Network Softwarization (NetSoft), 420–428

  21. McMahan B, Moore E, Ramage D, Hampson S, Arcas BAy (2017) Communication-Efficient Learning of Deep Networks from Decentralized Data. In: Singh, A, Zhu, J. (eds.) Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 54, pp. 1273–1282. PMLR. https://proceedings.mlr.press/v54/mcmahan17a.html

  22. Li T, Sahu AK, Zaheer M, Sanjabi M, Talwalkar A, Smith V (2020) Federated optimization in heterogeneous networks. Proc Mach Learn Syst 2:429–450

    Google Scholar 

  23. Xie C, Koyejo O, Gupta I (2019) Asynchronous federated optimization. ArXiv:1903.03934

  24. Sun R, Li Y, Shah T, Sham RWH, Szydlo T, Qian B, Thakker D, Ranjan R (2022) Fedmsa: a model selection and adaptation system for federated learning. Sensors 22(19):7244. https://doi.org/10.3390/s22197244

    Article  Google Scholar 

  25. Tang Z, Zhang Y, Shi S, He X, Han B, Chu X (2022) Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In: Chaudhuri K, Jegelka S, Song L, Szepesvari C, Niu G, Sabato S (eds.) Proceedings of the 39th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 162, pp. 21111–21132. PMLR. https://proceedings.mlr.press/v162/tang22d.html

  26. Zhu Z, Hong J, Zhou J (2021) Data-free knowledge distillation for heterogeneous federated learning. Proc Mach Learn Res 139:12878–12889

    Google Scholar 

  27. Murata T, Suzuki T (2021) Bias-variance reduced local sgd for less heterogeneous federated learning. In: Meila, M, Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 7872–7881. PMLR. https://proceedings.mlr.press/v139/murata21a.html

  28. Feng H, You Z, Chen M, Zhang T, Zhu M, Wu F, Wu C, Chen W (2021) Kd3a: Unsupervised multi-source decentralized domain adaptation via knowledge distillation. In: Meila, M, Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 3274–3283. PMLR. https://proceedings.mlr.press/v139/feng21f.html

  29. McMahan H.B, Moore E, Ramage D, Hampson S, Arcas BA (2016) Communication-efficient learning of deep networks from decentralized data. In: International Conference on Artificial Intelligence and Statistics

  30. Połap D, Woźniak M (2021) Meta-heuristic as manager in federated learning approaches for image processing purposes. Appl Soft Comput 113:107872. https://doi.org/10.1016/j.asoc.2021.107872

    Article  Google Scholar 

  31. Qolomany B, Ahmad K, Al-Fuqaha A, Qadir J (2020) Particle swarm optimized federated learning for industrial iot and smart city services. In: Global Communications Conference, pp. 1–6

  32. Victor N, Bhattacharya S, Maddikunta PKR, Alotaibi FM, Gadekallu TR, Jhaveri RH (2023) Fl-pso: A federated learning approach with particle swarm optimization for brain stroke prediction. 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing Workshops (CCGridW), pp 33–38

  33. Liu Y, Kang Y, Zou T, Pu Y, He Y, Ye X, Ouyang Y, Zhang Y-Q, Yang Q (2022) Vertical federated learning

  34. Liu Y, Zhang X, Kang Y, Li L, Chen T, Hong M, Yang Q (2022) Fedbcd: a communication-efficient collaborative learning framework for distributed features. IEEE Trans Signal Process 70:4277–4290. https://doi.org/10.1109/TSP.2022.3198176

    Article  MathSciNet  Google Scholar 

  35. Kang Y, Liu Y, Chen T (2020) Fedmvt: semi-supervised vertical federated learning with multiview training. ArXiv:abs/2008.10838

  36. Kang Y, Liu Y, Liang X (2022) Fedcvt: semi-supervised vertical federated learning with cross-view training. ACM Trans Intell Syst Technol 13:1–6. https://doi.org/10.1145/3510031

    Article  Google Scholar 

  37. Sun R, Zhang Y, Shah T, Sun J, Zhang S, Li W, Duan H, Wei B, Ranjan R (2024) From sora what we can see: a survey of text-to-video generation. arXiv preprint arXiv:

Download references

Acknowledgements

The authors would like to thank the anonymous referees for their useful comments. This work is supported by the Science and Technology Innovation 2030 - “New Generation Artificial Intelligence” Major Project, China (2020AAA0109300) and Shanghai Science and Technology Commission, the Shanghai Science and Technology Commission for their generous funding support through project number (22DZ2205600) and Meta-knowledge Driven Multimodal Federated Self-Supervised Learning Algorithm in Intelligent Manufacturing Scenarios (62376151).

Author information

Authors and Affiliations

  1. Shanghai University of Engineering Science, 333 Longteng Road, Shanghai, 201620, China

    Zilong Yin, Xiaoli Zhao, Haoyu Wang & Xin Zhang

  2. Donghua University, 2999 North Renmin Road, Shanghai, 201620, China

    Zhijun Fang

  3. Sanda University, 2727 Jinhai Road, Shanghai, 201209, China

    Xin Guo

Authors
  1. Zilong Yin
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Xiaoli Zhao
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Haoyu Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Xin Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Xin Guo
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Zhijun Fang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Xiaoli Zhao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yin, Z., Zhao, X., Wang, H. et al. Cluster knowledge-driven vertical federated learning. J Supercomput 80, 20229–20252 (2024). https://doi.org/10.1007/s11227-024-06232-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-024-06232-4

Keywords