Skip to main content
SpringerLink
Log in

FedUTN: federated self-supervised learning with updating target network

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Self-supervised learning (SSL) is capable of learning noteworthy representations from unlabeled data, which has mitigated the problem of insufficient labeled data to a certain extent. The original SSL method centered on centralized data, but the growing awareness of privacy protection restricts the sharing of decentralized, unlabeled data generated by a variety of mobile devices, such as cameras, phones, and other terminals. Federated Self-supervised Learning (FedSSL) is the result of recent efforts to create Federated learning, which is always used for supervised learning using SSL. Informed by past work, we propose a new FedSSL framework, FedUTN. This framework aims to permit each client to train a model that works well on both independent and identically distributed (IID) and independent and non-identically distributed (non-IID) data. Each party possesses two asymmetrical networks, a target network and an online network. FedUTN first aggregates the online network parameters of each terminal and then updates the terminals’ target network with the aggregated parameters, which is a radical departure from the update technique utilized in earlier studies. In conjunction with this method, we offer a novel control algorithm to replace EMA for the training operation. After extensive trials, we demonstrate that: (1) the feasibility of utilizing the aggregated online network to update the target network. (2) FedUTN’s aggregation strategy is simpler, more effective, and more robust. (3) FedUTN outperforms all other prevalent FedSSL algorithms and outperforms the SOTA algorithm by 0.5%\(\sim \) 1.6% under regular experiment con1ditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Algorithm 1
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Ashish V, Noam S, Niki P, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is All you Need. In: NIPS 2017: pp: 5998–6008

  2. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al. (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929

  3. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 10012– 10022

  4. Touvron H, Cord M, Douze M, Massa F, Sablayrolles A, Jégou H (2021) Training data-efficient image transformers & distillation through attention. In: International conference on machine learning, PMLR, pp 10347–10357

  5. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770– 778

  6. Doersch C, Gupta A, Efros AA (2015) Unsupervised visual representation learning by context prediction. In: Proceedings of the IEEE international conference on computer vision, pp 1422–1430

  7. Zhang R, Isola P, Efros AA (2016) Colorful image colorization. In: European conference on computer vision, Springer, pp 649–666. https://doi.org/10.1007/978-3-319-46487-9_40

  8. Noroozi M, Favaro P (2016) Unsupervised learning of visual representations by solving jigsaw puzzles. In: European conference on computer vision, Springer, pp 69–84. https://doi.org/10.1007/978-3-319-46466-4_5

  9. Gidaris S, Singh P, Komodakis N (2018) Unsupervised representation learning by predicting image rotations. arXiv:1803.07728

  10. Chen T, Kornblith S, Norouzi M, Hinton G (2020) A simple framework for contrastive learning of visual representations. In: International conference on machine learning, PMLR, pp 1597–1607

  11. He K, Fan H, Wu Y, Xie S, Girshick R (2020) Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  12. Grill J-B, Strub F, Altché F, Tallec C, Richemond P, Buchatskaya E, Doersch C, Avila Pires B, Guo Z, Gheshlaghi Azar M et al (2020) Bootstrap your own latent-a new approach to self-supervised learning. Adv Neural Inf Process Syst 33:21271–21284

    Google Scholar 

  13. Konečnỳ J, McMahan HB, Ramage D, Richtárik P (2016)

  14. McMahan HB, Moore E, Ramage DY, Arcas BA (2016)

  15. Reisizadeh A, Mokhtari A, Hassani H, Jadbabaie A, Pedarsani R (2020) Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. In: International conference on artificial intelligence and statistics, PMLR, pp 2021–2031

  16. Hamer J, Mohri M, Suresh AT (2020) Fedboost: A communication-efficient algorithm for federated learning. In: International conference on machine learning, PMLR, pp 3973–3983

  17. Duan M, Liu D, Chen X, Liu R, Tan Y, Liang L (2020) Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Trans Parallel Distrib Syst 32(1):59–71

    Article  Google Scholar 

  18. Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-training

  19. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9

    Google Scholar 

  20. Devlin J, Chang M-W, Lee K, Toutanova K (2018)

  21. Komodakis N, Gidaris S (2018) Unsupervised representation learning by predicting image rotations. In: International conference on learning representations (ICLR)

  22. He K, Chen X, Xie S, Li Y, Dollár P, Girshick R (2022) Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 16000–16009

  23. Li T, Sahu AK, Zaheer M, Sanjabi M, Talwalkar A, Smith V (2020) Federated optimization in heterogeneous networks. Proc Mach Learn Syst 2:429–450

    Google Scholar 

  24. Jin Y, Wei X, Liu Y, Yang Q (2020) A survey towards federated semi-supervised learning

  25. Zhang Z, Yao Z, Yang Y, Yan Y, Gonzalez JE, Mahoney MW (2020) Benchmarking semi-supervised federated learning. arXiv:2008.11364 17:

  26. Jeong W, Yoon J, Yang E, Hwang SJ (2020) Federated semi-supervised learning with inter-client consistency & disjoint learning. arXiv:2006.12097

  27. van Berlo B, Saeed A, Ozcelebi T (2020) Towards federated unsupervised representation learning. In: Proceedings of the third ACM international workshop on edge systems, analytics and networking, pp 31–36

  28. Zhang F, Kuang K, You Z, Shen T, Xiao J, Zhang Y, Wu C, Zhuang Y, Li X (2020) Federated unsupervised representation learning. arXiv:2010.08982

  29. Zhuang W, Gan X, Wen Y, Zhang S, Yi S (2021) Collaborative unsupervised visual representation learning from decentralized data. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 4912–4921

  30. Zhao Y, Li M, Lai L, Suda N, Civin D, Chandra V (2018) Federated learning with non-iid data. arXiv:1806.00582

  31. Chen X, He K (2021) Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 15750–15758

  32. Kolesnikov A, Zhai X, Beyer L (2019) Revisiting self-supervised visual representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1920–1929

  33. Zhai X, Oliver A, Kolesnikov A, Beyer L (2019) S4l: self-supervised semi-supervised learning. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 1476–1485

Download references

Funding

Partial financial support was received from the National Key Research and Development Program (No. 2018YFB21 00100).

Author information

Authors and Affiliations

  1. State Key Laboratory of Power Transmission Equipment & System Security and New Technology, Chongqing University, ShazhengJie, 400044, Chongqing, China

    Simou Li, Yuxing Mao, Jian Li, Yihang Xu, Jinsen Li, Xueshuo Chen & Siyang Liu

  2. Electric Power Research Institute, Yunnan Power Grid Co., Ltd, Yundaxilu, Kunming, 650217, Yunnan, China

    Siyang Liu & Xianping Zhao

Authors
  1. Simou Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuxing Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jian Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yihang Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jinsen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xueshuo Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Siyang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xianping Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Simou Li or Yuxing Mao.

Ethics declarations

Competing interests

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Jian Li, Yihang Xu, Jinsen Li, Xueshuo Chen, Siyang Liu and Xianping Zhao are contributed equally to this work.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, S., Mao, Y., Li, J. et al. FedUTN: federated self-supervised learning with updating target network. Appl Intell 53, 10879–10892 (2023). https://doi.org/10.1007/s10489-022-04070-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04070-6

Keywords

Search

Navigation