Skip to main content
SpringerLink
Log in

Evil vs evil: using adversarial examples to against backdoor attack in federated learning

  • Special Issue Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

As a distributed learning paradigm, federated learning (FL) has shown great success in aggregating information from different clients to train a shared global model. Unfortunately, by uploading carefully crafted updated models, a malicious client can embed a backdoor into the global model during FL’s training. Numerous secure aggregation strategies and robust training protocols have been proposed to defend FL against backdoor attacks. However, they are still challenged, either being bypassed by adaptive attacks or sacrificing the main task performance of FL. By conducting empirical studies of backdoor attacks in FL, we gain an interesting insight that adversarial perturbations can activate backdoors in backdoor models. Consequently, behavior differences of models fed by adversarial examples are compared for backdoor update detection. We propose a novel FL backdoor defense method using adversarial examples, denoted as \(\underline{E}vil\,\underline{v}s\, \underline{E}vil\) (EVE). Specifically, a small data set of clean examples for FL’s main task training is collected in the sever for adversarial examples generation. By observing the behavior of updated models under the adversarial examples, EVE uses a clustering algorithm to select benign models and to exclude the other models, without any loss of the main task performance of FL itself. Extensive evaluations across four data sets and the corresponding DNNs demonstrate the state-of-the-art (SOTA) defense performance of EVE compared with five baselines. In particular, EVE under 40% of malicious clients can reduce the attack success rate from 99% to 1%. In addition, we verify that EVE is still robust under the adaptive attacks. EVE is open sourced to facilitate future research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Li, L., Fan, Y., Tse, M., Lin, K.: A review of applications in federated learning. Comput. Ind. Eng. 149, 106854 (2020). https://doi.org/10.1016/j.cie.2020.106854

    Article  Google Scholar 

  2. Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: challenges, methods, and future directions. IEEE Signal Process. Mag. 37(3), 50–60 (2020). https://doi.org/10.1109/MSP.2020.2975749

    Article  Google Scholar 

  3. Yang, Q., Liu, Y., Chen, T., Tong, Y.: Federated machine learning: concept and applications. ACM Trans. Intell. Syst. Technol. 10(2), 12–11219 (2019). https://doi.org/10.1145/3298981

    Article  Google Scholar 

  4. Aledhari, M., Razzak, R., Parizi, R.M., Saeed, F.: Federated learning: a survey on enabling technologies, protocols, and applications. IEEE Access 8, 140699–140725 (2020). https://doi.org/10.1109/ACCESS.2020.3013541

    Article  Google Scholar 

  5. Li, Q., Wen, Z., Wu, Z., Hu, S., Wang, N., Liu, X., He, B.: A survey on federated learning systems: vision, hype and reality for data privacy and protection. CoRR (2019) arxiv:1907.09693

  6. Choudhury, O., Park, Y., Salonidis, T., Gkoulalas-Divanis, A., Sylla, I.: Predicting adverse drug reactions on distributed health data using federated learning. AMIA Annual symposium., November 16-20, 2019, Washington, DC, USA (2019). https://knowledge.amia.org/69862-amia-1.4570936/t004-1.4574923/t004-1.4574924/3200032-1.4575138/3203560-1.4575135

  7. Xu, J., Glicksberg, B.S., Su, C., Walker, P.B., Bian, J., Wang, F.: Federated learning for healthcare informatics. J. Heal. Inform. Res. 5(1), 1–19 (2021). https://doi.org/10.1007/s41666-020-00082-4

    Article  Google Scholar 

  8. Mothukuri, V., Parizi, R.M., Pouriyeh, S., Huang, Y., Dehghantanha, A., Srivastava, G.: A survey on security and privacy of federated learning. Future Gener. Comput. Syst. 115, 619–640 (2021). https://doi.org/10.1016/j.future.2020.10.007

    Article  Google Scholar 

  9. Ma, C., Li, J., Ding, M., Yang, H.H., Shu, F., Quek, T.Q.S., Poor, H.V.: On safeguarding privacy and security in the framework of federated learning. IEEE Netw. 34(4), 242–248 (2020). https://doi.org/10.1109/MNET.001.1900506

    Article  Google Scholar 

  10. Wang, H., Sreenivasan, K., Rajput, S., Vishwakarma, H., Agarwal, S., Sohn, J., Lee, K., Papailiopoulos, D.S.: Attack of the tails: Yes, you really can backdoor federated learning. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual (2020). https://proceedings.neurips.cc/paper/2020/hash/b8ffa41d4e492f0fad2f13e29e1762eb-Abstract.html

  11. Sun, Z., Kairouz, P., Suresh, A.T., McMahan, H.B.: Can you really backdoor federated learning? CoRR (2019) arxiv:1911.07963

  12. Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., Shmatikov, V.: How to backdoor federated learning. In: Chiappa, S., Calandra, R. (eds.) The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy]. Proceedings of Machine Learning Research, vol. 108, pp. 2938–2948 (2020). http://proceedings.mlr.press/v108/bagdasaryan20a.html

  13. Özdayi, M.S., Kantarcioglu, M., Gel, Y.R.: Defending against backdoors in federated learning with robust learning rate. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 9268–9276 (2021). https://ojs.aaai.org/index.php/AAAI/article/view/17118

  14. Yin, D., Chen, Y., Ramchandran, K., Bartlett, P.L.: Byzantine-robust distributed learning: towards optimal statistical rates. In: Dy, J.G., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018. Proceedings of Machine Learning Research, vol. 80, pp. 5636–5645 (2018). http://proceedings.mlr.press/v80/yin18a.html

  15. Blanchard, P., Mhamdi, E.M.E., Guerraoui, R., Stainer, J.: Machine learning with adversaries: Byzantine tolerant gradient descent. In: Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 119–129 (2017). https://proceedings.neurips.cc/paper/2017/hash/f4b9ec30ad9f68f89b29639786cb62ef-Abstract.html

  16. Pillutla, V.K., Kakade, S.M., Harchaoui, Z.: Robust aggregation for federated learning. CoRR (2019) arxiv:1912.13445

  17. Fung, C., Yoon, C.J.M., Beschastnikh, I.: The limitations of federated learning in sybil settings. In: Egele, M., Bilge, L. (eds.) 23rd International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2020, San Sebastian, Spain, October 14-15, 2020, pp. 301–316 (2020). https://www.usenix.org/conference/raid2020/presentation/fung

  18. Xie, C., Chen, M., Chen, P., Li, B.: CRFL: certifiably robust federated learning against backdoor attacks. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event. Proceedings of Machine Learning Research, vol. 139, pp. 11372–11382 (2021). http://proceedings.mlr.press/v139/xie21a.html

  19. Andreina, S., Marson, G.A., Möllering, H., Karame, G.: Baffle: Backdoor detection via feedback-based federated learning. In: 41st IEEE International Conference on Distributed Computing Systems, ICDCS 2021, Washington DC, USA, July 7-10, 2021, pp. 852–863 (2021). https://doi.org/10.1109/ICDCS51616.2021.00086

  20. Xie, C., Huang, K., Chen, P., Li, B.: DBA: distributed backdoor attacks against federated learning. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 (2020). https://openreview.net/forum?id=rkgyS0VFvr

  21. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings (2018). https://openreview.net/forum?id=rJzIBfZAb

  22. Krizhevsky A, Hinton G. Learning multiple layers of features from tiny images. Technical report, University of Toronto, pp. 1–60 (2009).

  23. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  24. Konečný, J., McMahan, H.B., Yu, F.X., Richtárik, P., Suresh, A.T., Bacon, D.: Federated learning: Strategies for improving communication efficiency. CoRR (2016) arxiv:1610.05492

  25. McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Singh, A., Zhu, X.J. (eds.) Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA. Proceedings of Machine Learning Research, vol. 54, pp. 1273–1282 (2017). http://proceedings.mlr.press/v54/mcmahan17a.html

  26. Lyu, L., Yu, H., Zhao, J., Yang, Q.: Threats to federated learning. In: Yang, Q., Fan, L., Yu, H. (eds.) Federated Learning - Privacy and Incentive. Lecture Notes in Computer Science, vol. 12500, pp. 3–16 (2020). https://doi.org/10.1007/978-3-030-63076-8_1

  27. Bhagoji, A.N., Chakraborty, S., Mittal, P., Calo, S.B.: Analyzing federated learning through an adversarial lens. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA. Proceedings of Machine Learning Research, vol. 97, pp. 634–643 (2019). http://proceedings.mlr.press/v97/bhagoji19a.html

  28. Chen, Y., Su, L., Xu, J.: Distributed statistical machine learning in adversarial settings: byzantine gradient descent. In: Psounis, K., Akella, A., Wierman, A. (eds.) Abstracts of the 2018 ACM International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS 2018, Irvine, CA, USA, June 18-22, 2018, p. 96 (2018). https://doi.org/10.1145/3219617.3219655

  29. Mhamdi, E.M.E., Guerraoui, R., Rouault, S.: The hidden vulnerability of distributed learning in byzantium. In: Dy, J.G., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018. Proceedings of Machine Learning Research, vol. 80, pp. 3518–3527 (2018). http://proceedings.mlr.press/v80/mhamdi18a.html

  30. Fu, S., Xie, C., Li, B., Chen, Q.: Attack-resistant federated learning with residual-based reweighting. CoRR (2019) arxiv:1912.11464

  31. Zhao, L., Hu, S., Wang, Q., Jiang, J., Shen, C., Luo, X., Hu, P.: Shielding collaborative learning: mitigating poisoning attacks through client-side detection. IEEE Trans. Dependable Secur. Comput. 18(5), 2029–2041 (2021). https://doi.org/10.1109/TDSC.2020.2986205

    Article  Google Scholar 

  32. Gu, T., Liu, K., Dolan-Gavitt, B., Garg, S.: Badnets: evaluating backdooring attacks on deep neural networks. IEEE Access 7, 47230–47244 (2019). https://doi.org/10.1109/ACCESS.2019.2909068

    Article  Google Scholar 

  33. Cao, X., Fang, M., Liu, J., Gong, N.Z.: Fltrust: Byzantine-robust federated learning via trust bootstrapping. In: 28th Annual Network and Distributed System Security Symposium, NDSS 2021, Virtually, February 21-25, 2021 (2021). https://www.ndss-symposium.org/ndss-paper/fltrust-byzantine-robust-federated-learning-via-trust-bootstrapping/

  34. Tzortzis, G., Likas, A.: The global kernel \(k\)-means algorithm for clustering in feature space. IEEE Trans. Neural Netw. 20(7), 1181–1194 (2009). https://doi.org/10.1109/TNN.2009.2019722

    Article  Google Scholar 

  35. Frey, B.J., Dueck, D.: Clustering by passing messages between data points. Science 315(5814), 972–976 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  36. Chang, T., He, Y., Li, P.: Efficient two-step adversarial defense for deep neural networks. CoRR (2018) arxiv:1810.03739

  37. Tramèr, F., Boneh, D.: Adversarial training and robustness for multiple perturbations. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5858–5868 (2019). https://proceedings.neurips.cc/paper/2019/hash/5d4ae76f053f8f2516ad12961ef7fe97-Abstract.html

  38. Liu, Y., Lee, W., Tao, G., Ma, S., Aafer, Y., Zhang, X.: ABS: scanning neural networks for back-doors by artificial brain stimulation. In: Cavallaro, L., Kinder, J., Wang, X., Katz, J. (eds.) Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS 2019, London, UK, November 11-15, 2019, pp. 1265–1282 (2019). https://doi.org/10.1145/3319535.3363216

  39. Chen, B., Carvalho, W., Baracaldo, N., Ludwig, H., Edwards, B., Lee, T., Molloy, I.M., Srivastava, B.: Detecting backdoor attacks on deep neural networks by activation clustering. In: Espinoza, H., hÉigeartaigh, S.Ó., Huang, X., Hernández-Orallo, J., Castillo-Effen, M. (eds.) Workshop on Artificial Intelligence Safety 2019 Co-located with the Thirty-Third AAAI Conference on Artificial Intelligence 2019 (AAAI-19), Honolulu, Hawaii, January 27, 2019. CEUR Workshop Proceedings, vol. 2301 (2019). http://ceur-ws.org/Vol-2301/paper_18.pdf

  40. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998). https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  41. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR (2017) arxiv:1708.07747

  42. Huang, G.B., Mattar, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database forstudying face recognition in unconstrained environments. In: Workshop on Faces in’Real-Life’Images: Detection, Alignment, and Recognition (2008)

  43. Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., Chandra, V.: Federated learning with non-iid data. CoRR (2018) arxiv:1806.00582

  44. Li, X., Huang, K., Yang, W., Wang, S., Zhang, Z.: On the convergence of fedavg on non-iid data. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 (2020). https://openreview.net/forum?id=HJxNAnVtDS

  45. Li, Y., Li, Y., Lv, Y., Jiang, Y., Xia, S.: Hidden backdoor attack against semantic segmentation models. CoRR (2021) arxiv:2103.04038

  46. Lin, J., Xu, L., Liu, Y., Zhang, X.: Composite backdoor attack for deep neural network by mixing existing benign features. In: Ligatti, J., Ou, X., Katz, J., Vigna, G. (eds.) CCS ’20: 2020 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, USA, November 9-13, 2020, pp. 113–131 (2020). https://doi.org/10.1145/3372297.3423362

  47. Brendel, W., Rauber, J., Bethge, M.: Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. In: 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings (2018). https://openreview.net/forum?id=SyZI0GWCZ

  48. Wong, E., Rice, L., Kolter, J.Z.: Fast is better than free: revisiting adversarial training. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 (2020). https://openreview.net/forum?id=BJx040EFvH

  49. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015). arxiv:1412.6572

  50. Kurakin, A., Goodfellow, I.J., Bengio, S.: Adversarial examples in the physical world. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings (2017). https://openreview.net/forum?id=HJGU3Rodl

  51. Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., Li, J.: Boosting adversarial attacks with momentum. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 9185–9193. Computer Vision Foundation / IEEE Computer Society (2018). https://doi.org/10.1109/CVPR.2018.00957. http://openaccess.thecvf.com/content_cvpr_2018/html/Dong_Boosting_Adversarial_Attacks_CVPR_2018_paper.html

  52. Fang, M., Cao, X., Jia, J., Gong, N.Z.: Local model poisoning attacks to byzantine-robust federated learning. In: Capkun, S., Roesner, F. (eds.) 29th USENIX Security Symposium, USENIX Security 2020, August 12-14, 2020, pp. 1605–1622 (2020). https://www.usenix.org/conference/usenixsecurity20/presentation/fang

Download references

Acknowledgements

This research was supported by the National Natural Science Foundation of China (No. 62072406), the National Key R &D Projects of China (No. 2018AAA0100801), the Key R &D Projects in Zhejiang Province (No. 2021C01117), the 2020 Industrial Internet Innovation Development Project (No. TC200H01V), “Ten Thousand Talents Program” in Zhejiang Province (No. 2020R52011), and the Key Lab of Ministry of Public Security (No. 2020DSJSYS001).

Author information

Authors and Affiliations

  1. College of Information Engineering, Zhejiang University of Technology, Hangzhou, 310023, China

    Tao Liu, Mingjun Li, Haibin Zheng & Jinyin Chen

  2. Institute of Cyberspace Security, Zhejiang University of Technology, Hangzhou, 310023, China

    Haibin Zheng & Jinyin Chen

  3. College of Computer and Computing Science, Zhejiang University City College, Hangzhou, 310015, China

    Zhaoyan Ming

Authors
  1. Tao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingjun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haibin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhaoyan Ming
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jinyin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinyin Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, T., Li, M., Zheng, H. et al. Evil vs evil: using adversarial examples to against backdoor attack in federated learning. Multimedia Systems 29, 553–568 (2023). https://doi.org/10.1007/s00530-022-00965-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-022-00965-z

Keywords

Search

Navigation