Abstract
Federated Learning (FL) is a machine learning paradigm where multiple data owners collaboratively train a model under the coordination of a central server, while keeping all data decentralized. Such a paradigm allows models to be trained effectively while avoiding data privacy leakage. However, federated learning is vulnerable to various kinds of failures as a result of both intentional (malicious) and none intentional (non-malicious) attacks. Existing studies on attacks in federated learning are mostly dedicated to the automatic defense against malicious attacks (e.g., data poisoning attacks). Relatively, less attention has been given to handling non-malicious failures (e.g., non-independent and identically distributed data failures), which are actually more common and difficult-to-handle in a federated learning setting. In this paper, we propose FedCare, a real-time visual diagnosis approach for handling failures in federated learning systems. The functionality of FedCare includes the identification of failures, the assessment of their nature (malicious or non-malicious), the study of their impact, and the recommendation of adequate defense strategies. Our design is multi-faceted, giving perspectives from the angles of model performance, anomaly/contribution assessment of clients, features maps, group activities, and client impact. We demonstrate the effectiveness of our approach through two case studies, a quantitative experiment and an expert interview.
Similar content being viewed by others
References
Yang Q, Liu Y, Chen T, Tong Y. Federated machine learning: concept and applications. ACM Transactions on Intelligent Systems and Technology, 2019, 10(2): 12
Regulation G D P. Regulation (EU) 2016/679 of the European parliament and of the council of 27 April 2016. Official Journal of the European Union, 2016
Kairouz P, McMahan H B, Avent B, Bellet A, Bennis M, Bhagoji A N, Bonawitz K, Charles Z, Cormode G, Cummings R, D’Oliveira R G L, Eichner H, El Rouayheb S, Evans D, Gardner J, Garrett Z, Gascón A, Ghazi B, Gibbons P B, Gruteser M, Harchaoui Z, He C, He L, Huo Z, Hutchinson B, Hsu J, Jaggi M, Javidi T, Joshi G, Khodak M, Konecný J, Korolova A, Koushanfar F, Koyejo S, Lepoint T, Liu Y, Mittal P, Mohri M, Nock R, Özgür A, Pagh R, Qi H, Ramage D, Raskar R, Raykova M, Song D, Song W, Stich S U, Sun Z, Suresh A T, Tramèr F, Vepakomma P, Wang J, Xiong L, Xu Z, Yang Q, Yu F X, Yu H, Zhao S. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 2021, 14(1–2): 1–210
Yuan J, Chen C, Yang W, Liu M, Xia J, Liu S. A survey of visual analytics techniques for machine learning. Computational Visual Media, 2021, 7(1): 3–36
Blanchard P, El Mhamdi E M, Guerraoui R, Stainer J. Machine learning with adversaries: byzantine tolerant gradient descent. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 118–128
Shen S, Tople S, Saxena P. Auror: defending against poisoning attacks in collaborative deep learning systems. In: Proceedings of the 32nd Annual Conference on Computer Security Applications. 2016, 508–519
McMahan H B, Moore E, Ramage D, Arcas B A Y. Federated learning of deep networks using model averaging. 2016, arXiv preprint arXiv: 1602.05629
Konečný J, McMahan H B, Ramage D, Richtárik P. Federated optimization: distributed machine learning for on-device intelligence. 2016, arXiv preprint arXiv: 1610.02527
Konečný J, McMahan H B, Yu F X, Richtárik P, Suresh A T, Bacon D. Federated learning: strategies for improving communication efficiency. 2016, arXiv preprint arXiv: 1610.05492
Yin D, Chen Y, Kannan R, Bartlett P. Byzantine-robust distributed learning: towards optimal statistical rates. In: Proceedings of the 35th International Conference on Machine Learning. 2018, 5636–5645
Chen Y, Su L, Xu J. Distributed statistical machine learning in adversarial settings: byzantine gradient descent. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 2017, 1(2): 44
Shen Y, Sanghavi S. Learning with bad training data via iterative trimmed loss minimization. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 5739–5748
Cretu G F, Stavrou A, Locasto M E, Stolfo S J, Keromytis A D. Casting out demons: sanitizing training data for anomaly sensors. In: Proceedings of 2008 IEEE Symposium on Security and Privacy. 2008, 81–95
Liu K, Dolan-Gavitt B, Garg S. Fine-pruning: defending against backdooring attacks on deep neural networks. In: Proceedings of the 21st International Symposium on Research in Attacks, Intrusions, and Defenses. 2018, 273–294
Wang B, Yao Y, Shan S, Li H, Viswanath B, Zheng H, Zhao B Y. Neural cleanse: identifying and mitigating backdoor attacks in neural networks. In: Proceedings of 2019 IEEE Symposium on Security and Privacy. 2019, 707–723
Madry A, Makelov A, Schmidt L, Tsipras D, Vladu A. Towards deep learning models resistant to adversarial attacks. In: Proceedings of the 6th International Conference on Learning Representations. 2018
Shafahi A, Najibi M, Ghiasi A, Xu Z, Dickerson J, Studer C, Davis L S, Taylor G, Goldstein T. Adversarial training for free! In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 302
Bonawitz K A, Eichner H, Grieskamp W, Huba D, Ingerman A, Ivanov V, Kiddon C, Konecný J, Mazzocchi S, McMahan B, Van Overveldt T, Petrou D, Ramage D, Roselander J. Towards federated learning at scale: system design. In: Proceedings of the 2nd Conference on Machine Learning and Systems. 2019
Peng X, Huang Z, Zhu Y, Saenko K. Federated adversarial domain adaptation. In: Proceedings of the 8th International Conference on Learning Representations. 2020
Augenstein S, McMahan H B, Ramage D, Ramaswamy S, Kairouz P, Chen M, Mathews R, Arcas B A Y. Generative models for effective ML on private, decentralized datasets. In: Proceedings of the 8th International Conference on Learning Representations. 2020
Mnih V, Hinton G. Learning to label aerial images from noisy data. In: Proceedings of the 29th International Conference on Machine Learning. 2012, 203–210
Natarajan N, Dhillon I S, Ravikumar P, Tewari A. Learning with noisy labels. In: Proceedings of the 26th International Conference on Neural Information Processing Systems. 2013, 1196–1204
Liu S, Wang X, Liu M, Zhu J. Towards better analysis of machine learning models: a visual analytics perspective. Visual Informatics, 2017, 1(1): 48–56
Hohman F, Kahng M, Pienta R, Chau D H. Visual analytics in deep learning: an interrogative survey for the next frontiers. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(8): 2674–2693
Spinner T, Schlegel U, Schäfer H, El-Assady M. explAIner: a visual analytics framework for interactive and explainable machine learning. IEEE Transactions on Visualization and Computer Graphics, 2020, 26(1): 1064–1074
Liu D, Cui W, Jin K, Guo Y, Qu H. DeepTracker: Visualizing the training process of convolutional neural networks. ACM Transactions on Intelligent Systems and Technology, 2019, 10(1): 6
Liu M, Shi J, Li Z, Li C, Zhu J, Liu S. Towards better analysis of deep convolutional neural networks. IEEE Transactions on Visualization and Computer Graphics, 2017, 23(1): 91–100
Chung S, Suh S, Park C, Kang K, Choo J, Kwon B C. ReVACNN: realtime visual analytics for convolutional neural network. In: Proceedings of the ACM SIGKDD Workshop on Interactive Data Exploration and Analytics. 2016
Wang J, Gou L, Zhang W, Yang H, Shen H W. DeepVID: deep visual interpretation and diagnosis for image classifiers via knowledge distillation. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(6): 2168–2180
Wang Q, Yuan J, Chen S, Su H, Qu H, Liu S. Visual genealogy of deep neural networks. IEEE Transactions on Visualization and Computer Graphics, 2020, 26(11): 3340–3352
Kahng M, Andrews P Y, Kalro A, Chau D H. ActiVis: visual exploration of industry-scale deep neural network models. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(1): 88–97
Kwon B C, Choi M J, Kim J T, Choi E, Kim Y B, Kwon S, Sun J, Choo J. RetainVis: visual analytics with interpretable and interactive recurrent neural networks on electronic medical records. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(1): 299–309
Strobelt H, Gehrmann S, Huber B, Pfister H, Rush A M. Visual analysis of hidden state dynamics in recurrent neural networks. 2016, arXiv preprint arXiv: 1606.07461
Kahng M, Thorat N, Chau D H, Viégas F B, Wattenberg M. GAN lab: understanding complex deep generative models using interactive visual experimentation. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(1): 310–320
Wang J, Gou L, Yang H, Shen H W. GANViz: a visual analytics approach to understand the adversarial game. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(6): 1905–1917
Zhao X, Wu Y, Lee D L, Cui W. iForest: interpreting random forests via visual analytics. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(1): 407–416
Ma Y, Fan A, He J, Nelakurthi A R, Maciejewski R. A visual analytics framework for explaining and diagnosing transfer learning processes. IEEE Transactions on Visualization and Computer Graphics, 2021, 27(2): 1385–1395
Krause J, Perer A, Bertini E. INFUSE: interactive feature selection for predictive modeling of high dimensional data. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(12): 1614–1623
Wang J, Gou L, Shen H W, Yang H. DQNViz: a visual analytics approach to understand deep Q-networks. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(1): 288–298
Bilal A, Jourabloo A, Ye M, Liu X, Ren L. Do convolutional neural networks learn class hierarchy? IEEE Transactions on Visualization and Computer Graphics, 2018, 24(1): 152–162
Zhang J, Wang Y, Molino P, Li L, Ebert D S. Manifold: a model-agnostic framework for interpretation and diagnosis of machine learning models. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(1): 364–373
Strobelt H, Gehrmann S, Behrisch M, Perer A, Pfister H, Rush A M. Seq2seq-Vis: a visual debugging tool for sequence-to-sequence models. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(1): 353–363
Cabrera Á A, Epperson W, Hohman F, Kahng M, Morgenstern J, Chau D H. FAIRVIS: visual analytics for discovering intersectional bias in machine learning. In: Proceedings of 2019 IEEE Conference on Visual Analytics Science and Technology. 2019, 46–56
Ahn Y, Lin Y R. FairSight: visual analytics for fairness in decision making. IEEE Transactions on Visualization and Computer Graphics, 2020, 26(1): 1086–1095
Liu S, Xiao J, Liu J, Wang X, Wu J, Zhu J. Visual diagnosis of tree boosting methods. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(1): 163–173
Ma Y, Xie T, Li J, Maciejewski R. Explaining vulnerabilities to adversarial machine learning through visual analytics. IEEE Transactions on Visualization and Computer Graphics, 2020, 26(1): 1075–1085
Ma Y, Xu J, Wu X, Wang F, Chen W. A visual analytical approach for transfer learning in classification. Information Sciences, 2017, 390: 54–69
Wang Q, Ming Y, Jin Z, Shen Q, Liu D, Smith M J, Veeramachaneni K, Qu H. ATMseer: increasing transparency and controllability in automated machine learning. In: Proceedings of 2019 CHI Conference on Human Factors in Computing Systems. 2019, 681
Murugesan S, Malik S, Du F, Koh E, Lai T M. DeepCompare: visual and interactive comparison of deep learning model performance. IEEE Computer Graphics and Applications, 2019, 39(5): 47–59
Yu W, Yang K, Bai Y, Yao H, Rui Y. Visualizing and comparing convolutional neural networks. 2014, arXiv preprint arXiv: 1412.6631
Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado G S, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mane D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viegas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X. TensorFlow: large-scale machine learning on heterogeneous distributed systems. 2016, arXiv preprint arXiv: 1603.04467
Zeng H, Haleem H, Plantaz X, Cao N, Qu H. CNNComparator: comparative analytics of convolutional neural networks. 2017, arXiv preprint arXiv: 1710.05285
Wei X, Li Q, Liu Y, Yu H, Chen T, Yang Q. Multi-agent visualization for explaining federated learning. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence. 2019, 6572–6574
Wang X, Chen W, Xia J, Wen Z, Zhu R, Schreck T. HetVis: a visual analysis approach for identifying data heterogeneity in horizontal federated learning. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(1): 310–319
Phong L T, Aono Y, Hayashi T, Wang L, Moriai S. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security, 2018, 13(5): 1333–1345
Shokri R, Shmatikov V. Privacy-preserving deep learning. In: Proceedings of the 53rd Annual Allerton Conference on Communication, Control, and Computing. 2015, 1310–1321
McMahan B, Moore E, Ramage D, Hampson S, Arcas B A y. Communication-efficient learning of deep networks from decentralized data. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. 2017, 1273–1282
Cao D, Chang S, Lin Z, Liu G, Sun D. Understanding distributed poisoning attack in federated learning. In: Proceedings of the 25h IEEE International Conference on Parallel and Distributed Systems. 2019, 233–239
Lin J, Du M, Liu J. Free-riders in federated learning: attacks and defenses. 2019, arXiv preprint arXiv: 1911.12560
Chen Y, Su L, Xu J. Distributed statistical machine learning in adversarial settings: byzantine gradient descent. In: Proceedings of 2018 ACM International Conference on Measurement and Modeling of Computer Systems. 2018, 96–96
Bagdasaryan E, Veit A, Hua Y, Estrin D, Shmatikov V. How to backdoor federated learning. In: Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics. 2020, 2938–2948
Beyer K S, Goldstein J, Ramakrishnan R, Shaft U. When is “nearest neighbor” meaningful? In: Proceedings of the 7th International Conference on Database Theory. 1999, 217–235
Zong B, Song Q, Min M R, Cheng W, Lumezanu C, Cho D K, Chen H. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In: Proceedings of the 6th International Conference on Learning Representations. 2018
Xie C, Koyejo S, Gupta I. Zeno: distributed stochastic gradient descent with suspicion-based fault-tolerance. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 6893–6901
van der Maaten L, Hinton G. Visualizing data using t-SNE. Journal of Machine Learning Research, 2008, 9(86): 2579–2605
Marsland S. Machine Learning - An Algorithmic Perspective. New York: Chapman and Hall, 2009
Shannon C E. A mathematical theory of communication. ACM SIGMOBILE Mobile Computing and Communications Review, 2001, 5(1): 3–55
Fruchterman T M J, Reingold E M. Graph drawing by force-directed placement. Software: Practice and Experience, 1991, 21(11): 1129–1164
Zhou K, Yang Y, Qiao Y, Xiang T. Domain adaptive ensemble learning. IEEE Transactions on Image Processing, 2021, 30: 8008–8018
LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278–2324
Ganin Y, Lempitsky V. Unsupervised domain adaptation by backpropagation. In: Proceedings of the 32nd International Conference on Machine Learning. 2015, 1180–1189
Netzer Y, Wang T, Coates A, Bissacco A, Wu B, Ng A Y. Reading digits in natural images with unsupervised feature learning. In: Prpceedings of NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011
Caldas S, Duddu S M K, Wu P, Li T, Konečný J, McMahan H B, Smith V, Talwalkar A. LEAF: a benchmark for federated settings. 2018, arXiv preprint arXiv: 1812.01097
LeCun Y, Boser B, Denker J S, Henderson D, Howard R E, Hubbard W, Jackel L D. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1989, 1(4): 541–551
Cohen G, Afshar S, Tapson J, van Schaik A. EMNIST: extending MNIST to handwritten letters. In: Proceedings of 2017 International Joint Conference on Neural Networks. 2017, 2921–2926
Ji S, Pan S, Long G, Li X, Jiang J, Huang Z. Learning private neural language modeling with attentive aggregation. In: Proceedings of 2019 International Joint Conference on Neural Networks. 2019, 1–8
Liu Z, Chen Y, Yu H, Liu Y, Cui L. GTG-shapley: efficient and accurate participant contribution evaluation in federated learning. ACM Transactions on Intelligent Systems and Technology, 2022, 13(4): 60
Liu Z, Chen Y, Zhao Y, Yu H, Liu Y, Bao R, Jiang J, Nie Z, Xu Q, Yang Q. Contribution-aware federated learning for smart healthcare. In: Proceedings of the 36th AAAI Conference on Artificial Intelligence. 2022, 12396–12404
Acknowledgements
This work was supported by the National Natural Science Foundation of China (Grant Nos. 62132017, 61772456, 61761136020), the Zhejiang Provincial Natural Science Foundation of China (LD24F020011), and the “Pioneer and Leading Goose” R&D Program of Zhejiang (2024C01167).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests Wei CHEN is an Editorial Board member of the journal and a co-author of this article. To minimize bias, they were excluded from all editorial decision-making related to the acceptance of this article for publication. The remaining authors declare no conflict of interest.
Additional information
Tianye Zhang received her BS in mathematics from Zhejiang University, China in 2016. She is currently a PhD student in the College of Computer Science and Technology at the Zhejiang University, China. Her research interests include data mining and visual analytics.
Haozhe Feng received his BS in mathematics statistics from Zhejiang University, China in 2018. He is currently a PhD student in the College of Computer Science and Technology at the Zhejiang University, China. His research interests include representation learning and distributed machine learning.
Wenqi Huang is the leader of Artificial Intelligence and Intelligent Software Team in R&D Center, Digital Grid Research Institute, China Southern Power Grid. She holds BS (2010) and PhD (2015) degrees from the Department of Information Science and Electronic Engineering, Zhejiang University, China. Her research interests span artificial intelligence, data mining and blockchain application technology in the field of power industry.
Lingyu Liang is the senior technical expert of Artificial Intelligence and Intelligent Software Team in R&D Center, Digital Grid Research Institute, China Southern Power Grid. He holds BS (2010) and MS (2014) from School of Information Scicnce and Technology, Tsinghua University, China. His research interests span artificial intelligence, data mining, and intelligent decision-making in the field of power industry.
Huanming Zhang is the researcher of Artificial Intelligence and Intelligent Software Team in R&D Center, Digital Grid Research Institute, China Southern Power Grid. He holds BS (2019) from School of Electrical Engineering, Beijing Jiaotong University and MS (2022) from School of Electrical Engineering, South China University of Technology, China. His research interests span artificial intelligence, data mining, intelligent decision-making, and large language model in the field of power industry.
Zexian Chen received his BS in computer science and technology from the Zhejiang University, China in 2018. He is currently a master student in the College of Computer Science and Technology at the Zhejiang University, China. His research interests include information visualization and visual analytics.
Anthony K. H. Tung is a professor in the Department of Computer Science, National University of Singapore, Singapore(NUS) and a junior faculty member in the NUS Graduate School for Integrative Sciences and Engineering and a SINGA supervisor. His research interests include various aspects of databases and data mining (KDD) including buffer management, frequent pattern discovery, spatial clustering, outlier detection, and classification analysis.
Wei Chen is a professor at the State Key Lab of CAD & CG, Zhejiang University, China. His research interests are visualization and visual analysis, and has published more than 30 IEEE/ACM Transactions and IEEE VIS papers. He actively served as guest or associate editors of IEEE Transactions on Visualization and Computer Graphics, IEEE Transactions on Intelligent Transportation Systems, and Journal of Visualization.
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Zhang, T., Feng, H., Huang, W. et al. FedCare: towards interactive diagnosis of federated learning systems. Front. Comput. Sci. 19, 197335 (2025). https://doi.org/10.1007/s11704-024-3735-7
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11704-024-3735-7