Abstract
Federated learning (FL) is a promising decentralized machine learning approach that enables multiple distributed clients to train a model jointly while keeping their data private. However, in real-world scenarios, the supervised training data stored in local clients inevitably suffer from imperfect annotations, resulting in subjective, inconsistent and biased labels. These noisy labels can harm the collaborative aggregation process of FL by inducing inconsistent decision boundaries. Unfortunately, few attempts have been made towards noise-tolerant federated learning, with most of them relying on the strategy of transmitting overhead messages to assist noisy labels detection and correction, which increases the communication burden as well as privacy risks. In this paper, we propose a simple yet effective method for noise-tolerant FL based on the well-established co-training framework. Our method leverages the inherent discrepancy in the learning ability of the local and global models in FL, which can be regarded as two complementary views. By iteratively exchanging samples with their high confident predictions, the two models “teach each other” to suppress the influence of noisy labels. The proposed scheme enjoys the benefit of overhead cost-free and can serve as a robust and efficient baseline for noise-tolerant federated learning. Experimental results demonstrate that our method outperforms existing approaches, highlighting the superiority of our method.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
B. McMahan, E. Moore, D. Ramage, S. Hampson, B. A. Y. Areas. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Fort Lauderdale, USA, vol. 54, pp. 1273–1282, 2017.
T. Lin, L. J. Kong, S. U. Stich, M. Jaggi. Ensemble distillation for robust model fusion in eederated learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, Article number 198, 2020.
R. Shokri, V. Shmatikov. Privacy-preserving deep learning. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, USA, pp.1310-1321, 2015. DOI: https://doi.org/10.1145/2810103.2813687.
C. Y. Zhang, S. Bengio, M. Hardt, B. Recht, O. Vinyals. Understanding depp learning (still) requires rethinking generalization. Communications of the ACM, vol. 64, no. 3, pp. 107–115, 2021. DOI: https://doi.org/10.1145/3446776.
B. Han, Q. M. Yao, T. L. Liu, G. Niu, I. W. Tsang, J. T. Kwok, M. Sugiyama. A survey of label-noise representation learning: Past, present and future, [Online], Available: https://arxiv.org/abs/2011.04406, 2020.
H. Song, M Kim, D. Park, Y. Shin, J. G Lee. Learning from nossy hibete with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, to be pubühhed. DOI: https://doi.org/10.1109/TNNLS.2022.3152527.
D. Karimi, H. R. Dou, S. K. Warfield, A. Gholipour. Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis. Medical Image Analysis, vol. 65, Article number 101759, 2020. DOI: https://doi.org/10.1016/j.media.2020.101719.
X. Zhou, X. M. Liu, J. J. Jiang, X. Gao, X. Y. Ji. Asymmetric loss functions for learning with noisy labels. In Proceedings of the 38th International Conference on Machine Learning, vol. 139, pp. 12846–12856, 2021.
Y. Q. Chen, X. D. Yang, X. Qin, H. Yu, B. Chen, Z. Q. Shen. FOCUS: Dealing with label quality disparity in federated learning, [Online], Available: https://arxiv.org//abs/2001.11359, 2020.
S. Yang, H. Park, J. Byun, C. Kim. Robust federated learning with noisy labels. IEEE Intelligent Systems, vol. 37, no. 2, pp. 35–43, 2022. DOI: https://doi.org/10.1109/MIS.2022.3151466.
K. Tam, L. Li, B. Han, C. Z. Xu, H. Z. Fu. Federated noisy client learning, [Online], Available: https://arxiv.org/abs/2106.13239, 2021.
P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, R. G. L. D’Oliveira, H. Eichner, S. E. Rouayheb, D. Evans, J. Gardner, Z. Garrett, A. Gascón, B. Ghazi, P. B. Gibbons, M. Gruteser, Z. Harchaoui, C. Y. He, L. He, Z. Y. Huo, B. Hutchinson, J. Hsu, M. Jaggi, T. Javidi, G. Joshi, M. Khodak, J. Konecny, A. Korolova, F. Koushan ar, S. Koyejo, T. Lepoint, Y. Liu, P. Mittal, M. Mohri, R. Nock, A. Özgür, R. Pagh, H. Qi, D. Ramage, R. Raskar, M. Raykova, D. Song, W. K. Song, S. U. Stich, Z. T. Sun, A. T. Suresh, F. Tramer, P. Vepakomma, J. Y. Wang, L. Xiong, Z. Xu, Q. Yang, F. X. Yu, H. Yu, S. Zhao Advances and open problems in federated learning Foundations and Trends.® in Machine Learning, vol. 14, pp. 1–210, 2021. DOI: https://doi.org/10.1561/2200000083.
A. Blum, T. Mitchell Combining abeted and unlebeled data with co-training. In Proceedings of the 11th Annual Conference on Computational Learning Theory, ACM, Madison, USA, pp. 92–100, 1998. DOI: https://doi.org/10.1145/279943.279962.
B. Han, Q. M. Yao, X. R. Yu, G. Niu, M. Xu, W. H. Hu, I. W. Tang, M. Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 8536–8546, 2018.
W. Wang, Z. H. Zhou. A new analysis of co-training. In Proceedings of the 27th International Conference on Machine Learning, Omnipress, Israel, pp. 1135–1142, 2010.
S. U. Stich. Local SGD converges fast and communicates little. In Proceedings of the 7th International Conference on Learning Representations, New Orleans, USA, pp. 1–19, 2019.
C. L. Zhang, S. Y. Li, J. Z. Xia, W. Wang, F. Yan, Y. Liu. BatchCrypt: Efficient homomorphic encryption for crosssilo federated learning. In Proceedings of USENIX Annual Technical Conference, Article number 33, 2020.
S. Hardy, W. Henecka, H. Ivey-Law, R. Nock, G. Patrini, G. Smith, B. Thorne. Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption, [Online], Available: https://arxiv.org/abs/1711.10677, 2017.
K. Wei, J. Li, M. Ding, C. Ma, H. H. Yang, F. Farokhi, S. Jin, T. Q. S. Quek, H. V. Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, vol. 15, pp. 3454–3469, 2220. DOI: https://doi.org/10.1109/tifs.2.2020.2988575.
R. C. Geyer, T. Klein, M. Nabi. Differentially private federated learning: A client level perspective, [Online], Available: https://arxiv.org/abs/1712.07557, 2017.
H. B. McMahan, D. Ramage, K. Talwar, L. Zhang. Learning differentially private recurrent language models. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada, 2018.
F. L. Zhang, Y. C. Li, S. Y. Lin, Y. F. Shao, J. J. Jiang, X. M. Liu. Large sparse kernels for federated learning, [Online], Available: https://openreview.net/forum?id=ZCv4E1unfJP, 2023.
Q. Yang, Y. Liu, T. J. Chen, Y. X. Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology, vol. 10, no. 2, Article number 12, 2019. DOI: https://doi.org/10.1145/3298981.
W. Y. B. Lim, N. C. Luong, D. T. Hoang, Y. T. Jiao, Y. C. Liang, Q. Yang, D. Niyato, C. Y. Miao. Federated learning in mobile edge networks: A comprehensive survey. IEEE Communications Surveys & Tutorials, vol. 22, no. 3, pp. 2031–2063, 2020. DOI: https://doi.org/10.1109/COMST.2020.2986024.
T. Li, A. K. Sahu, A. Talwalkar, V. Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 50–60, 2020. DOI: https://doi.org/10.1109/MSP.2020.2975749.
V. Smith, C. K. Chiang, M. Sanjabi, A. S. Talwalkar. Federated multi-task learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, vol. 30, pp. 4427–4437, 2017
T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, V. Smith. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems, Austin, USA, pp. 429–450, 2020.
S. P. Karimireddy, S. Kale, M. Mohri, S. J. Reddi, S. U. Stich, A. T. Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In Proceedings of the 37th International Conference on Machine Learning, vol. 119, pp. 5132–5143, 2020.
Q. B. Li, B. S. He, D. Song. Model-contrastive federated learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 10708–10717, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.01057.
T. Shen, J. Zhang, X. K. Jia, F. D. Zhang, G. Huang, P. Zhou, K. Kuang, F. Wu, C. Wu. Federated mutual learning, [Online], Available: https://arxiv.org/abs/2006.16765, 2020.
B. Zhao, P. Sun, T. Wang, K. Y. Jiang. FedInv: Byzantine-robust federated learning by inversing local model updates. In Proceedings of the 36th AAAI conference on Artificial Intelligence, Calomnia, USA, pp.9171-9179, 2022. DOI: https://doi.org/10.1609/aaai.v36i8.20903.
T. Tuor, S. Q. Wang, B. J. Ko, C. C. Liu, K. K. Leung. Overcoming noisy and irrelevant data in federated learning. In Proceedings of the 25th International Conference on Pattern Recognition, IEEE, Milan, Italy, pp. 5020–5027, 2021. DOI: https://doi.org/10.1109/ICPR48806.2021.9412599.
S. M Dauern, C. Y. Liu, Z. S. Cao, X. P. Jin, P. Y. Han. Fed-DR-Filter: Using global data representation to reduce the impact of noisy labels on the performance of federated learning Future Generation Computer Systems, vol 137, pp. 336–348, 2022. DOI: https://doi.org/10.1016/j.future.2022.07.013.
X. W. Fang, M. Ye. Robust federated learning with noisy and heterogeneous clients. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, New Orleans, USA, pp. 10062–10071, 2022. DOI: https://doi.org/10.1109/CVPR52688.2022.00983.
X. Zhou, X. M. Liu, C. Y. Wang, D. M. Zhai, J. J. Jiang, X. Y. Ji. Learning with noisy labels via sparse regularization. In Proceedings of IEEEfCVF International Conference on Computer Vision, IEEE, Montreal, Canada, pp. 72–81, 2021. DOI: https://doi.org/10.1109/ICCV48922.2021.00014.
X. Zhou, X. M. Liu, D. M. Zhai, J. J. Jiang, X. Y. Ji. Asymmetric loss functions fbr noise-tolerent learning: Theory and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 7, pp. 8094–8109, 2023. DOI: https://doi.org/10.1109/TPAMI.2023.3236459.
A. K. Menon, B. van Rooyen, N. Natarajan. Learning from binary labels with instance-dependent corruption, [Online], Available: https://arxiv.org/abs/1605.00751, 2016.
X. B. Xia, T. L. Liu, N. N. Wang, B. Han, C. Gong, G. Niu, M. Sugiyama. Are anchor points really indispensable in label-noise learning?. In Proceedings of the 33rd International Conference o Neural Information Processing Systems, Vancouver, Canada, Article number 614, 2019.
J. C. Cheng, T. L. Liu, K. Ramamohanarao, D. C. Tao. Learning with bounded instance and label-dependent label noise. In Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria, vol. 119, pp. 1789–1799, 2020.
S. Sukhbaatar, J. Bruna, M. Paluri, L. Bourdev, R Fergus. Training convolutional networks with noisy labels, [Online], Available: https://arxiv.org/abs/1406.2080, 2014.
J. Goldberger, E. Ben-Reuven. Training deep neural-networks using a noise adaptation layer. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France, 2017.
G. Patrini, A. Rozza, A. K. Menon, R. Nock, L. Z. Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 2233–2241, 2017. DOI: https://doi.org/10.1109/CVPR.2017.240.
D. Hendrycks, M. Mazeika, D. Wilson, K. Gimpel. Using trusted data to train deep networks on labels corrupted by severe noise. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 10477–10486, 2018.
Z. L. Zhang, M. R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, Monreaak Canada, pp. 8792–8802, 2018.
A. K. Menon, A. S. Rawat, S. J. Reddi, S. Kumar. Can gradient ciippmg mitigate tabel noise? In Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
Y. S. Wang, X. J. Ma, Z. Y. Chen, Y. Luo, J. F. Yi, J. Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Republic of Korea, pp. 322–330, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00041.
D. Arpit, S. Jastrzębski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, S. Lacoste-Julien. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, vol. 70, pp. 233–242, 2017.
L. Jiang, Z. Y. Zhou, T. Leung, L. J. Li, L. Fei-Fei. MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, vol. 80, pp. 2309–2318, 2018.
X. R. Yu, B. Han, J. C. Yao, G. Niu, I. W. Tsang, M. Sugiyama. How does disagreement help generalization against label corruption?. In Proceedings of the 36th International Conference on Machine Learning, Long Beach, USA, vol. 97, pp. 7164–7173, 2019.
E. Malach, S. Shaew-Shwartz. Decoupling “when to update” from “how to update”. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 961–971, 2017.
F. Mo, A. S. Shamsabadi, K. Katevas, S. Demetriou, I. Leontiadis, A. Cavallaro, H. Haddadi. DarkneTZ: Towards model privacy at the edge using trusted execution environments. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services, ACM, Toronto, Cnadaa, pp.6000744, 2020. DOI: https://doi.org/10.1155/3386901.3388946.
F. Mo, H. Haddadi, K. Katevas, E. Marin, D. Perino, N. Kourtellis. PPFL: Privacy-preserving federated learning with trusted execution environments. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services, ACM, pp. 94–108, 2021. DOI: https://doi.org/10.1145/3458864.3466628.
A. Ghosh, H. Kumar, P. S. Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the 31st AAAI conference on Artificial Intelligence, San Francisco, USA, pp. 1919–1925, 2017. DOI: https://doi.org/10.1609/aaai.v31i1.10894.
S. Y. Qaao, W. Shen, Z. S. Zhang, B. Wang, A. Yuille. Deep co-training for semi-supervised image recognition. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 142–159, 2018. DOI: https://doi.org/10.1007/978-3-030-01267-09.
T. Xiao, T. Xia, Y. Yang, C. Huang, X. G. Wang. Learning from massive noisy labeled data for image classification. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, pp. 2691–2699, 2015. DOI: https://doi.org/10.1109/CVPR.2015.7298885.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. M. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. J. Bai, S. Chintala. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, Article number 721, 2019.
Y. Kim, J. Yim, J. Yun, J. Kim. NLNL: Negative learning for noisy labels. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Republic of Korea, pp. 101–110, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00019.
X. J. Ma, H. X. Huang, Y. S. Wang, S. Romano, S. M. Erfani, J. Bailey. Normalized loss functions for deep learning with noisy labels. In Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria, vol. 119, pp. 6543–6553, 2020.
Acknowledgements
This work was supported by National Natural Science Foundation of China (Nos. 92270116 and 62071155).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
The authors declared that they have no conflicts of interest to this work.
Additional information
Colored figures are available in the online version at https://link.springer.com/journal/11633
Shiyi Lin received the B. Eng. and M. Eng. degrees in instrument science and technology from Harbin Institute of Technology, China in 2018 and 2020, respectively. She is currently a Ph. D. degree candidate in computer science at Harbin Institute of Technology (HIT), China.
Her research interests include unsupervised learning and federated learning.
Deming Zhai received the B. Sc., M. Sc. and Ph. D. (Hons.) degrees in computer science from Harbin Institute of Technology (HIT), China in 2007, 2009, and 2014, respectively. She is currently an associate professor with Department of Computer Science, HIT, China. In 2011, she was with Hong Kong University of Science and Technology, China, as a visiting student. In 2012, she was with the GRASP Laboratory, University of Pennsylvania, USA, as a visiting scholar. From August 2014 to April 2016, she worked as a project researcher at National Institute of Informatics (NII), Japan.
Her research interests include machine learning and its application in computer version.
Feilong Zhang received the B. Eng. and M. Eng. degrees in instrument science and technology from Harbin Institute of Technology, China in 2018 and 2020, respectively. He is currently a Ph. D. degree candidate in computer science from Harbin Institute of Technology (HIT), China.
His research interests include computer vision, machine learning and federated learning.
Junjun Jiang received the B. Sc. degree in mathematics from Department of Mathematics, Huaqiao University, China in 2009, and the Ph. D. degree in computer science from School of Computer, Wuhan University, China in 2014. From 2015 to 2018, he was an associate professor at China University of Geosciences, China. Since 2016, he has been a project researcher with National Institute of Informatics, Japan. He is currently a professor with School of Computer Science and Technology, Harbin Institute of Technology, China. He won the Finalist of the World’s FIRST 10K Best Paper Award at ICME 2017, and the Best Student Paper Runner-up Award at MMM 2015. He received the 2016 China Computer Federation (CCF) Outstanding Doctoral Dissertation Award and 2015 ACM Wuhan Doctoral Dissertation Award, China.
His research interests include image processing and computer vision.
Xianming Liu received the B. Sc., M. Sc. and Ph. D. degrees in computer science from Harbin Institute of Technology (HIT), China in 2006, 2008 and 2012, respectively. In 2011, he spent half a year at Department of Electrical and Computer Engineering, McMaster University, Canada, as a visiting student, where he was a post-doctoral fellow from 2012 to 2013. He was a project researcher with National Institute of Informatics (NII), Japan from 2014 to 2017. He is currently a professor with School of Computer Science and Technology, HIT, China. He has published over 50 international conference and journal publications, including top IEEE journals, such as T-IP, T-CSVT, T-IFS, and T-MM, and top conferences, such as ICML, ICLR, CVPR, ICCV, etc. He was a recipient of the IEEE ICME 2016 Best Student Paper Award.
His research interests include trustworthy AI, computational imaging, biomedical signal compression and 3D signal processing and analysis.
Xiangyang Ji received the B. Sc. degree in materials science and the M. Sc. degree in computer science from Harbin Institute of Technology, China in 1999 and 2001, respectively, and the Ph. D. degree in computer science from Institute of Computing Technology, Chinese Academy of Sciences, China in 2008. He joined Tsinghua University, China in 2008, where he is currently a professor with Department of Automation, School of Information Science and Technology. He has authored over 100 referred conference and journal papers.
His research interests include signal processing, image/video compressing, and intelligent imaging.
Rights and permissions
About this article
Cite this article
Lin, S., Zhai, D., Zhang, F. et al. Overhead-free Noise-tolerant Federated Learning: A New Baseline. Mach. Intell. Res. 21, 526–537 (2024). https://doi.org/10.1007/s11633-023-1449-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11633-023-1449-1