research-article

FedCD: A Classifier Debiased Federated Learning Framework for Non-IID Data

Authors:

Tianlong Zhang,

Junping DuAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 8994 - 9002

https://doi.org/10.1145/3581783.3611966

Published: 27 October 2023 Publication History

Abstract

One big challenge to federated learning is the non-IID data distribution caused by imbalanced classes. Existing federated learning approaches tend to bias towards classes containing a larger number of samples during local updates, which causes unwanted drift in the local classifiers. To address this issue, we propose a classifier debiased federated learning framework named FedCD for non-IID data. We introduce a novel hierarchical prototype contrastive learning strategy to learn fine-grained prototypes for each class. The prototypes characterize the sample distribution within each class, which helps align the features learned in the representation layer of every client's local model. At the representation layer, we use fine-grained prototypes to rebalance the class distribution on each client and rectify the classification layer of each local model. To alleviate the bias of the classification layer of the local models, we incorporate a global information distillation method to enable the local classifier to learn decoupled global classification information. We also adaptively aggregate the class-level classifiers based on their quality to reduce the impact of unreliable classes in each aggregated classifier. This mitigates the impact of client-side classifier bias on the global classifier. Comprehensive experiments conducted on various datasets show that our method, FedCD, effectively corrects classifier bias and outperforms state-of-the-art federated learning methods.

References

[1]

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. 2021a. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263 (2021).

[2]

Durmus Alp Emre Acar, Yue Zhao, Ruizhao Zhu, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. 2021b. Debiasing model updates for improving personalized federated training. In International Conference on Machine Learning. PMLR, 21--31.

[3]

Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. 2021. Provably secure federated learning against malicious clients. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 6885--6893.

[4]

Hong-You Chen and Wei-Lun Chao. 2020. Fedbe: Making bayesian model ensemble applicable to federated learning. arXiv preprint arXiv:2009.01974 (2020).

[5]

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597--1607.

[6]

Xiaomeng Chen, Yingxia Shao, Zhe Xue, and Ziqiang Yu. 2021. Multi-modal COVID-19 discovery with collaborative federated learning. In 2021 IEEE 7th International Conference on Cloud Computing and Intelligent Systems (CCIS). IEEE, 52--56.

[7]

Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. 2017. EMNIST: Extending MNIST to handwritten letters. In 2017 international joint conference on neural networks (IJCNN). IEEE, 2921--2926.

[8]

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. 2021. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning. PMLR, 2089--2099.

[9]

Don Kurian Dennis, Tian Li, and Virginia Smith. 2021. Heterogeneity for the win: One-shot federated clustering. In ICML. PMLR, 2611--2620.

[10]

Kate Donahue and Jon Kleinberg. 2021. Model-sharing games: Analyzing federated learning under voluntary participation. In AAAI, Vol. 35. 5303--5311.

[11]

Haya Elayan, Moayad Aloqaily, and Mohsen Guizani. 2021a. Deep federated learning for IoT-based decentralized healthcare systems. In 2021 International Wireless Communications and Mobile Computing (IWCMC). IEEE, 105--109.

[12]

Haya Elayan, Moayad Aloqaily, and Mohsen Guizani. 2021b. Sustainability of healthcare data analysis IoT-based systems using deep federated learning. IEEE Internet of Things Journal, Vol. 9, 10 (2021), 7338--7346.

[13]

Angelo Feraudo, Poonam Yadav, Vadim Safronov, Diana Andreea Popescu, Richard Mortier, Shiqiang Wang, Paolo Bellavista, and Jon Crowcroft. 2020. CoLearn: Enabling federated learning in MUD-compliant IoT edge networks. In Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and Networking. 25--30.

Digital Library

[14]

Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. 2021. Clustered sampling: Low-variance and improved representativity for clients selection in federated learning. In ICML. PMLR, 3407--3416.

[15]

Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. 2022. FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10112--10121.

[16]

Zeli Guan, Yawen Li, Zhe Xue, Yuxin Liu, Hongrui Gao, and Yingxia Shao. 2021. Federated graph neural network for cross-graph node classification. In 2021 IEEE 7th International Conference on Cloud Computing and Intelligent Systems (CCIS). IEEE, 418--422.

[17]

Sungwon Han, Sungwon Park, Fangzhao Wu, Sundong Kim, Chuhan Wu, Xing Xie, and Meeyoung Cha. 2022. FedX: Unsupervised Federated Learning with Cross Knowledge Distillation. arXiv preprint arXiv:2207.09158 (2022).

[18]

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9729--9738.

[19]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770--778.

[20]

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. 2019. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 (2019).

[21]

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. 2020. Federated visual classification with real-world data distribution. In ECCV. Springer, 76--92.

[22]

Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. 2021. Personalized cross-silo federated learning on non-iid data. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 7865--7873.

[23]

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning. PMLR, 5132--5143.

[24]

Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).

[25]

Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. 2021c. Adaptive Prototype Learning and Allocation for Few-Shot Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8334--8343.

[26]

Qinbin Li, Bingsheng He, and Dawn Song. 2021a. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10713--10722.

[27]

Qiushi Li, Wenwu Zhu, Chao Wu, Xinglin Pan, Fan Yang, Yuezhi Zhou, and Yaoxue Zhang. 2020b. InvisibleFL: federated learning over non-informative intermediate updates against multimedia privacy leakages. In Proceedings of the 28th ACM International Conference on Multimedia. 753--762.

Digital Library

[28]

Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. 2021b. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning. PMLR, 6357--6368.

[29]

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020a. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, Vol. 2 (2020), 429--450.

[30]

Yawen Li, Wenling Li, and Zhe Xue. 2022. Federated learning with stochastic quantization. International Journal of Intelligent Systems, Vol. 37, 12 (2022), 11600--11621.

Digital Library

[31]

Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. 2021. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, Vol. 34 (2021), 5972--5984.

[32]

Yongqiang Ma, Yingxia Shao, Zhe Xue, and Ziqiang Yu. 2021. Urban Fatigue Driving Prediction With Federated Learning. In 2021 IEEE 7th International Conference on Cloud Computing and Intelligent Systems (CCIS). IEEE, 47--51.

[33]

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR, 1273--1282.

[34]

Umberto Michieli and Pietro Zanuttigh. 2021. Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations. In CVPR. 1114--1124.

[35]

Xutong Mu, Yulong Shen, Ke Cheng, Xueli Geng, Jiaxuan Fu, Tao Zhang, and Zhiwei Zhang. 2023. Fedproc: Prototypical contrastive federated learning on non-iid data. Future Generation Computer Systems, Vol. 143 (2023), 93--104.

Digital Library

[36]

Qiying Pan, Yifei Zhu, and Lingyang Chu. 2023. Lumos: Heterogeneity-aware Federated Graph Learning over Decentralized Devices. IEEE International Conference on Data Engineering (ICDE) (2023).

[37]

Yu Qiao, Md Munir, Apurba Adhikary, Huy Q Le, Avi Deb Raha, Chaoning Zhang, Choong Seon Hong, et al. 2023. MP-FedCL: Multi-Prototype Federated Contrastive Learning for Edge Intelligence. arXiv preprint arXiv:2304.01950 (2023).

[38]

Hao Sun, Yuan Jia, Hui Dong, Dibo Dong, and Jianping Zheng. 2020. Combining additive manufacturing with microfluidics: an emerging method for developing novel organs-on-chips. Current Opinion in Chemical Engineering, Vol. 28 (2020), 1--9.

[39]

Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. 2022. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 8432--8440.

[40]

Minxue Tang, Xuefei Ning, Yitu Wang, Jingwei Sun, Yu Wang, Hai Li, and Yiran Chen. 2022. FedCor: Correlation-Based Active Client Selection Strategy for Heterogeneous Federated Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10102--10111.

[41]

Paul Voigt and Axel Von dem Bussche. 2017. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, Vol. 10, 3152676 (2017), 10-5555.

[42]

Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. 2020b. Federated learning with matched averaging. arXiv preprint arXiv:2002.06440 (2020).

[43]

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. 2020a. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, Vol. 33 (2020), 7611--7623.

[44]

Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), Vol. 10, 2 (2019), 1--19.

Digital Library

[45]

Yu Zang, Zhe Xue, Shilong Ou, Yunfei Long, Hai Zhou, and Junping Du. 2023. FedPcf: An Integrated Federated Learning Framework with Multi-Level Prospective Correction Factor. In Proceedings of the 2023 ACM International Conference on Multimedia Retrieval. 490--498.

Digital Library

[46]

Lin Zhang, Li Shen, Liang Ding, Dacheng Tao, and Ling-Yu Duan. 2022. Fine-tuning global model via data-free knowledge distillation for non-iid federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10174--10183.

[47]

Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. 2022. Decoupled Knowledge Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11953--11962.

[48]

Jian Zhao, Jianshu Li, Xiaoguang Tu, Fang Zhao, Yuan Xin, Junliang Xing, Hengzhu Liu, Shuicheng Yan, and Jiashi Feng. 2019. Multi-prototype networks for unconstrained set-based face recognition. arXiv preprint arXiv:1902.04755 (2019).

Cited By

Ling TShi SWang HHu CWang DCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Federated Morozov Regularization for Shortcut Learning in Privacy Preserving Learning with Watermarked Image DataProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681480(4899-4908)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681480
Zhu GLiu XNiu JTang SWu XZhang JCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical RepresentationsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681260(11060-11069)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681260
Li ALi YXue ZGuan ZZhuang M(2024)Learning Hierarchy-Aware Federated Graph Embedding for Link Prediction2024 IEEE International Conference on Big Data and Smart Computing (BigComp)10.1109/BigComp60711.2024.00059(329-336)Online publication date: 18-Feb-2024
https://doi.org/10.1109/BigComp60711.2024.00059
Show More Cited By

Index Terms

FedCD: A Classifier Debiased Federated Learning Framework for Non-IID Data
1. Computing methodologies
  1. Machine learning
    1. Machine learning algorithms
2. Security and privacy
  1. Software and application security
    1. Domain-specific security and privacy architectures

Recommendations

Federated learning on non-IID data: A survey
Abstract
Federated learning is an emerging distributed machine learning framework for privacy preservation. However, models trained in federated learning usually have worse performance than those trained in the standard centralized learning ...
Exploring personalization via federated representation Learning on non-IID data
Abstract
Federated Learning (FL) can learn a global model across decentralized data over different clients. However, it is susceptible to statistical heterogeneity of client-specific data. Clients focus on optimizing for their individual target ...
Non-IID data and Continual Learning processes in Federated Learning: A long road ahead
Abstract
Federated Learning is a novel framework that allows multiple devices or institutions to train a machine learning model collaboratively while preserving their data private. This decentralized approach is prone to suffer the consequences of data ...
Highlights
- We review the advances in Federated Learning, particularly on heterogeneous data.
- We consider time-evolving situations in Federated Learning.
- We establish a classification of the literature according to their data heterogeneity.

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

CCF-Tencent Open Research Fund
National Natural Science Foundation of China

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
554
Total Downloads

Downloads (Last 12 months)287
Downloads (Last 6 weeks)15

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ling TShi SWang HHu CWang DCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Federated Morozov Regularization for Shortcut Learning in Privacy Preserving Learning with Watermarked Image DataProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681480(4899-4908)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681480
Zhu GLiu XNiu JTang SWu XZhang JCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical RepresentationsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681260(11060-11069)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681260
Li ALi YXue ZGuan ZZhuang M(2024)Learning Hierarchy-Aware Federated Graph Embedding for Link Prediction2024 IEEE International Conference on Big Data and Smart Computing (BigComp)10.1109/BigComp60711.2024.00059(329-336)Online publication date: 18-Feb-2024
https://doi.org/10.1109/BigComp60711.2024.00059
Li ALi YShao Y(2024)Federated learning for supervised cross-modal retrievalWorld Wide Web10.1007/s11280-024-01249-427:4Online publication date: 26-Jun-2024
https://dl.acm.org/doi/10.1007/s11280-024-01249-4

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten