skip to main content
10.1145/3580305.3599344acmconferencesArticle/Chapter ViewAbstractPublication PageskddConference Proceedingsconference-collections
research-article

FedAPEN: Personalized Cross-silo Federated Learning with Adaptability to Statistical Heterogeneity

Published: 04 August 2023 Publication History

Abstract

In cross-silo federated learning (FL), the data among clients are usually statistically heterogeneous (aka not independent and identically distributed, non-IID) due to diversified data sources, lowering the accuracy of FL. Although many personalized FL (PFL) approaches have been proposed to address this issue, they are only suitable for data with specific degrees of statistical heterogeneity. In the real world, the heterogeneity of data among clients is often immeasurable due to privacy concern, making the targeted selection of PFL approaches difficult. Besides, in cross-silo FL, clients are usually from different organizations, tending to hold architecturally different private models. In this work, we propose a novel FL framework, FedAPEN, which combines mutual learning and ensemble learning to take the advantages of private and shared global models while allowing heterogeneous models. Within FedAPEN, we propose two mechanisms to coordinate and promote model ensemble such that FedAPEN achieves excellent accuracy on various data distributions without prior knowledge of data heterogeneity, and thus, obtains the adaptability to data heterogeneity. We conduct extensive experiments on four real-world datasets, including: 1) Fashion MNIST, CIFAR-10, and CIFAR-100, each with ten different types and degrees of label distribution skew; and 2) eICU with feature distribution skew. The experiments demonstrate that FedAPEN almost obtains superior accuracy on data with varying types and degrees of heterogeneity compared with baselines.

Supplementary Material

MP4 File (rtfp0376-2min-promo.mp4)
Promotional Video

References

[1]
Divyansh Aggarwal, Jiayu Zhou, and Anil K Jain. 2021. Fedface: Collaborative learning of face recognition model. In 2021 IEEE International Joint Conference on Biometrics (IJCB). IEEE, 1--8.
[2]
Alexander Brecko, Erik Kajati, Jiri Koziorek, and Iveta Zolotova. 2022. Federated Learning for Edge Computing: A Survey. Applied Sciences, Vol. 12, 18 (2022), 9124.
[3]
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. 2021. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning. PMLR, 2089--2099.
[4]
Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. 2020. Adaptive personalized federated learning. arXiv preprint arXiv:2003.13461 (2020).
[5]
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. 2020. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, Vol. 33 (2020), 3557--3568.
[6]
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. 2020. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, Vol. 33 (2020), 19586--19597.
[7]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Commun. ACM, Vol. 63, 11 (2020), 139--144.
[8]
Filip Hanzely and Peter Richtárik. 2020. Federated learning of a mixture of global and local models. arXiv preprint arXiv:2002.05516 (2020).
[9]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision. 1026--1034.
[10]
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, Vol. 2, 7 (2015).
[11]
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. 2021. Personalized Cross-Silo Federated Learning on Non-IID Data. In AAAI Conference on Artificial Intelligence. 7865--7873.
[12]
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. 2021. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, Vol. 14, 1-2 (2021), 1--210.
[13]
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning. PMLR, 5132--5143.
[14]
Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).
[15]
Daliang Li and Junpu Wang. 2019. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581 (2019).
[16]
Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. 2021a. Federated learning on non-iid data silos: An experimental study. arXiv preprint arXiv:2102.02079 (2021).
[17]
Qinbin Li, Bingsheng He, and Dawn Song. 2021b. Practical One-Shot Federated Learning for Cross-Silo Setting. In the International Joint Conference on Artificial Intelligence. 1484--1490.
[18]
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, Vol. 2 (2020), 429--450.
[19]
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. 2020. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, Vol. 33 (2020), 2351--2363.
[20]
Bingyan Liu, Yifeng Cai, Ziqi Zhang, Yuanchun Li, Leye Wang, Ding Li, Yao Guo, and Xiangqun Chen. 2021. DistFL: Distribution-aware Federated Learning for Mobile Scenarios. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 5, 4 (2021), 1--26.
[21]
Jun Luo and Shandong Wu. 2022. Adapt to Adaptation: Learning Personalization for Cross-Silo Federated Learning. In International Joint Conference on Artificial Intelligence. 2166--2173.
[22]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agü era y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In International Conference on Artificial Intelligence and Statistics, Vol. 54. 1273--1282.
[23]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, Vol. 32 (2019).
[24]
Tom J Pollard, Alistair EW Johnson, Jesse D Raffa, Leo A Celi, Roger G Mark, and Omar Badawi. 2018. The eICU Collaborative Research Database, a freely available multi-center database for critical care research. Scientific data, Vol. 5, 1 (2018), 1--13.
[25]
Hyowoon Seo, Jihong Park, Seungeun Oh, Mehdi Bennis, and Seong-Lyun Kim. 2020. Federated knowledge distillation. arXiv preprint arXiv:2011.02367 (2020).
[26]
Tao Shen, Jie Zhang, Xinkang Jia, Fengda Zhang, Gang Huang, Pan Zhou, Fei Wu, and Chao Wu. 2020. Federated Mutual Learning. CoRR, Vol. abs/2006.16765 (2020). showeprint[arXiv]2006.16765 https://arxiv.org/abs/2006.16765
[27]
Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. 2022b. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems (2022).
[28]
Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. 2022a. FedProto: Federated Prototype Learning across Heterogeneous Clients. In AAAI Conference on Artificial Intelligence. 8432--8440.
[29]
Haoyu Wang, Handong Zhao, Yaqing Wang, Tong Yu, Jiuxiang Gu, and Jing Gao. 2022. FedKC: Federated Knowledge Composition for Multilingual Natural Language Understanding. In Proceedings of the ACM Web Conference 2022. 1839--1850.
[30]
Jinze Wu, Qi Liu, Zhenya Huang, Yuting Ning, Hao Wang, Enhong Chen, Jinfeng Yi, and Bowen Zhou. 2021. Hierarchical personalized federated learning for user modeling. In Proceedings of the Web Conference 2021. 957--968.
[31]
Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017).
[32]
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology, Vol. 10, 2 (2019), 1--19.
[33]
Tao Yu, Eugene Bagdasaryan, and Vitaly Shmatikov. 2020. Salvaging federated learning by local adaptation. arXiv preprint arXiv:2002.04758 (2020).
[34]
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. 2013. Domain adaptation under target and conditional shift. In International conference on machine learning. PMLR, 819--827.
[35]
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. 2018. Deep Mutual Learning. In Conference on Computer Vision and Pattern Recognition. 4320--4328.
[36]
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. 2018. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 (2018).
[37]
Zhi Zhou, Xu Chen, En Li, Liekang Zeng, Ke Luo, and Junshan Zhang. 2019. Edge intelligence: Paving the last mile of artificial intelligence with edge computing. Proc. IEEE, Vol. 107, 8 (2019), 1738--1762.
[38]
Zhi-Hua Zhou. 2021. Ensemble learning. Springer.
[39]
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. 2021. Data-free knowledge distillation for heterogeneous federated learning. In International Conference on Machine Learning. 12878--12889.

Cited By

View all
  • (2025)SFML: A personalized, efficient, and privacy-preserving collaborative traffic classification architecture based on split learning and mutual learningFuture Generation Computer Systems10.1016/j.future.2024.107487162(107487)Online publication date: Jan-2025
  • (2024)COALAProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694666(62723-62742)Online publication date: 21-Jul-2024
  • (2024)Horizontal Federated Recommender System: A SurveyACM Computing Surveys10.1145/365616556:9(1-42)Online publication date: 8-May-2024
  • Show More Cited By

Index Terms

  1. FedAPEN: Personalized Cross-silo Federated Learning with Adaptability to Statistical Heterogeneity

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
      August 2023
      5996 pages
      ISBN:9798400701030
      DOI:10.1145/3580305
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 04 August 2023

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. adaptability to heterogeneity
      2. federated learning
      3. model heterogeneity
      4. statistical heterogeneity

      Qualifiers

      • Research-article

      Funding Sources

      • National Science Foundation of China
      • Key Research Project of Zhejiang Province

      Conference

      KDD '23
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

      Upcoming Conference

      KDD '25

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)711
      • Downloads (Last 6 weeks)76
      Reflects downloads up to 17 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)SFML: A personalized, efficient, and privacy-preserving collaborative traffic classification architecture based on split learning and mutual learningFuture Generation Computer Systems10.1016/j.future.2024.107487162(107487)Online publication date: Jan-2025
      • (2024)COALAProceedings of the 41st International Conference on Machine Learning10.5555/3692070.3694666(62723-62742)Online publication date: 21-Jul-2024
      • (2024)Horizontal Federated Recommender System: A SurveyACM Computing Surveys10.1145/365616556:9(1-42)Online publication date: 8-May-2024
      • (2024)BlockDFL: A Blockchain-based Fully Decentralized Peer-to-Peer Federated Learning FrameworkProceedings of the ACM Web Conference 202410.1145/3589334.3645425(2914-2925)Online publication date: 13-May-2024
      • (2024)CQUPT-HDS: Health Big Data Security Monitoring Scheme for the Whole Life Cycle2024 IEEE International Conference on Medical Artificial Intelligence (MedAI)10.1109/MedAI62885.2024.00069(476-481)Online publication date: 15-Nov-2024
      • (2024)FlocOff: Data Heterogeneity Resilient Federated Learning With Communication-Efficient Edge OffloadingIEEE Journal on Selected Areas in Communications10.1109/JSAC.2024.343152642:11(3262-3277)Online publication date: Nov-2024
      • (2024)Adaptive Knowledge Recomposition for Personalized Federated Learning via Discrete Wavelet TransformIEEE Internet of Things Journal10.1109/JIOT.2024.343599411:21(35610-35624)Online publication date: 1-Nov-2024
      • (2024)IOFL: Intelligent-Optimization-Based Federated Learning for Non-IID DataIEEE Internet of Things Journal10.1109/JIOT.2024.335494211:9(16693-16699)Online publication date: 1-May-2024
      • (2024)Towards consensual representation: Model-agnostic knowledge extraction for dual heterogeneous federated fault diagnosisNeural Networks10.1016/j.neunet.2024.106618179(106618)Online publication date: Nov-2024
      • (2024)Mitigating data imbalance and generating better prototypes in heterogeneous Federated Graph LearningKnowledge-Based Systems10.1016/j.knosys.2024.111876296(111876)Online publication date: Jul-2024
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media