Federating from History in Streaming Federated Learning
Authors: 
Ruirui Zhang, Yifei Zou, Zhenzhen Xie, Xiao Zhang, Peng Li, Zhipeng Cai, Xiuzhen Cheng, Dongxiao YuAuthors Info & Claims
Ruirui Zhang, Yifei Zou, Zhenzhen Xie, Xiao Zhang, Peng Li, Zhipeng Cai, Xiuzhen Cheng, Dongxiao YuAuthors Info & ClaimsPages 151 - 160
Abstract
To address the online learning problem in distributed systems, Streaming Federated learning (SFL) enables immediate model training by clients upon collecting new data, finding wide applications in AI-enabled Internet-of-Things and sensor networks. Given the variability in data distribution across different historical periods, the ability to recall and rapidly apply previously encountered data distributions significantly enhances the efficiency and accuracy of model training. In this paper, a demo based on the real-world temperature datasets is presented to demonstrate the importance of history knowledge in local training and the federating process of SFL, which also shows that vanilla federated learning without considering the history knowledge may even be harmful to model training. Observing this, we propose Fed-HIST, a Federated learning framework that enables the clients to learn from the HISTory knowledge of the whole distributed learning system. Unlike direct raw data storage, Fed-HIST employs model architectures to capture the data distributions, offering a more space-efficient and privacy-preserving method of knowledge storage on a server pool. Additionally, a model similarity comparison scheme is designed to retrieve beneficial knowledge from the pool uploaded by the clients in the past. Such a history-aware federation can enhance the efficiency of training each client, only requiring the recurrence of similar data distributions among SFL participants. We validate our framework through extensive simulations on MNIST, Fashion-MINST, CIFAR10, and CIFAR100 datasets, benchmarking against 9 baselines and highlighting the importance of federating from history in SFL problem through necessary ablation studies.
References
[1]
G. Canonaco, A. Bergamasco, A. Mongelluzzo, and M. Roveri. 2021. Adaptive Federated Learning in Presence of Concept Drift. In IJCNN 2021. IEEE, 1--7.
[2]
F. E. Casado, D. Lema, M. F. Criado, R. Iglesias, C. Vázquez Regueiro, and S. Barro. 2022. Concept drift detection and adaptation for federated and continual learning. Multim. Tools Appl. 81, 3 (2022), 3397--3419.
[3]
Y. Chen, Z. Chai, Y. Cheng, and H. Rangwala. 2021. Asynchronous Federated Learning for Sensor Data with Concept Drift. In IEEE BigData 2021. IEEE, 4822--4831.
[4]
Li Deng. 2012. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine 29, 6 (2012), 141--142.
[5]
C. T. Dinh, N. H. Tran, and T. D. Nguyen. 2020. Personalized Federated Learning with Moreau Envelopes. CoRR abs/2006.08848 (2020).
[6]
C. Gong, Z. Zheng, F. Wu, Y. Shao, B. Li, and G. Chen. 2023. To Store or Not? Online Data Selection for Federated Learning with Limited Storage. In WWW 2023. ACM, 3044--3055.
[7]
Y. Gu and Y. Bai. 2023. LDIA: Label distribution inference attack against federated learning in edge computing. Journal of Information Security and Applications 74 (2023), 103475.
[8]
S. Hassan, A. Brennan, G. Muntean, and J. McManis. 2023. User Profile-Based Viewport Prediction Using Federated Learning in Real-Time 360-Degree Video Streaming. In BMSB 2023. 1--7.
[9]
T. Hsu, H. Qi, and M. Brown. 2019. Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification. CoRR abs/1909.06335 (2019).
[10]
C. Hu, Z. Chen, and E. Larsson. 2023. Dynamic Scheduling For Federated Edge Learning With Streaming Data. In ICASSPW 2023. 1--5.
[11]
Y. Jin, L. Jiao, Z. Qian, S. Zhang, and S. Lu. 2021. Budget-Aware Online Control of Edge Federated Learning on Streaming Data With Stochastic Inputs. IEEE J. Sel. Areas Commun. 39, 12 (2021), 3704--3722.
[12]
E. Jothimurugesan, K. Hsieh, J. Wang, G. Joshi, and P. B. Gibbons. 2023. Federated Learning under Distributed Concept Drift. In AISTATS 2023 (Proceedings of Machine Learning Research, Vol. 206). PMLR, 5834--5853.
[13]
A. Krizhevsky, G. Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).
[14]
T. Li, A. Kumar Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. 2020. Federated Optimization in Heterogeneous Networks. In MLSys 2020.
[15]
X. Li, M.Jiang, X. Zhang, M. Kamp, and Q. Dou. 2021. FedBN: Federated Learning on Non-IID Features via Local Batch Normalization. In ICLR 2021.
[16]
O. Marfoq, G. Neglia, L. Kameni, and R. Vidal. 2023. Federated Learning for Data Streams. In AISTATS 2023 (Proceedings of Machine Learning Research, Vol. 206). PMLR, 8889--8924.
[17]
C. Bernard Mawuli, L. Che, J. Kumar, S. Ud Din, Z. Qin, Q. Yang, and J. Shao. 2023. FedStream: Prototype-Based Federated Learning on Distributed Concept-Drifting Data Streams. IEEE Trans. Syst. Man Cybern. Syst. 53, 11 (2023), 7112--7124.
[18]
C. B. Mawuli, J. Kumar, E. Nanor, S. Fu, L. Pan, Q. Yang, W. Zhang, and J. Shao. 2023. Semi-supervised federated learning on evolving data streams. Information Sciences 643 (2023), 119235.
[19]
B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Agüera y Arcas. 2017. Communication-Efficient Learning of Deep Networks from Decentralized Data. In AISTATS 2017 (Proceedings of Machine Learning Research, Vol. 54). PMLR, 1273--1282.
[20]
NIWA. 2024. New Zealand's National Climate Database. https://cliflo.niwa.co.nz/ Dataset.
[21]
A. Robins. 1995. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science 7, 2 (1995), 123--146.
[22]
Y. Tao and M. T. Ozsu. 2009. Mining data streams with periodically changing distributions. In CIKM 2009. 887--896.
[23]
A. Tsymbal. 2004. The problem of concept drift: definitions and related work. Computer Science Department, Trinity College Dublin 106, 2 (2004), 58.
[24]
H. Wang, J. Bian, and J. Xu. 2024. On the Local Cache Update Rules in Streaming Federated Learning. IEEE Internet Things J. 11, 6 (2024), 10808--10816.
[25]
A. Wuerkaixi, S. Cui, J. Zhang, K. Yan, B. Han, G. Niu, L. Fang, C. Zhang, and M. Sugiyama. 2024. Accurate Forgetting for Heterogeneous Federated Continual Learning. In ICLR 2024.
[26]
H. Xiao, K. Rasul, and R. Vollgraf. 2017. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms.
[27]
M Yogitha and KS Srinivas. 2023. Using Federated Learning in Anomaly Detection and Analytics on Real-time Streaming Data of Healthcare. In ICGSP 2023 (ICGSP '23). Association for Computing Machinery, 29--34.
[28]
J. Yoon, W. Jeong, G. Lee, E. Yang, and S. Ju Hwang. 2020. Federated Continual Learning with Adaptive Parameter Communication. CoRR abs/2003.03196 (2020).
[29]
R.R. Zhang et al. 2024. Technical Report of Federating from History in Streaming Federated Learning. https://github.com/SheryMo/Technical-Report-of-FedHIST
[30]
Met Éireann. 2024. Dublin Climate Database. https://www.met.ie/climate/available-data/historical-data Dataset.
Recommendations
Heterogeneous Federated Learning: State-of-the-art and Research Challenges
Federated learning (FL) has drawn increasing attention owing to its potential use in large-scale industrial applications. Existing FL works mainly focus on model homogeneous settings. However, practical FL typically faces the heterogeneity of data ...
Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?
MobiHoc '24: Proceedings of the Twenty-fifth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile ComputingFederated Learning (FL) has gained significant popularity due to its effectiveness in training machine learning models across diverse sites without requiring direct data sharing. While various algorithms along with their optimization analyses have shown ...
Social Welfare Maximization for Federated Learning with Network Effects
MobiHoc '24: Proceedings of the Twenty-fifth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile ComputingA proper mechanism design can help federated learning (FL) to achieve good social welfare by coordinating self-interested clients through the learning process. However, existing mechanisms neglect the network effects of client participation, leading to ...
Comments
Information & Contributors
Information
Published In
October 2024
511 pages
ISBN:9798400705212
DOI:10.1145/3641512
- General Chair:
- Symeon Papavassiliou,
- Program Chair:
- Stefan Schmid
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Key R&D Program of Shandong Province
- the National Natural Science Foundation of China (NSFC)
- Shandong Science Fund for Excellent Young Scholars
Conference
MobiHoc '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 296 of 1,843 submissions, 16%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 171Total Downloads
- Downloads (Last 12 months)171
- Downloads (Last 6 weeks)30
Reflects downloads up to 03 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in