Fuxun Yu
ABSTRACT
Federated learning learns from scattered data by fusing collaborative models from local nodes. However, conventional coordinate-based model averaging by FedAvg ignored the random information encoded per parameter and may suffer from structural feature misalignment. In this work, we propose Fed2, a feature-aligned federated learning framework to resolve this issue by establishing a firm structure-feature alignment across the collaborative models. Fed2 is composed of two major designs: First, we design a feature-oriented model structure adaptation method to ensure explicit feature allocation in different neural network structures. Applying the structure adaptation to collaborative models, matchable structures with similar feature information can be initialized at the very early training stage. During the federated learning process, we then propose a feature paired averaging scheme to guarantee aligned feature distribution and maintain no feature fusion conflicts under either IID or non-IID scenarios. Eventually, Fed2 could effectively enhance the federated learning convergence performance under extensive homo- and heterogeneous settings, providing excellent convergence speed, accuracy, and computation/communication efficiency.
Supplemental Material
- A. Gonzalez-Garcia, D. Modolo, and V. Ferrari. Do semantic parts emerge in convolutional neural networks? International Journal of Computer Vision, 126(5):476--494, 2018.Google Scholar
Digital Library
- Andrew G Howard and et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861, 2017.Google Scholar
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097--1105, 2012.Google ScholarDigital Library
- Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50--60, 2020.Google ScholarCross Ref
- T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith. Federated optimization in heterogeneous networks. In Proceedings of the 3rd MLSys Conference, 2018.Google Scholar
- X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang. On the convergence of fedavg on non-iid data. In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2019.Google Scholar
- H Brendan McMahan and et al. Communication-efficient learning of deep networks from decentralized data. International Conference on Artificial Intelligence and Statistics (AISTATS), 2017.Google Scholar
- H. Mostafa. Robust federated learning through representation matching and adaptive hyper-parameters. arXiv:1912.13075, 2019.Google Scholar
- Z. Qin, F. Yu, C. Liu, and X. Chen. How convolutional neural network see the world-a survey of convolutional neural network visualization methods. arXiv preprint:1804.11191, 2018.Google Scholar
- Z. Qin, F. Yu, C. Liu, and X. Chen. Functionality-oriented convolutional filter pruning. In Proceedings of 30th British Machine Vision Conference (BMVC), 2019.Google Scholar
- W. Samek, A. Binder, G. Montavon, S. Lapuschkin, and K. Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(11):2660--2673, 2016.Google Scholar
- Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014.Google Scholar
- S. P. Singh and M. Jaggi. Model fusion via optimal transport. arXiv:1910.05653, 2019.Google Scholar
- H. Wang and et al. Federated learning with matched averaging. In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.Google Scholar
- Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.Google ScholarDigital Library
- Qiang Yang and et al. Federated learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 13(3):1--207, 2019.Google ScholarCross Ref
- Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1--19, 2019.Google Scholar
- Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv:1506.06579, 2015.Google Scholar
- Fuxun Yu, Zhuwei Qin, and Xiang Chen. Distilling critical paths in convolutional neural networks. arXiv:1811.02643, 2018.Google Scholar
- M. Yurochkin and et al. Probabilistic federated neural matching. 2018.Google Scholar
- M. Yurochkin and et al. Bayesian nonparametric federated learning of neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 7252--7261, 2019.Google Scholar
- Xiangyu Zhang and et al. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.Google Scholar
- Yue Zhao and et al. Federated learning with non-iid data. arXiv:1806.00582, 2018.Google Scholar
- B. Zhou and et al. Learning deep features for discriminative localization. In Proceedings of Computer Vision and Pattern Recognition (CVPR), pages 2921--2929, 2016.Google Scholar
Index Terms
- Fed2: Feature-Aligned Federated Learning
Recommendations
Deep learning: systematic review, models, challenges, and research directions
AbstractThe current development in deep learning is witnessing an exponential transition into automation applications. This automation transition can provide a promising framework for higher performance and lower complexity. This ongoing transition ...
Federated Knowledge Transfer for Heterogeneous Visual Models
MMAsia '22: Proceedings of the 4th ACM International Conference on Multimedia in AsiaFederated learning (FL) is a privacy-preserving distributed learning paradigm that enables collaborative training of machine learning models among multiple participants. However, despite recent progress, existing federated learning systems can still not ...
FLAMES2Graph: An Interpretable Federated Multivariate Time Series Classification Framework
KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data MiningIncreasing privacy concerns have led to decentralized and federated machine learning techniques that allow individual clients to consult and train models collaboratively without sharing private information. Some of these applications, such as medical ...
Comments