Zhao Yang
ABSTRACT
Federated learning (FL), a new distributed technology, allows us to train the global model on the edge and embedded devices without local data sharing. However, due to the wide distribution of different types of devices, FL faces severe heterogeneity issues. The accuracy and efficiency of FL deployment at the edge are severely impacted by heterogeneous data and heterogeneous systems. In this paper, we perform joint FL model personalization for heterogeneous systems and heterogeneous data to address the challenges posed by heterogeneities. We begin by using model inference efficiency as a starting point to personalize network scale on each node. Furthermore, it can be used to guide the efficient FL training process, which can help to ease the problem of straggler devices and improve FL's energy efficiency. During FL training, federated search is then used to acquire highly accurate personalized network structures. By taking into account the unique characteristics of FL deployment at edge devices, the personalized network structures obtained by our federated search framework with a lightweight search controller can achieve competitive accuracy with state-of-the-art (SOTA) methods, while reducing inference and training energy consumption by up to 3.57× and 1.82×, respectively.
- K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konecny, S. Mazzocchi, H. B. McMahan, T. V. Overveldt, D. Petrou, D. Ramage, and J. Roselander. 2019. Towards federated learning at scale: System design. In SysML.Google Scholar
- Y. H. Chen, T. J. Yang, J. S. Emer, and V. Sze. 2019. Eyeriss v2 : A Flexible Accelerator for Emerging Deep Neural Networks on Mobile Devices. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9, 2 (2019), 292--308.Google ScholarCross Ref
- R. C. Geyer, T. Klein, and M. Nabi. 2017. Differentially private federated learning: A client level perspective. In arXiv:1712.07557.Google Scholar
- A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage. 2018. Federated learning for mobile keyboard prediction. In arXiv:1811.03604.Google Scholar
- C. He, M. Annavaram, and S. Avestimehr. 2020. Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search. In arXiv:2004.08546.Google Scholar
- K. Hegde, J. Yu, R. Agrawal, M. Yan, M. Pellauer, and C. W. Fletcher. 2018. Ucnn: Exploiting computational reuse in deep neural networks via weight repetition. In 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture. 674--687.Google Scholar
- H. Kwon, A. Samajdar, and T. Krishna. 2018. MAERI: Enabling Flexible Dataflow Mapping over DNN Accelerators via Programmable Interconnects. In Proceedings of the International Conference on Architectural Support for Programming Languages and Operation Systems.Google Scholar
- A. Li, J. Sun, B. Wang, L. Duan, S. Li, Y. Chen, and H. Li. 2021. LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets. In 2021 ACM/IEEE 6th Symposium on Edge Computing (SEC).Google Scholar
- H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. 2017. Pruning filters for efficient convnets. In 5th International Conference on Learning Representations.Google Scholar
- P. P. Liang, T. Liu, Z. Liu, N. B. Allen, R. P. Auerbach, D. Brent, R. Salakhutdinov, and L. P. Morency. 2020. Think locally, act globally: Federated learning with local and global representations. In arXiv:2001.01523.Google Scholar
- J. Luo, J. Yang, X. Ye, X. Guo, and W. Zhao. 2021. FedSkel: Efficient Federated Learning on Heterogeneous Systems with Skeleton Gradients Update. In Proceedings of the 30th ACM International Conference on Information Knowledge Management. 3283--3287.Google Scholar
- Y. Ma, Y. Cao, S. Vrudhula, and J. Seo. 2017. Optimizing Loop Operation and Dataflow in FPGA Acceleration of Deep Convolutional Neural Networks. In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. 45--54.Google Scholar
- H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics. 1273--1282.Google Scholar
- Monsoon. [n. d.]. High voltage power monitor. https://www.msoon.com/high-voltage-power-monitorGoogle Scholar
- E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin. 2014. Large-scale evolution of image classifiers. In Proceedings of the 34th International Conference on Machine Learning, Vol. 70. 2902--2911.Google Scholar
- J. Weng, S. Liu, Z. Wang, V. Dadu, and T. Nowatzki. 2020. A Hybrid Systolic-Dataflow Architecture for Inductive Matrix Algorithms. In 2020 IEEE International Symposium on High Performance Computer Architecture. 703--716.Google Scholar
- M. Xu, Y. Zhao, K. Bian, G. Huang, Q. Mei, and X. Liu. 2020. Federated Neural Architecture Search. In arXiv:2002.06352.Google Scholar
- Y. Jin Y. Chen, X. Sun. 2019. Communication-efficient federated deep learning with layerwise asynchronous model update and temporally weighted aggregation. IEEE transactions on neural networks and learning systems 31, 10 (2019), 4229--4238.Google ScholarCross Ref
- F. Yu, W. Zhang, Z. Qin, Z. Xu, D. Wang, C. Liu, Z. Tian, and X. Chen. 2021. Fed2: Feature-Aligned Federated Learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining. 2066--2074.Google Scholar
- B. Zoph and Q. V. Le. 2017. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations.Google Scholar
- B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. 2018. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8697--8710.Google Scholar
Index Terms
- Personalized Heterogeneity-Aware Federated Search Towards Better Accuracy and Energy Efficiency
Recommendations
AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning
MICRO '21: MICRO-54: 54th Annual IEEE/ACM International Symposium on MicroarchitectureFederated learning enables a cluster of decentralized mobile devices at the edge to collaboratively train a shared machine learning model, while keeping all the raw training samples on device. This decentralized training approach is demonstrated as a ...
Joint heterogeneity-aware personalized federated search for energy efficient battery-powered edge computing
AbstractThe limited battery capacity of edge devices has a significant impact on the deployment of Federated Learning (FL). As a result, maintaining computation sustainability is a critical issue for edge FL. Furthermore, the heterogeneities ...
Highlights- Multiple heterogeneity issues bring computational efficiency and sustainability problems to edge FL.
Energy-efficient Personalized Federated Search with Graph for Edge Computing
Special Issue ESWEEK 2023Federated Learning (FL) is a popular method for privacy-preserving machine learning on edge devices. However, the heterogeneity of edge devices, including differences in system architecture, data, and co-running applications, can significantly impact the ...
Comments