Cited By
View all- Wei MYu WChen D(2025)AccDFL: Accelerated Decentralized Federated Learning for Healthcare IoT NetworksIEEE Internet of Things Journal10.1109/JIOT.2024.348612212:5(5329-5345)Online publication date: 1-Mar-2025
Accelerating the Decentralized Federated Learning via Manipulating Edges
Authors: 
Mingyang Zhou, Gang Liu, KeZhong Lu, Rui Mao, Hao LiaoAuthors Info & Claims
Mingyang Zhou, Gang Liu, KeZhong Lu, Rui Mao, Hao LiaoAuthors Info & ClaimsPages 2945 - 2954
Abstract
Federated learning enables collaborative AI training across organizations without compromising data privacy. Decentralized federated learning (DFL) improves this by offering enhanced reliability and security through peer-to-peer (P2P) model sharing. However, DFL faces challenges in terms of slow convergence rate due to complex P2P graphs. To address this issue, we propose an efficient algorithm to accelerate DFL by introducing a limited number of k of edges into the P2P graphs. Specifically, we establish a connection between the convergence rate and the second smallest eigenvalue of the laplacian matrix of the P2P graph. We prove that finding the optimal set of edges to maximize this eigenvalue is an NP-complete problem. Our quantitative analysis shows the positive effect of strategic edge additions on improving this eigenvalue. Based on the analysis, we then propose an efficient algorithm to compute the best set of candidate edges to maximize the second smallest eigenvalue, and consequently the convergence rate is maximized. Our algorithm has a low time complexity of O(krn^2). Experimental results on diverse datasets validate the effectiveness of our proposed algorithms in accelerating DFL convergence.
Supplemental Material
MP4 File
Supplemental video
- Download
- 20.74 MB
References
[1]
Lun Wang, Yang Xu, Hongli Xu, Min Chen, and Liusheng Huang. Accelerating decentralized federated learning in heterogeneous edge computing. IEEE Transactions on Mobile Computing, 22(9):5001--5016, 2023.
[2]
Liangqi Yuan, Lichao Sun, Philip S Yu, and Ziran Wang. Decentralized federated learning: A survey and perspective. arXiv preprint arXiv:2306.01603, 2023.
[3]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273--1282. PMLR, 2017.
[4]
Angelia Nedi?, Alex Olshevsky, and Michael G Rabbat. Network topology and communication-computation tradeoffs in decentralized optimization. Proceedings of the IEEE, 106(5):953--976, 2018.
[5]
Jianyu Wang, Anit Kumar Sahu, Gauri Joshi, and Soummya Kar. Matcha: A matching-based link scheduling strategy to speed up distributed optimization. IEEE Transactions on Signal Processing, 70:5208--5221, 2022.
[6]
Xi Lin, Jun Wu, Jianhua Li, Xi Zheng, and Gaolei Li. Friend-as-learner: Sociallydriven trustworthy and efficient wireless federated edge learning. IEEE Transactions on Mobile Computing, 22(1):269--283, 2023.
[7]
Yuntao Wang, Zhou Su, Yanghe Pan, Tom H Luan, Ruidong Li, and Shui Yu. Social-aware clustered federated learning with customized privacy preservation. arXiv preprint arXiv: 2212.13992, 2022.
[8]
Mahdi Jalili and Xinghuo Yu. Enhancement of synchronizability in networks with community structure through adding efficient inter-community links. IEEE Transactions on Network Science and Engineering, 3(2):106--116, 2016.
[9]
Mark Newman. Networks: An Introduction. Oxford University Press, 03 2010.
[10]
Chaoyang He, Conghui Tan, Hanlin Tang, Shuang Qiu, and Ji Liu. Central server free federated learning over single-sided trust social networks. arXiv preprint arXiv:1910.04956, 2019.
[11]
Chen Chen, Hanghang Tong, B Aditya Prakash, Tina Eliassi-Rad, Michalis Faloutsos, and Christos Faloutsos. Eigen-optimization on large graphs by edge manipulation. ACM Transactions on Knowledge Discovery from Data (TKDD), 10(4):1--30, 2016.
[12]
Ziran Wang, Rohit Gupta, Kyungtae Han, Haoxin Wang, Akila Ganlath, Nejib Ammar, and Prashant Tiwari. Mobility digital twin: Concept, architecture, case study, and future challenges. IEEE Internet of Things Journal, 9(18):17452--17467, 2022.
[13]
Sarthak Pati, Ujjwal Baid, Brandon Edwards, Micah Sheller, Shih-Han Wang, G Anthony Reina, Patrick Foley, Alexey Gruzdev, Deepthi Karkada, Christos Davatzikos, et al. Federated learning enables big data for rare cancer boundary detection. Nature communications, 13(1):7346, 2022.
[14]
Anusha Lalitha, Osman Cihan Kilinc, Tara Javidi, and Farinaz Koushanfar. Peerto- peer federated learning on graphs. arXiv preprint arXiv:1901.11173, 2019.
[15]
Chaoyang He, Emir Ceyani, Keshav Balasubramanian, Murali Annavaram, and Salman Avestimehr. Spreadgnn: Serverless multi-task federated learning for graph neural networks. arXiv preprint arXiv:2106.02743, 2021.
[16]
Hong Xing, Osvaldo Simeone, and Suzhi Bi. Federated learning over wireless device-to-device networks: Algorithms and convergence analysis. IEEE Journal on Selected Areas in Communications, 39(12):3723--3741, 2021.
[17]
Stefanie Warnat-Herresthal, Hartmut Schultze, Krishnaprasad Lingadahalli Shastry, Sathyanarayanan Manamohan, Saikat Mukherjee, Vishesh Garg, Ravi Sarveswara, Kristian Händler, Peter Pickkers, N Ahmad Aziz, et al. Swarm learning for decentralized and confidential clinical machine learning. Nature, 594(7862):265--270, 2021.
[18]
Tuncer Can Aysal, Mehmet Ercan Yildiz, Anand D Sarwate, and Anna Scaglione. Broadcast gossip algorithms for consensus. IEEE Transactions on Signal processing, 57(7):2748--2761, 2009.
[19]
Aurélien Bellet, Rachid Guerraoui, Mahsa Taziki, and Marc Tommasi. Personalized and private peer-to-peer machine learning. In International Conference on Artificial Intelligence and Statistics, pages 473--481. PMLR, 2018.
[20]
John N Tsitsiklis. Problems in decentralized decision making and computation. PhD thesis, Massachusetts Institute of Technology, 1984.
[21]
Ali Jadbabaie, Jie Lin, and A Stephen Morse. Coordination of groups of mobile autonomous agents using nearest neighbor rules. IEEE Transactions on automatic control, 48(6):988--1001, 2003.
[22]
Alex Olshevsky. Linear time average consensus and distributed optimization on fixed graphs. SIAM Journal on Control and Optimization, 55(6):3990--4014, 2017.
[23]
Lin Xiao and Stephen Boyd. Fast linear iterations for distributed averaging. Systems & Control Letters, 53(1):65--78, 2004.
[24]
Angelia Nedic and Asuman Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control, 54(1):48--61, 2009.
[25]
Angelia Nedi? and Alex Olshevsky. Distributed optimization over time-varying directed graphs. IEEE Transactions on Automatic Control, 60(3):601--615, 2014.
[26]
S Sundhar Ram, A Nedi?, and Venugopal V Veeravalli. A new class of distributed optimization algorithms: Application to regression of distributed data. Optimization Methods and Software, 27(1):71--88, 2012.
[27]
David Kempe, Alin Dobra, and Johannes Gehrke. Gossip-based computation of aggregate information. In 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings., pages 482--491. IEEE, 2003.
[28]
Florence Bénézit, Vincent Blondel, Patrick Thiran, John Tsitsiklis, and Martin Vetterli. Weighted gossip: Distributed averaging using non-doubly stochastic matrices. In 2010 ieee international symposium on information theory, pages 1753--1757. IEEE, 2010.
[29]
Alejandro D Domínguez-García and Christoforos N Hadjicostis. Distributed strategies for average consensus in directed graphs. In 2011 50th IEEE Conference on Decision and Control and European Control Conference, pages 2124--2129. IEEE, 2011.
[30]
S Sundhar Ram, Angelia Nedi?, and Venugopal V Veeravalli. Distributed stochastic subgradient projection algorithms for convex optimization. Journal of optimization theory and applications, 147:516--545, 2010.
[31]
Soomin Lee, Angelia Nedi?, and Maxim Raginsky. Stochastic dual averaging for decentralized online optimization on time-varying communication graphs. IEEE Transactions on Automatic Control, 62(12):6407--6414, 2017.
[32]
Konstantinos I Tsianos and Michael G Rabbat. Efficient distributed online prediction and stochastic optimization with approximate distributed averaging. IEEE Transactions on Signal and Information Processing over Networks, 2(4):489--506, 2016.
[33]
Jianyu Wang, Anit Kumar Sahu, Zhouyi Yang, Gauri Joshi, and Soummya Kar. Matcha: Speeding up decentralized sgd via matching decomposition sampling. In 2019 Sixth Indian Control Conference (ICC), pages 299--300. IEEE, 2019.
[34]
Pan Zhou, Qian Lin, Dumitrel Loghin, Beng Chin Ooi, Yuncheng Wu, and Hongfang Yu. Communication-efficient decentralized machine learning over heterogeneous networks. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pages 384--395. IEEE, 2021.
[35]
Zhenheng Tang, Shaohuai Shi, and Xiaowen Chu. Communication-efficient decentralized learning with sparsification and adaptive peer selection. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS), pages 1207--1208. IEEE, 2020.
[36]
Alex Arenas, Albert Díaz-Guilera, Jurgen Kurths, Yamir Moreno, and Changsong Zhou. Synchronization in complex networks. Physics reports, 469(3):93--153, 2008.
[37]
Stefano Boccaletti, Alexander N Pisarchik, Charo I Del Genio, and Andreas Amann. Synchronization: from coupled systems to complex networks. Cambridge University Press, 2018.
[38]
LouisMPecora and Thomas L Carroll. Master stability functions for synchronized coupled systems. Physical review letters, 80(10):2109, 1998.
[39]
JF Heagy, TL Carroll, and LM Pecora. Synchronous chaos in coupled oscillator systems. Physical Review E, 50(3):1874, 1994.
[40]
Scott Freitas, Diyi Yang, Srijan Kumar, Hanghang Tong, and Duen Horng Chau. Graph vulnerability and robustness: A survey. IEEE Transactions on Knowledge and Data Engineering, 35(6):5915--5934, 2022.
[41]
Zuobai Zhang, Zhongzhi Zhang, and Guanrong Chen. Minimizing spectral radius of non-backtracking matrix by edge removal. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2657--2667, 2021.
[42]
Aric Hagberg and Daniel A Schult. Rewiring networks for synchronization. Chaos: An interdisciplinary journal of nonlinear science, 18(3), 2008.
[43]
Qingyun Wang, Matja? Perc, Zhisheng Duan, and Guanrong Chen. Impact of delays and rewiring on the dynamics of small-world neuronal networks with two types of coupling. Physica A: Statistical Mechanics and its Applications, 389(16):3299--3306, 2010.
[44]
Stefan Hellrigel, Nicholas Jarman, and Cees van Leeuwen. Adaptive rewiring in weighted networks. Cognitive Systems Research, 55:205--218, 2019.
[45]
Mahdi Jalili and Xinghuo Yu. Enhancing pinning controllability of complex networks through link rewiring. IEEE Transactions on Circuits and Systems II: Express Briefs, 64(6):690--694, 2017.
[46]
Jianyu Wang and Gauri Joshi. Cooperative sgd: A unified framework for the design and analysis of local-update sgd algorithms. J. Mach. Learn. Res., 22(1), jan 2021.
[47]
Xiangru Lian, Ce Zhang, Huan Zhang, Cho-Jui Hsieh,Wei Zhang, and Ji Liu. Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 1--11. Curran Associates, Inc., 2017.
[48]
Chen Chen, Ruiyue Peng, Lei Ying, and Hanghang Tong. Network connectivity optimization: Fundamental limits and effective algorithms. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1167--1176, 2018.
[49]
Gilbert W Stewart. Matrix perturbation theory. 1990.
[50]
Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008.
[51]
Mark EJ Newman. Fast algorithm for detecting community structure in networks. Physical review E, 69(6):066133, 2004.
[52]
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases, 1(4), 2009.
[53]
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321--357, 2002.
[54]
Luke Nicholas Darlow, Elliot J. Crowley, Antreas Antoniou, and Amos J. Storkey. CINIC-10 is not imagenet or CIFAR-10. CoRR, abs/1810.03505, 2018.
[55]
Luis Candanedo. Occupancy Detection . UCI Machine Learning Repository, 2016.
Index Terms
- Accelerating the Decentralized Federated Learning via Manipulating Edges
Recommendations
BTTB preconditioners for BTTB systems
In this paper, we consider solving the BTTB system ${\cal T}_{m,n}[f] {\bf{x}} = {\bf{b}}$ by the preconditioned conjugate gradient (PCG) method, where ${\cal T}_{m,n}[f]$ denotes the m m block Toeplitz matrix with n n Toeplitz blocks (BTTB) generated by a (2 , 2 )-periodic continuous function f ( ...
Stair Matrices and Their Generalizations with Applications to Iterative Methods I: A Generalization of the Successive Overrelaxation Method
Stair matrices and their generalizations are introduced. Some properties of the matrices are presented. Like triangular matrices this class of matrices provides bases of matrix splittings for iterative methods. A remarkable feature of iterative methods ...
An alternating preconditioner for saddle point problems
Based on matrix splittings, a new alternating preconditioner with two parameters is proposed for solving saddle point problems. Some theoretical analyses for the eigenvalues of the associated preconditioned matrix are given. The choice of the parameters ...
Comments
Information & Contributors
Information
Published In
May 2024
4826 pages
ISBN:9798400701719
DOI:10.1145/3589334
- General Chairs:
- Tat-Seng Chua,
- Chong-Wah Ngo,
- Proceedings Chair:
- Roy Ka-Wei Lee,
- Program Chairs:
- Ravi Kumar,
- Hady W. Lauw
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
WWW '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,899 of 8,196 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 304Total Downloads
- Downloads (Last 12 months)304
- Downloads (Last 6 weeks)29
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
Cited By
View all- Wei MYu WChen D(2025)AccDFL: Accelerated Decentralized Federated Learning for Healthcare IoT NetworksIEEE Internet of Things Journal10.1109/JIOT.2024.348612212:5(5329-5345)Online publication date: 1-Mar-2025
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in