Guo Zhenning
ABSTRACT
In this paper, we present a distributed machine learning algorithm over a network with fixed-delay tolerance. The network is directed and strongly connected. The training dataset is distributed to all agents in the network. We combine the distributed convex optimization (which utilizes double linear iterations) and corresponding machine learning algorithm. Each agent can only access its own local dataset. Suppose the delay between any pair of agents is time-invariant. The simulation shows that our algorithm is able to work under delayed transmission, in the sense that over time at each agent t the ratio of the estimate value xi(t) and scaling variable yi(t) can converge to the optimal point of the global cost function corresponding to the machine learning problem.
- Suyog Gupta, Wei Zhang, and Fei Wang. Model accuracy and runtime tradeoff in distributed deep learning: A systematic study. In Data Mining (ICDM), 2016 IEEE 16th International Conference on, pages 171--180. IEEE, 2016.Google Scholar
Cross Ref
- Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V Le, et al. Large scale distributed deep networks. In Advances in neural information processing systems, pages 1223--1231, 2012. Google ScholarDigital Library
- Sixin Zhang, Anna E Choromanska, and Yann LeCun. Deep learning with elastic averaging sgd. In Advances in Neural Information Processing Systems, pages 685--693, 2015. Google ScholarDigital Library
- H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication-efficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629, 2016.Google Scholar
- Zhanhong Jiang, Aditya Balu, Chinmay Hegde, and Soumik Sarkar. Collaborative deep learning in fixed topology networks. In Advances in Neural Information Processing Systems, pages 5904--5914, 2017. Google ScholarDigital Library
- Lin Xiao and Stephen Boyd. Fast linear iterations for distributed averaging. Systems & Control Letters, 53(1):65--78, 2004.Google ScholarCross Ref
- David Kempe, Alin Dobra, and Johannes Gehrke. Gossip-based computation of aggregate information. In Foundations of Computer Science, 2003. Proceedings. 44th Annual IEEE Symposium on, pages 482--491. IEEE, 2003. Google ScholarDigital Library
- Ji Liu and A Stephen Morse. Asynchronous distributed averaging using double linear iterations. In American Control Conference (ACC), 2012, pages 6620--6625. IEEE, 2012.Google ScholarCross Ref
- Christoforos N Hadjicostis and Themistoklis Charalambous. Average consensus in the presence of delays in directed graph topologies. IEEE Transactions on Automatic Control, 59(3):763--768, 2014.Google ScholarCross Ref
- Angelia Nedic and Asuman Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control, 54(1):48--61, 2009.Google ScholarCross Ref
- Kun Yuan, Qing Ling, and Wotao Yin. On the convergence of decentralized gradient descent. SIAM Journal on Optimization, 26(3):1835--1854, 2016.Google ScholarDigital Library
- Angelia Nedić and Alex Olshevsky. Distributed optimization over time-varying directed graphs. IEEE Transactions on Automatic Control, 60(3):601--615, 2015.Google ScholarCross Ref
- https://www.kaggle.com/andonians/random-linear-regression.Google Scholar
Index Terms
- Distributed Machine Learning over Directed Network with Fixed Communication Delays
Recommendations
Distributed Machine Learning: Foundations, Trends, and Practices
WWW '17 Companion: Proceedings of the 26th International Conference on World Wide Web CompanionIn recent years, artificial intelligence has achieved great success in many important applications. Both novel machine learning algorithms (e.g., deep neural networks), and their distributed implementations play very critical roles in the success. In ...
Load Balancing for Machine Learning Platform in Heterogeneous Distribute Computing Environment
RACS '20: Proceedings of the International Conference on Research in Adaptive and Convergent SystemsWith the recent rapid development of computing power, interest in machine learning research on large data sets is increasing significantly. The machine learning is used in a wide variety of fields, from information retrieval, data mining, and speech ...
Machine Learning: The State of the Art
The two fundamental problems in machine learning (ML) are statistical analysis and algorithm design. The former tells us the principles of the mathematical models that we establish from the observation data. The latter defines the conditions on which ...
Comments