skip to main content
10.1145/3144789.3144815acmotherconferencesArticle/Chapter ViewAbstractPublication PagesiciipConference Proceedingsconference-collections
research-article

DAC-SGD: A Distributed Stochastic Gradient Descent Algorithm Based on Asynchronous Connection

Published: 17 July 2017 Publication History

Abstract

In the data mining practice, it happens that the algorithm used in mining tasks needs to deal with the multiple distributed data source, while the required datasets are located in different companies or organizations and reside in different system and technology environments. In traditional mining solutions or algorithms, data located in different source need to be copied and integrated into a homogenous computation environment, and then the mining can be executed, which leads to large data transmission and high storage costs. Even the data mining can be in feasible due to the data ownership problems. In this paper, a distributed asynchronous connection approach for the well-used stochastic gradient descent algorithm (SGD) was presented, and a distributed implementation for it was done to cope with the multiple distributed data source problems. In which, the main process of the algorithm was executed asynchronously in distributed computation node and the model can be trained locally in multiple data sources based on their own computation environment, so as to avoid the data integration and centralized processing. And the feasibility and performance for the proposed algorithm was evaluated based on experimental studies.

References

[1]
Feng N, Recht B, Re C, et al. HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent{J}. Advances in Neural Information Processing Systems, 2011, 24:693--701.
[2]
Li M, Andersen D G, Smola A, et al. Communication efficient distributed machine learning with the parameter server{J}. Advances in Neural Information Processing Systems, 2014, 1:19--27.
[3]
Alekh Agarwal, John C. Duchi. Distributed Delayed Stochastic Optimization, Decision & Control, 2011, 8350(2):5451--5452
[4]
Lian X, Huang Y, Li Y, et al. Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization{J}. Mathematics, 2015.
[5]
Nemirovski A, Juditsky A, Lan G, et al. Robust Stochastic Approximation Approach to Stochastic Programming{J}. Siam Journal on Optimization, 2009, 19(4):1574--1609.
[6]
Zhang T. Solving large scale linear prediction problems using stochastic gradient descent algorithms{C}//International Conference on Machine Learning. Omnipress. 2004:116.
[7]
W. Frawley and G. Piatetsky-Shapiro and C. Matheus (Fall 1992). Knowledge Discovery in Databases: An Overview. AI Magazine: pp. 213--228. ISSN 0738-4602.
[8]
D. Hand, H. Mannila, P. Smyth (2001). "Principles of Data Mining". MIT Press, Cambridge, MA. ISBN 0-262-08290-X.
[9]
Ge R, Huang F, Jin C, et al. Escaping From Saddle Points ---Online Stochastic Gradient for Tensor Decomposition{J}. Mathematics, 2015.
[10]
De Sa C, Olukotun K, Ré C. Global convergence of stochastic gradient descent for some non-convex matrix problems{C}// International Conference on International Conference on Machine Learning. JMLR.org, 2015:2332--2341.
[11]
Schmidt M, Roux N L, Bach F. Erratum to: Minimizing finite sums with the stochastic average gradient{J}. Mathematical Programming, 2013, 26(5):1--1.
[12]
IEEE. Accelerating Stochastic Gradient Descent using Predictive Variance Reduction{J}. Advances in Neural Information Processing Systems, 2013:315--323.
[13]
Murty K G, Kabadi S N. Some NP-complete problems in quadratic and nonlinear programming{J}. Mathematical Programming, 1987, 39(2):117--129.
[14]
Bishop, Christopher M. Pattern Recognition and Machine Learning (Information Science and Statistics){C}// Springer-Verlag New York, Inc. 2006:049901.
[15]
Murphy K P. Machine Learning: A Probabilistic Perspective{M}. MIT Press, 2012.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ICIIP '17: Proceedings of the 2nd International Conference on Intelligent Information Processing
July 2017
211 pages
ISBN:9781450352871
DOI:10.1145/3144789
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

  • Wanfang Data: Wanfang Data, Beijing, China
  • International Engineering and Technology Institute, Hong Kong: International Engineering and Technology Institute, Hong Kong

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 July 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Asynchronous Connection
  2. Distributed Data Mining
  3. Machine Learning
  4. Multiple Distributed Data Source
  5. Stochastic Gradient Descent

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

IIP'17

Acceptance Rates

ICIIP '17 Paper Acceptance Rate 32 of 202 submissions, 16%;
Overall Acceptance Rate 87 of 367 submissions, 24%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 56
    Total Downloads
  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 01 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media