Distributed B-SDLM: Accelerating the Training Convergence of Deep Neural Networks Through Parallelism

Abstract

This paper proposes an efficient asynchronous stochastic second order learning algorithm for distributed learning of neural networks (NNs). The proposed algorithm, named distributed bounded stochastic diagonal Levenberg-Marquardt (distributed B-SDLM), is based on the B-SDLM algorithm that converges fast and requires only minimal computational overhead than the stochastic gradient descent (SGD) method. The proposed algorithm is implemented based on the parameter server thread model in the MPICH implementation. Experiments on the MNIST dataset have shown that training using the distributed B-SDLM on a 16-core CPU cluster allows the convolutional neural network (CNN) model to reach the convergence state very fast, with speedups of $6.03{\times }$ and $12.28{\times }$ to reach 0.01 training and 0.08 testing loss values, respectively. This also results in significantly less time taken to reach a certain classification accuracy ($5.67{\times }$ and $8.72{\times }$ faster to reach $99\,\%$ training and $98\,\%$ testing accuracies on the MNIST dataset, respectively).