Abstract
Large-scale distributed deep learning is of great importance in various applications. For distributed training, the inter-node gradient communication often becomes the performance bottleneck. Gradient sparsification has been proposed to reduce the communication overhead. However, the sparsification only arbitrarily selects a small fraction of gradients, while the efficiency of selection dimension has been overlooked. Furthermore, gradient staleness is inevitable after applying the sparsification which will ultimately lead to model divergence. In this paper, we propose a staleness-compensated sparse stochastic gradient descent algorithm, GradSA, to improve the training efficiency. Layer-level gradients are sparsified to reduce the communication overhead, which conform to the characteristics of the network structure, and historical accumulation of the approximated gradients is utilized to speed up convergence. We demonstrate the model convergence acceleration and the efficiency of our layer-level selection over existing state-of-the-art works, such as DGC, TernGrad, and 8-Bit quantization.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We use a famous DNN, ResNet, used in vision tasks. Except where otherwise indicated, all experiments are performed on distributed MXNet [3] with 4 workers.
- 2.
Since DGC [11] is not yet open-source, we basically implement its core optimizations, i.e., momentum correction, warm-up start, gradient clipping et al. in MXNet.
References
Aji, A.F., Heafield, K.: Sparse communication for distributed gradient descent. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, pp. 440–445. Association for Computational Linguistics, Copenhagen, Denmark (2017)
Banner, R., Hubara, I., Hoffer, E., Soudry, D.: Scalable methods for 8-bit training of neural networks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS 2018, pp. 5151–5159. Curran Associates Inc., Montreal, Canada (2018)
Chen, T., et al.: Mxnet: a flexible and efficient machine learning library for heterogeneous distributed systems. In: Proceedings of Workshop on Machine Learning Systems at the 28th Annual Conference on Neural Information Processing Systems, LearningSys 2015, pp. 1–6. Montreal, Canada (2015)
Deng, J., Dong, W., Socher, R., Li, L., Li, F.: ImageNet: a large-scale hierarchical image database. In: Proceedings of the 22nd IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 248–255. IEEE, Miami, USA (2009)
Dryden, N., Jacobs, S.A., Moon, T., Van Essen, B.: Communication quantization for data-parallel training of deep neural networks. In: Proceedings of the Workshop on Machine Learning in High Performance Computing Environments, MLHPC 2016, pp. 1–8. IEEE, Salt Lake City, USA (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, pp. 770–778. IEEE, Las Vegas, USA (2016)
Jeffreys, H., Jeffreys, B.: Methods of Mathematical Physics. Cambridge University Press, Cambridge (1999)
Krizhevsky, A., Hinton, G.E.: Learning multiple layers of features from tiny images. Technical report, University of Toronto (2009)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th Annual Conference on Neural Information Processing Systems, NIPS 2012, pp. 1097–1105. Curran Associates Inc, Lake Tahone, USA (2012)
Li, M., et al.: Scaling distributed machine learning with the parameter server. In: Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014, pp. 583–598. USENIX, Broomfield, USA (2014)
Lin, Y., Han, S., Mao, H., Wang, Y., Dally, B.: Deep gradient compression: reducing the communication bandwidth for distributed training. In: Proceedings of the 6th International Conference on Learning Representations, ICLR 2018. Curran Associates Inc, Vancouver, Canada (2018)
Ma, Z., Zhang, Z., Ding, Z., Fan, P., Li, H.: Key techniques for 5G wireless communications: network architecture, physical layer, and MAC layer perspectives. Sci. China Inf. Sci. 58(4), 41301:1–041301:20 (2015)
Seide, F., Fu, H., Droppo, J., Li, G., Yu, D.: 1-Bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In: Proceedings of the 16th Annual Conference of the International Speech Communication Association, INTERSPEECH 2014, Singapore, pp. 1058–1062 (2014)
Strom, N.: Scalable distributed DNN training using commodity GPU cloud computing. In: Proceedings of the 16th Annual Conference of the International Speech Communication Association, INTERSPEECH 2015, Dresden, Germany, pp. 1488–1492 (2015)
Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: Proceedings of the 30th International Conference on Machine Learning, ICML 2013, pp. 1139–1147. PMLR, Atlanta, USA (2013)
Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the 28th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, pp. 1–9. IEEE, Boston, USA (2015)
Tang, H., Yu, C., Lian, X., Zhang, T., Liu, J.: DoubleSqueeze: parallel stochastic gradient descent with double-pass error-compensated compression. In: Proceedings of the 36th International Conference on Machine Learning, ICML 2019, pp. 6155–6165. PMLR, Long Beach, USA (2019)
Wangni, J., Wang, J., Liu, J., Zhang, T.: Gradient sparsification for communication-efficient distributed optimization. In: Proceedings of the 32th Annual Conference on Neural Information Processing Systems, NIPS 2018, pp. 1299–1309. Curran Associates Inc, Montreal, Canada (2018)
Wen, W., et al.: Terngrad: ternary gradients to reduce communication in distributed deep learning. In: Proceedings of the 31st Annual Conference on Neural Information Processing Systems, NIPS 2017, pp. 1509–1519. Curran Associates Inc, Long Beach, USA (2017)
Wu, J., Huang, W., Huang, J., Zhang, T.: Error compensated quantized SGD and its applications to large-scale distributed optimization. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, pp. 5325–5333. PMLR, Stockholmsmässan, Sweden (2018)
Yan, F., Ruwase, O., He, Y., Chilimbi, T.: Performance modeling and scalability optimization of distributed deep learning systems. In: Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2015, pp. 1355–1364. ACM, Sydney, Australia (2015)
Zheng, S., et al.: Asynchronous stochastic gradient descent with delay compensation. In: Proceedings of the 34th International Conference on Machine Learning, ICML 2017, pp. 4120–4129. PMLR, Sydney, Australia (2017)
Acknowledgment
This work is supported by National Natural Science Foundation of China under grants No. 61832006 and No. 61672250.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, B., Jiang, W., Zhao, S., Jin, H., He, B. (2020). GradSA: Gradient Sparsification and Accumulation for Communication-Efficient Distributed Deep Learning. In: Yu, Z., Becker, C., Xing, G. (eds) Green, Pervasive, and Cloud Computing. GPC 2020. Lecture Notes in Computer Science(), vol 12398. Springer, Cham. https://doi.org/10.1007/978-3-030-64243-3_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-64243-3_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64242-6
Online ISBN: 978-3-030-64243-3
eBook Packages: Computer ScienceComputer Science (R0)