Abstract
Deep Belief Network (DBN) is widely used for modelling and analysis of all kinds of actual problems. However, it’s easy to have a computational bottleneck problem when training DBN in a single computational node. And traditional parallel full-batch gradient descent exists the problem that the speed of convergence is slow when we use it to train DBN. To solve this problem, the article proposes a parallel mini-batch gradient descent algorithm based on Spark and uses it to train DBN. The experiment shows the method is faster than parallel full-batch gradient and the convergence result is better when batch size is relatively small. We use the method to train the DBN, and apply it to text classification. We also discuss how the size of batch impacts on the weights of network. The experiments show that it can improve the precision and recall of text classification compared with SVM when batch size is small.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ng, A., Ngiam, J., Foo, C.Y.: Deep learning (2014)
Bengio, Y.: Learning deep architectures for AI. Foundations and trends® in Machine Learning 2(1), 1–127 (2009)
Hinton, G., Osindero, S., The, Y.W.: A fast learning algorithm for deep belief nets. Neural Computation 18(7), 1527–1554 (2006)
Dean, J., Corrado, G., Monga, R.: Large scale distributed deep networks. In: Advances in Neural Information Processing Systems, pp. 1223–1231 (2012)
Seide, F., Fu, H., Droppo, J.: 1-Bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs. In: Fifteenth Annual Conference of the International Speech Communication Association (2014)
De Grazia, M.D.F., Stoianov, I., Zorzi, M.: Parallelization of deep networks. In: Proceedings of 2012 European Symposium on Artificial NN, Computational Intelligence and Machine Learning, pp. 621–626 (2012)
Sainath, T.N., Kingsbury, B., Ramabhadran, B., et al.: Making deep belief networks effective for large vocabulary continuous speech recognition. In: 2011 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 30–35. IEEE (2011)
Haykin, S.S.: Neural networks and learning machines. Pearson Education, Upper Saddle River (2009)
Bengio, Y., Lamblin, P., Popovici, D.: Greedy layer-wise training of deep networks. Advances in Neural Information Processing Systems 19, 153 (2007)
Fischer, A., Igel, C.: An introduction to restricted Boltzmann machines. In: Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, pp. 14–36. Springer, Heidelberg (2012)
Liu, J.S.: The collapsed Gibbs sampler in Bayesian computations with applications to a gene regulation problem. Journal of the American Statistical Association 89(427), 958–966 (1994)
Zaharia, M., Chowdhury, M., Franklin, M.J.: Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, pp. 10–10 (2010)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Communications of the ACM 51(1), 107–113 (2008)
Zaharia, M., Chowdhury, M., Das, T.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation. USENIX Association, p. 2 (2012)
Salton, G., McGill, M.J.: Introduction to modern information retrieval (1983)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Cognitive Modeling 5 (1988)
Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2(3), 27 (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Yang, J., He, S. (2016). The Optimization of Parallel DBN Based on Spark. In: Lavangnananda, K., Phon-Amnuaisuk, S., Engchuan, W., Chan, J. (eds) Intelligent and Evolutionary Systems. Proceedings in Adaptation, Learning and Optimization, vol 5. Springer, Cham. https://doi.org/10.1007/978-3-319-27000-5_13
Download citation
DOI: https://doi.org/10.1007/978-3-319-27000-5_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-26999-3
Online ISBN: 978-3-319-27000-5
eBook Packages: EngineeringEngineering (R0)