Abstract
In this paper, we propose a new fast training methodology for learning of Deep Neural Networks (DNNs) via Singular Value Decomposition (SVD). The fast training methodology uses a supervised pre-adjusting process to adjust roughly parameters of weight matrices of DNNs and change distributions of singular values. SVD is applied to pre-adjusted DNNs, reducing quantities of parameters in DNNs. An unconventional Back Propagation (BP) algorithm is used to train the models restructured by SVD, which has lower time complexity than the conventional BP algorithm. Experimental results indicate that on Large Vocabulary Continuous Speech Recognition (LVCSR) tasks, using the fast training methodology, the unconventional BP algorithm achieves almost 2 times speed-up without any loss of recognition performance and almost 4 times speed-up with only a tiny loss of recognition performance.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Geoffrey, E.H., Li, D., Dong, Y., George, E.D., Abdel-rahman, M., Navdeep, J., Andrew, S., Vincent, V., Patrick, N., Tara, S., Brian, K.: Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups. IEEE Signal Processing Magazine 1(6), 82–97 (2012)
George, E.D., Dong, Y., Li, D., Alex, A.: Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition. IEEE Transactions on Audio, Speech and Language Processing 20(1), 30–42 (2012)
Abdel-rahman, M., George, E.D., Geoffrey, E.H.: Acoustic Modeling using Deep Belief Networks. IEEE Transactions on Audio, Speech, and Language Processing 20(1), 14–22 (2012)
Navdeep, J., Patrick, N., Andrew, W.S., Vincent, V.: Application of Pretrained Deep Neural Networks to Large Vocabulary Speech Recognition. In: Proceedings of Interspeech (2012)
Matthew, D.Z., Marc’Aurelio, R., Rajat, M., Mark, Z.M., Ke, Y., Quoc, V.L., Patrick, N., Andrew, W.S., Vincent, V., Jeffrey, D., Geoffrey, E.H.: On Rectified Linear Units for Speech Processing. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3517–3521 (2013)
Alex, G., Abdel-rahman, M., Geoffrey, E.H.: Speech Recognition with Deep Recurrent Neural Networks. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6645–6649 (2013)
Li, D., Geoffrey, E.H., Brian, K.: New Types of Deep Neural Network Learning for Speech Recognition and Related Applications: An Overview. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8599–8603 (2013)
Andrew, L.M., Awni, Y.H., Andrew, Y.N.: Rectifier Nonlinearities Improve Neural Network Acoustic Models. In: Proceedings of International Conference on Machine Learning, ICML (2013)
Dong, Y., Li, D., Frank, S.: The Deep Tensor Neural Network With Applications to Large Vocabulary Speech Recognition. IEEE Transactions on Audio, Speech and Language Processing 21(2), 388–396 (2013)
Hang, S., Gang, L., Dong, Y., Frank, S.: Error Back Propagation for Sequence Training of Context-Dependent Deep Networks for Conversational Speech Transcription. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6664–6668 (2013)
Jeffrey, D., Greg, C., Rajat, M., Kai, C., Matthieu, D., Quoc, V.L., Mark, Z.M., Marc’Aurelio, R., Andrew, W.S., Paul, A.T., Ke, Y., Andrew, Y.N.: Large Scale Distributed Deep Networks. In: Proceedings of Annual Conference on Neural Information Processing Systems (NIPS), pp. 1232–1240 (2012)
Georg, H., Vincent, V., Andrew, W.S., Patrick, N., Marc’Aurelio, R., Matthieu, D., Jeffrey, D.: Multilingual Acoustic Models Using Distributed Deep Neural Networks. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8619–8623 (2013)
Shanshan, Z., Ce, Z., Zhao, Y., Rong, Z., Bo, X.: Asynchronous Stochastic Gradient Descent for DNN Training. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6660–6663 (2013)
Zhongwen, L., Hongzhi, L., Xincai, W.: Artificial Neural Network Computation on Graphic Process Unit. In: Proceedings of IEEE International Joint Conference on Neural Networks (IJCNN), vol. 1, pp. 622–626 (2005)
Virginia, C.K., Alan, J.L.: The Singular Value Decomposition: Its Computation and some Applications. IEEE Transactions on Automatic Control 25(2), 164–176 (1980)
Jian, X., Jinyu, L., Yifan, G.: Restructuring of Deep Neural Network Acoustic Models with Singular Value Decomposition. In: Proceedings of Interspeech, pp. 2365–2369 (2013)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Cai, C., Ke, D., Xu, Y., Su, K. (2014). Fast Learning of Deep Neural Networks via Singular Value Decomposition. In: Pham, DN., Park, SB. (eds) PRICAI 2014: Trends in Artificial Intelligence. PRICAI 2014. Lecture Notes in Computer Science(), vol 8862. Springer, Cham. https://doi.org/10.1007/978-3-319-13560-1_65
Download citation
DOI: https://doi.org/10.1007/978-3-319-13560-1_65
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-13559-5
Online ISBN: 978-3-319-13560-1
eBook Packages: Computer ScienceComputer Science (R0)