Abstract
Backpropagation (BP) is the most widely used algorithm for the training of deep neural networks (DNN) and is also considered a de facto standard algorithm. However, the BP algorithm often requires a lot of computation time, which remains a major challenge. Thus, to reduce the time complexity of the BP algorithm, several methods have been proposed so far, but few do not apply to the BP algorithm. In the meantime, a new DNN algorithm based on nonnegative matrix factorization (NMF) has been proposed, and the algorithm has different convergence characteristics from the BP algorithm. We found that the NMF-based method could lead to rapid performance improvement in DNNs training, and we developed a technique to accelerate the training time of the BP algorithm. In this paper, we propose a novel training method for accelerating the BP algorithm by using an NMF-based algorithm. Furthermore, we present a technique to boost the efficiency of our proposed method by concurrently training DNNs with the BP and NMF-based algorithms. The experimental results indicate that our method significantly improves the training time of the BP algorithm.
The present study is supported in part by JST/ACT-I (No. JPMJPR16U6), NEDO and JSPS/Grants-in-Aid for Scientific Research (Nos. 17K12690, 18H03250, 19KK0255).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abadi, M., et al.: Tensorflow: a system for large-scale machine learning. In: 12th \(\{\)USENIX\(\}\) Symposium on Operating Systems Design and Implementation (\(\{\)OSDI\(\}\) 2016), pp. 265–283 (2016)
Abdul Hamid, N., Mohd Nawi, N., Ghazali, R., Mohd Salleh, M.N.: Accelerating learning performance of back propagation algorithm by using adaptive gain together with adaptive momentum and adaptive learning rate on classification problems. In: Kim, T., Adeli, H., Robles, R.J., Balitanas, M. (eds.) UCMA 2011. CCIS, vol. 151, pp. 559–570. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20998-7_62
Arai, R., Imakura, A., Sakurai, T.: An improvement of the nonlinear semi-NMF based method by considering bias vectors and regularization for deep neural networks. Int. J. Mach. Learn. Comput. 8, 191–197 (2018). https://doi.org/10.18178/ijmlc.2018.8.3.686
Bengio, Y., Lamblin, P., Popovici, P., Larochelle, H.: Greedy layer-wise training of deep networks. In: Advances in Neural Information Processing Systems, vol. 19. MIT Press, Cambridge (2007)
Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml
Huber, P., Wiley, J., InterScience, W.: Robust Statistics. Wiley, New York (1981)
Imakura, A., Inoue, Y., Sakurai, T., Futamura, Y.: Parallel implementation of the nonlinear semi-NMF based alternating optimization method for deep neural networks. Neural Process. Lett. 47, 1–13 (2017). https://doi.org/10.1007/s11063-017-9642-2
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2014). http://arxiv.org/abs/1412.6980, cite arxiv:1412.6980 Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego (2015)
Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Fürnkranz, J., Joachims, T. (eds.) Proceedings of the 27th International Conference on Machine Learning (ICML 2010), pp. 807–814 (2010)
Otair, M., Walid, A.S.: Speeding up back-propagation neural networks. In: Proceedings of the 2005 Informing Science and IT Education Joint Conference, vol. 1 (0002)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning Representations by Back-propagating Errors. Nature 323(6088), 533–536 (1986). https://doi.org/10.1038/323533a0. http://www.nature.com/articles/323533a0
Sakurai, T., Imakura, A., Inoue, Y., Futamura, Y.: Alternating optimization method based on nonnegative matrix factorizations for deep neural networks. In: Hirose, A., Ozawa, S., Doya, K., Ikeda, K., Lee, M., Liu, D. (eds.) ICONIP 2016. LNCS, vol. 9950, pp. 354–362. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46681-1_43
van der Walt, S., Colbert, S.C., Varoquaux, G.: The numpy array: a structure for efficient numerical computation. Comput. Sci. Eng. 13(2), 22–30 (2011). http://dblp.uni-trier.de/db/journals/cse/cse13.html#WaltCV11
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Baek, S., Imakura, A., Sakurai, T., Kataoka, I. (2021). Accelerating the Backpropagation Algorithm by Using NMF-Based Method on Deep Neural Networks. In: Uehara, H., Yamaguchi, T., Bai, Q. (eds) Knowledge Management and Acquisition for Intelligent Systems. PKAW 2021. Lecture Notes in Computer Science(), vol 12280. Springer, Cham. https://doi.org/10.1007/978-3-030-69886-7_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-69886-7_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-69885-0
Online ISBN: 978-3-030-69886-7
eBook Packages: Computer ScienceComputer Science (R0)