Abstract
The twin issues of loss quality (accuracy) and training time are critical in choosing a stochastic optimizer for training deep neural networks. Optimization methods for machine learning include gradient descent, simulated annealing, genetic algorithm and second order techniques like Newton’s method. However, the popular method for optimizing neural networks is gradient descent. Overtime, researchers have made gradient descent more responsive to the requirements of improved quality loss (accuracy) and reduced training time by progressing from using simple learning rate to using adaptive moment estimation technique for parameter tuning. In this work, we investigate the performances of established stochastic gradient descent algorithms like Adam, RMSProp, Adagrad, and Adadelta in terms of training time and loss quality. We show practically, using series of stochastic experiments, that adaptive moment estimation has improved the gradient descent optimization method. Based on the empirical outcomes, we recommend further improvement of the method by using higher moments of gradient for parameter tuning (weight update). The output of our experiments also indicate that neural network is a stochastic algorithm.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Brownlee, J.: How to choose loss functions when training deep learning neural networks. In: Deep Learning Performance (2019)
Shridhar, K.: A beginners guide to deep learning (2017)
Zhou, Z., Feng, J.: Deep forest: towards an alternative to deep neural networks. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 3553–3559. AAAI Press (2017)
Garnelo, M., Schwarz, J., Rosenbaum, D., Rezende, V.F., Eslami, S.M., Teh, Y.W.: Neural processes, arXiv preprint arXiv:1807.01622 (2018)
Damianou, A., Lawrence, N.: Deep Gaussian processes. In: Artificial Intelligence and Statistics, pp. 207–215 (2013)
Pandey, P.: Demystifying neural networks: a mathematical approach (Part 2) (2018)
Zeiler, M.D.: Adadelta: an adaptive learning rate method, arXiv preprint arXiv:1212.5701 (2012)
Tieleman, T., Hinton, G.: Lecture 6.5-rmsprop: divide the gradient by a running average of its recent magnitude. COURSERA Neural Netw. Mach. Learn. 4(2), 26–31 (2012)
Dauphin, Y.N., Pascanu, R., Caglar, G., Kyunghyun, C., Ganguli, S., Bengio, Y.: Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In: Advances in Neural Information Processing Systems, pp. 2933–2941 (2014)
Kawaguchi, K.: Deep learning without poor local minima. In: Advances in Neural Information Processing Systems (NIPS) (2016)
Kim, D., Fessler, J.A.: Optimized first-order methods for smooth convex minimization. Math. Prog. 151, 8–107 (2016)
Aji, A.F., Heafield, K.: Combining global sparse gradients with local gradients. In: ICLR Conference (2019)
Walia, A.S.: Types of optimization algorithms used in neural networks and ways to optimize gradient descent (2017)
Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2015)
Koushik, J., Hayashi, H.: Improving stochastic gradient descent with feedback. In: Conference Paper at ICLR (2017)
Polyak, B.T., Juditsky, A.B.: Acceleration of stochastic approximation by averaging (PDF). SIAM J. Control Optim. 30(4), 838–855 (1992)
Zhang, S., Choromanska, A., LeCun, Y.: Deep learning with elastic averaging SGD. In: Neural Information Processing Systems Conference (NIPS) (2015)
Davies, C., Dembinska, A.: Computing moments of discrete order statistics from non-identical distributions. J. Comput. Appl. Math. 328(15), 340–354 (2018)
Qian, N.: On the momentum term in gradient descent learning algorithms. Neural Netw. Official J. Int. Neural Netw. Soc. 12(1), 145–151 (1999)
Lockett, A.: What is the most popular learning rate decay formula in machine learning? The University of Texas at Austin (2012)
Darken, C., Chang, J., Moody, J.: Learning rate schedules for faster stochastic gradient search. In: Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop (1992)
Mei, S.: A mean field view of the landscape of two-layer neural networks. In: Proceedings of the National Academy of Sciences (2018)
Okewu, E., Adewole, P., Sennaike, O.: Experimental comparison of stochastic optimizers in deep learning. In: Misra, S., et al. (eds.) ICCSA 2019. LNCS, vol. 11623, pp. 704–715. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24308-1_55
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Okewu, E., Misra, S., Lius, FS. (2020). Parameter Tuning Using Adaptive Moment Estimation in Deep Learning Neural Networks. In: Gervasi, O., et al. Computational Science and Its Applications – ICCSA 2020. ICCSA 2020. Lecture Notes in Computer Science(), vol 12254. Springer, Cham. https://doi.org/10.1007/978-3-030-58817-5_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-58817-5_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58816-8
Online ISBN: 978-3-030-58817-5
eBook Packages: Computer ScienceComputer Science (R0)