Abstract
Gradient descent method is the preferred method to optimize neural networks and many other machine learning algorithms. Especially with the wide use of deep learning in recent years, gradient descent algorithm has become more and more important. In gradient descent algorithm, learning rate is a very important parameter. The setting of learning rate directly affects the performance of the final model. The existing learning rate optimization algorithms adjusts learning rate based on the idea of step-by-step reduction. Different from this idea, this paper based on human walking law proposes a new optimization algorithm, the consolidate step-by-step algorithm (CSBS), which determines the learning rate according to the gradient of each iteration. In this paper, MNIST data set is used to verify the performance of the algorithm. The experimental results show that the CSBS algorithm accelerates the convergence speed of the model and reduces the sensitivity to the initial parameters.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ruder, S.: An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 (2016)
Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4(5), 1–17 (1964)
Nesterov, Y.: A method of solving a convex programming problem with convergence rate \(O(1/k^2)\)[C]. In: Soviet Mathematics Doklady (1983)
Sutskever, I., Martens, J., Dahl, G., et al.: On the importance of initialization and momentum in deep learning. In: International Conference on Machine Learning, pp. 1139–1147. PMLR (2013)
Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12(7), 2121–2159 (2011)
Kingma, D., Ba, J.: Adam: a method for stochastic optimization. Comput. Sci. (2014)
Smith, L.N.: Cyclical learning rates for training neural networks. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464–472. IEEE (2017)
Babichev, D., Bach, F.: Constant step size stochastic gradient descent for probabilistic modeling. arXiv preprint arXiv:1804.05567 (2018)
Kiran, R., Kumar, P., Bhasker, B.: DNNRec: a novel deep learning based hybrid recommender system. Expert Syst. Appl. 144, 113054 (2020)
Otter, D.W., Medina, J.R., Kalita, J.K.: A survey of the usages of deep learning for natural language processing. IEEE Trans. Neural Netw. Learn. Syst. 32(2), 604–624 (2020)
Özyurt, F.: Efficient deep feature selection for remote sensing image recognition with fused deep learning architectures. J. Supercomput. 76(11), 8413–8431 (2019). https://doi.org/10.1007/s11227-019-03106-y
Konar, J., Khandelwal, P., Tripathi, R.: Comparison of various learning rate scheduling techniques on convolutional neural network. In: 2020 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS), pp. 1–5. IEEE (2020)
Acknowledgments
This work was partially supported by National Natural Science Foundation of China (Grant No. 61972179), Guangdong Basic and Applied Basic Research Foundation (Grant No. 2020A1515011476).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
You, S., Gao, W., Li, Z., Yang, Q., Tian, M., Zhu, S. (2022). Dynamic Adjustment of the Learning Rate Using Gradient. In: Zu, Q., Tang, Y., Mladenovic, V., Naseer, A., Wan, J. (eds) Human Centered Computing. HCC 2021. Lecture Notes in Computer Science, vol 13795. Springer, Cham. https://doi.org/10.1007/978-3-031-23741-6_6
Download citation
DOI: https://doi.org/10.1007/978-3-031-23741-6_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23740-9
Online ISBN: 978-3-031-23741-6
eBook Packages: Computer ScienceComputer Science (R0)