Abstract
This paper presents modified, faster momentum minimization algorithms based on existing ones. The modified algorithms monotonically decrease the objective function and do not allow it to oscillate. The modification scheme aims to enhance momentum minimizers by incorporating contemporary line search procedures and restarts, akin to the state-of-the-art unconstrained minimizers. We also investigate the unique resource of oscillation-free momentum minimizers for their further acceleration. In particular, the wider range of variation in the friction-related coefficient within the model significantly impacts the performance time.
Our previously developed techniques can be used to prove the convergence of modified algorithms. In this paper, we focus on the technical and experimental aspects of these algorithms. To determine the efficiency of the new algorithms, numerical experiments were conducted on standard optimization test functions and on single-layer neural networks for several datasets.
Comparisons were made with the best unconstrained minimization algorithms – lcg, L-BFGS and ADAM. Oscillation-free momentum algorithms are significantly easier to design and implement than lcg and L-BFGS, while still being competitive in terms of performance. Collections of minimizers and test functions have been uploaded to GitHub.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Liu, D.C., Nocedal, J.: On the limited memory bfgs method for large scale optimization. Math. Program. (Ser. B) 45(3), 503–528 (1989)
Hager, W.W., Zhang, H.: The limited memory conjugate gradient method. SIAM J. Optim. 23(4), 2150–2168 (2013)
Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: ICML’13: Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28 June 2013, pp. III-1139-III-1147
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv:1609.04747 [cs.LG], https://doi.org/10.48550/arXiv.1609.04747
Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations, pp. 1–13 (2015)
Reddi, S.J., Kale, S., Kumar, S.: On the Convergence of Adam and Beyond. arXiv:1904.09237 [cs.LG.] https://doi.org/10.48550/arXiv.1904.09237
Gelashvili, K., Khutsishvili, I., Gorgadze, L., Alkhazishvili, L.: Speeding up the Convergence of the Polyak’s Heavy Ball Algorithm. Trans. A. Razmadze Math. Inst. 172(2), 176–188 (2018)
Gelashvili, K.: Add-in for Solvers of unconstrained minimization to eliminate lower bounds of variables by transformation. Trans. A. Razmadze Math. Inst. 173, 39–46 (2019)
Source code for CG_DESCENT Version 6.8 (C code), March 7, 2015. Software \(|\) William Hager (ufl.edu)
Ernesto P. Adorio, U.P.: MVF-Multivariete Test Functions Library in C for Unconstrained Global Optimization. http://www.geocities.ws/eadorio/mvf.pdf
Alkhazishvili, L., Gorgadze, L.: A Collection of test functions for unconstrained optimization solvers with serial and parallel C++ implementations. http://eprints.tsu.ge/1206/1/A%20collection_%20%20%20%20L.Alkhazishvili_L.Gorgadze.pdf
LeCun, Y., Cortes, C., Burges, C.J.C.: THE MNIST DATABASE of handwritten digits. http://yann.lecun.com/exdb/mnist/
Lin, C.-J., Moré, J.J.: Incomplete Cholesky factorizations with limited memory. SIAM J. Sci. Comput. 21(1), 24–45 (1999)
Nesterov, Y.: A method for unconstrained convex minimization problem with the rate of convergence o(1/k2). Doklady ANSSSR (translated as Soviet. Math. Docl. ), 269, 543–547
Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. Ser. A 91(2), 201–213 (2002)
cpp-btree. https://code.google.com/archive/p/cpp-btree/
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gelashvili, K., Gogishvili, P. (2024). Speeding up the Oscillation-Free Modified Heavy Ball Algorithm. In: Pereira, A.I., Mendes, A., Fernandes, F.P., Pacheco, M.F., Coelho, J.P., Lima, J. (eds) Optimization, Learning Algorithms and Applications. OL2A 2023. Communications in Computer and Information Science, vol 1981. Springer, Cham. https://doi.org/10.1007/978-3-031-53025-8_35
Download citation
DOI: https://doi.org/10.1007/978-3-031-53025-8_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-53024-1
Online ISBN: 978-3-031-53025-8
eBook Packages: Computer ScienceComputer Science (R0)