Abstract
A search method based on the backpropagation rule commonly used for training neural networks is proposed here for the optimisation of smooth nonlinear functions. The use of the Resilient backPROPagation (RPROP) heuristic rule for local minimisation is described. The details of employing the directional step length determined by RPROP along with a simple restarting scheme are provided. In the approach proposed here direct use of the directional step determined by the heuristic without using any line search conditions takes place. The overall algorithm has been tested on a number of benchmark functions found in the literature with very positive results. The test problems’ dimension ranges from 100 to 50,000. The results obtained show that the suggested search direction method results to a highly efficient algorithm suitable for large scale optimisation.
Similar content being viewed by others
References
Anastasiadis, A., Magoulas, G., Vrahatis, M.: New globally convergent training scheme based on the resilient propagation algorithm. Neurocomputing 64, 253–270 (2005)
Andrei, N.: An unconstrained optimization test functions collection. Adv. Model. Optim. 10(1), 147–161 (2008)
Fletcher, R.: Pract. Methods Optim., 2nd edn. Wiley-Interscience, New York, NY, USA (1987)
Hecht-Nielsen, R.: Theory of the backpropagation neural network. In: International Joint Conference on Neural Networks, vol. 1, pp. 593–605. Washington, DC, USA (1989)
Igel, C., Hüsken, M.: Empirical evaluation of the improved RPROP learning algorithms. Neurocomputing 50, 105–123 (2003)
Kotsialos, A.: Non-Smooth Optimization Based on Resilient Backpropagation Search for Unconstrained and Simply Bounded Problems. Optimization Methods and Software (available on line) (2012)
Kotsialos, A., Papageorgiou, M.: Nonlinear optimal control applied to coordinated ramp metering. IEEE Trans. Control Syst. Technol. 10(6), 920–933 (2004)
Lukšan, L., Vlček, J.: Test Problems for Unconsgtrained Optimization. Institute of Computing Science, Academy of Sciences of the Czech Republic, Technical Report (2003)
Mastorocostas, P., Hilas, C.: A computational intelligence forecasting system for telecommunications time series. Eng. Appl. Artif. Intell 25(1), 200–206 (2012)
Nelder, J., Mead, R.: A simplex method for function minimization. Comput. J. 7, 308–313 (1965)
Ng, S., Leung, S., Luk, A., Wu, Y.: Convergence analysis of generalized back-propagation algorithm with modified gradient function. In: Proceedings of IEEE International Joint Conference on Neural Networks, pp. 3672–3678. Vancouver, BC, Canada (2006)
Nocedal, J., Wright, S.: Numer. Optim. Springer, New York (1999)
Prechelt, L.: PROBEN1—a set of neural network benchmark problems and benchmarking rules. Faculty of Informatics, University of Karlsruhe, Technical Report (1994)
Riedmiller, M., Braun, H.: A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In: IEEE International Conference On, Neural Networks, pp. 586–591 (1993)
Zhan, Z.H., Zhang, J., Li, Y., Chung, H.H.: Adaptive particle swarm optimization. IEEE Trans. Syst. Man Cybern. Part B Cybern. 39(6), 1362–1381 (2009)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kotsialos, A. Nonlinear optimisation using directional step lengths based on RPROP. Optim Lett 8, 1401–1415 (2014). https://doi.org/10.1007/s11590-013-0668-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11590-013-0668-8