Hybrid Learning Algorithms for Feed-Forward Neural Networks

Abstract

Since the introduction of the backpropagation algorithm as a learning rule for neural networks much effort has been spent trying to develop faster alternatives. Normally, the proposed variations use a fixed strategy e.g. adaptively changing learning rates, or the use of second order information of the error surface. If the chosen heuristic does not fit the actual shape of the error surface, the computed weight changes will be far from the optimal ones.

In this paper, we propose a hybrid learning algorithm, which basically uses adaptive step sizes for the weight changes, but adaptively includes second order information of the error surface if a valley of the error function is reached. The algorithm is a combination of RPROP and one dimensional secant steps of the kind used extensively by Quickprop: hence its name, QRPROP.