Abstract
The limitations of the least squares based training algorithm is dominated by stalling problem and evaluation error by transformation matrix to obtain an unacceptable solution. This paper presents a new approach for the recurrent networks training algorithm based upon the Layer-by-Layer Least Squares based algorithm to overcome the aforementioned problems. In accordance with our proposed algorithm, all the weights are evaluated by the least squares method without the evaluation of transformation matrix to speed up the rate of convergence. A probabilistic mechanism, based upon the modified weights updated equations, is introduced to eliminate the stalling problem experienced by the pure least squares type computation. As a result, the merits of the proposed algorithm are capable of providing an ability of escaping from local minima to obtain a good optimal solution and still maintaining the characteristic of fast convergence.
Similar content being viewed by others
References
R.E. Rumelhard, G.E. Hinton and R.J. Williams, “Learning internal representation by error baclpropagation”, Parallel Distributed Processing: Explorations in Microstructure of Cognition, Vol. 1, 1986.
R.J. Williams and D. Ziper, “A learning algorithm for continually running fully recurrent neural networks”, Neural Computation, Vol. 1, No. 2, pp. 270–280, 1989.
R.A. Jacobs, “Increased rates of convergence through learning rate adaptation”, NeuralNetworks, Vol. 1, pp. 295–307, 1988.
C.M. Kuan, “A recurrent Newton algorithm and its convergence properties”, IEEE Trans. on Neural Networks, Vol. 6, No. 3, pp. 779–783, 1995.
R.J. Willams, “Training Recurrent Networks Using the Extended Kalman Filter”, International Joint Conference on Neural Networks, Vol. IV, pp. 241–246, Baltimore, 1992.
G.V. Puskorius and L.A. Feldkamp, “Recurrent networks training with the decoupled extended Kalman filter algorithm”, in Proceeding of the 1992 SPIE Conference on the Science of Artificial Neural Networks, Vol. 1710, pp. 461–473, Orlando, 1992.
G.V. Puskorius and L.A. Feldkamp, “Neurocontrol of Nonlinear Dynamical Systems with Kalman Filter Trained Recurrent Networks”, IEEE Trans. on Neural Networks, Vol. 5, No. 2, pp. 279–297, 1994.
F. Biegler-König and F. Bärmann, “A learning algorithm for multilayered neural networks based on linear least squares problems”, Neural Networks, Vol. 6, pp. 127–131, 1993.
Y.F. Yam and T.W.S. Chow, “Accelerated training algorithm for feedforward neural networks based on least squares method”, Neural Processing Letter, Vol. 2, No. 4, pp. 20–25, 1995.
J.Y.F. Yam and T.W.S. Chow, “Extended Least Squares Based Algorithm for Training Feedforward Networks”, IEEE Trans. on Neural Networks, Vol. 8, no. 3, pp. 806–810, May, 1997.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Cho, Sy., Chow, T.W. A Layer-by-Layer Least Squares based Recurrent Networks Training Algorithm: Stalling and Escape. Neural Processing Letters 7, 15–25 (1998). https://doi.org/10.1023/A:1009672319869
Issue Date:
DOI: https://doi.org/10.1023/A:1009672319869