Skip to main content
Log in

A Layer-by-Layer Least Squares based Recurrent Networks Training Algorithm: Stalling and Escape

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

The limitations of the least squares based training algorithm is dominated by stalling problem and evaluation error by transformation matrix to obtain an unacceptable solution. This paper presents a new approach for the recurrent networks training algorithm based upon the Layer-by-Layer Least Squares based algorithm to overcome the aforementioned problems. In accordance with our proposed algorithm, all the weights are evaluated by the least squares method without the evaluation of transformation matrix to speed up the rate of convergence. A probabilistic mechanism, based upon the modified weights updated equations, is introduced to eliminate the stalling problem experienced by the pure least squares type computation. As a result, the merits of the proposed algorithm are capable of providing an ability of escaping from local minima to obtain a good optimal solution and still maintaining the characteristic of fast convergence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. R.E. Rumelhard, G.E. Hinton and R.J. Williams, “Learning internal representation by error baclpropagation”, Parallel Distributed Processing: Explorations in Microstructure of Cognition, Vol. 1, 1986.

  2. R.J. Williams and D. Ziper, “A learning algorithm for continually running fully recurrent neural networks”, Neural Computation, Vol. 1, No. 2, pp. 270–280, 1989.

    Google Scholar 

  3. R.A. Jacobs, “Increased rates of convergence through learning rate adaptation”, NeuralNetworks, Vol. 1, pp. 295–307, 1988.

    Google Scholar 

  4. C.M. Kuan, “A recurrent Newton algorithm and its convergence properties”, IEEE Trans. on Neural Networks, Vol. 6, No. 3, pp. 779–783, 1995.

    Google Scholar 

  5. R.J. Willams, “Training Recurrent Networks Using the Extended Kalman Filter”, International Joint Conference on Neural Networks, Vol. IV, pp. 241–246, Baltimore, 1992.

    Google Scholar 

  6. G.V. Puskorius and L.A. Feldkamp, “Recurrent networks training with the decoupled extended Kalman filter algorithm”, in Proceeding of the 1992 SPIE Conference on the Science of Artificial Neural Networks, Vol. 1710, pp. 461–473, Orlando, 1992.

    Google Scholar 

  7. G.V. Puskorius and L.A. Feldkamp, “Neurocontrol of Nonlinear Dynamical Systems with Kalman Filter Trained Recurrent Networks”, IEEE Trans. on Neural Networks, Vol. 5, No. 2, pp. 279–297, 1994.

    Google Scholar 

  8. F. Biegler-König and F. Bärmann, “A learning algorithm for multilayered neural networks based on linear least squares problems”, Neural Networks, Vol. 6, pp. 127–131, 1993.

    Google Scholar 

  9. Y.F. Yam and T.W.S. Chow, “Accelerated training algorithm for feedforward neural networks based on least squares method”, Neural Processing Letter, Vol. 2, No. 4, pp. 20–25, 1995.

    Google Scholar 

  10. J.Y.F. Yam and T.W.S. Chow, “Extended Least Squares Based Algorithm for Training Feedforward Networks”, IEEE Trans. on Neural Networks, Vol. 8, no. 3, pp. 806–810, May, 1997.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cho, Sy., Chow, T.W. A Layer-by-Layer Least Squares based Recurrent Networks Training Algorithm: Stalling and Escape. Neural Processing Letters 7, 15–25 (1998). https://doi.org/10.1023/A:1009672319869

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1009672319869

Navigation