Abstract
We consider regularization methods to improve the recently introduced backpropagation-decorrelation (BPDC) online algorithm for O(N) training of fully recurrent networks. While BPDC combines one-step error backpropagation and the usage of temporal memory of a network dynamics by means of decorrelation of activations, it is an online algorithm using only instantaneous states and errors. As enhancement we propose several ways to introduce memory in the algorithm for regularization. Simulation results of standard tasks show that different such strategies cause different effects either improving training performance at the cost of overfitting or degrading training errors.
An erratum to this chapter can be found at http://dx.doi.org/10.1007/11550907_163 .
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Kolen, J.F., Kremer, S.C. (eds.): A Field Guide to Dynamical Recurrent Networks. Wiley, Chichester (2001)
Hammer, B., Steil, J.J.: Tutorial: Perspectives on learning with recurrent neural networks. In: Proc. ESANN, pp. 357–369 (2002)
Steil, J.J.: Backpropagation-decorrelation: online recurrent learning with O(N) complexity. In: Proc. IJCNN, pp. 843–848 (2004)
Atiya, A.B., Parlos, A.G.: New Results on Recurrent Network Training: Unifying the Algorithms and Accelerating Convergence. IEEE Trans. Neural Networks 11, 697–709 (2000)
Schiller, U.D., Steil, J.J.: Analyzing the weight dynamics of recurrent learning algorithms. Neurocomputing 63C, 5–23 (2005)
Jaeger, H.: Adaptive nonlinear system identification with echo state networks. In: NIPS, vol. 15, pp. 593–600 (2003)
Natschläger, T., Maass, W., Markram, H.: The “liquid computer”: A novel strategy for real-time computing on time series. TELEMATIK 8, 39–43 (2002)
Steil, J.J.: Stability of backpropagation-decorrelation efficient O(N) recurrent learning. In: Proc. ESANN, d-facto publications (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Steil, J.J. (2005). Memory in Backpropagation-Decorrelation O(N) Efficient Online Recurrent Learning. In: Duch, W., Kacprzyk, J., Oja, E., Zadrożny, S. (eds) Artificial Neural Networks: Formal Models and Their Applications – ICANN 2005. ICANN 2005. Lecture Notes in Computer Science, vol 3697. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11550907_103
Download citation
DOI: https://doi.org/10.1007/11550907_103
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-28755-1
Online ISBN: 978-3-540-28756-8
eBook Packages: Computer ScienceComputer Science (R0)