Abstract
We present an algorithm inspired by diffusion networks for learning the input/output mapping of temporal sequences with recurrent neural networks. Noise is added to the activation dynamics of the neurons of the hidden layer and annealed during learning of an output path probability distribution. Noise therefore plays the role of a learning parameter. We compare some results obtained on 2 temporal tasks with this “dynamic noise annealing” algorithm with other learning algorithms. Finally we discuss why adding noise to the state space variables can be better than adding stochasticity in the weight space.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Kirkpatrick S., Gelatt C.D., Vecchi M.P. Optimization by Simulated Annealing. Science 220 (1983) 671–680
Bengio Y., Simard P., Frasconi P. Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE T. Neural Networ. 5-2 (1994) 157–166
Hochreiter S., Schmidhuber J. Long Short-Term Memory. Neural Comput. 9-8 (1997) 1735–1780
Movellan J. R.,and J.L. McClelland J. L. Learning continuous probability distributions with symmetric diffusion networks. Cognitive Sci. 17 (1992) 463–496
Movellan J. R., Mineiro P., Williams R. J. Modeling Path Distributions Using Partially Observable Diffusion Networks. TechReport, CogSci, UCSD (1999)
Elman J.L. Finding Structure in Time. Cognitive Sci. 14 (1990) 179–211
Oksendal B. Stochastic differential equations. Springer-Verlag (1992)
Weigend A. S., Gershenfeld, N. A. Times Series Prediction: Forecasting the future and understanding the past. Addison-Wesley (1994)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Sottas, PE., Gerstner, W. (2002). Dynamic Noise Annealing for Learning Temporal Sequences with Recurrent Neural Networks. In: Dorronsoro, J.R. (eds) Artificial Neural Networks — ICANN 2002. ICANN 2002. Lecture Notes in Computer Science, vol 2415. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-46084-5_185
Download citation
DOI: https://doi.org/10.1007/3-540-46084-5_185
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44074-1
Online ISBN: 978-3-540-46084-8
eBook Packages: Springer Book Archive