Abstract
Traditional Recurrent Neural Networks (RNNs) perform poorly on learning tasks involving long time-lag dependencies. More recent approaches such as LSTM and its variants significantly improve on RNNs ability to learn this type of problem. We present an alternative approach to encoding temporal dependencies that associates temporal features with nodes rather than state values, where the nodes explicitly encode dependencies over variable time delays. We show promising results comparing the network’s performance to LSTM variants on an extended Reber grammar task.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Lang, K.J., Waibel, A.H., Hinton, G.E.: A time-delay neural network architecture for isolated word recognition. Neural Networks 3(1), 23–43 (1990)
Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5(2), 157–166 (1994)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9(8), 1735–1780 (1997)
Fahlman, S.E.: The recurrent cascade-correlation architecture. In: Advances in Neural Information Processing Systems, vol. 3, pp. 190–196 (1991)
Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to Forget: Continual Prediction with LSTM (2000)
Williams, R.J., Zipser, D.: Gradient-based learning algorithms for recurrent networks and their computational complexity. Backpropagation: Theory, architectures, and applications, 433–486 (1995)
Elman, J.L.: Finding structure in time. Cognitive science 14(2), 179–211 (1990)
Mozer, M.C.: Induction of multiscale temporal structure. In: Advances in Neural Information Processing Systems, vol. 4, pp. 275–282 (1992)
Ring, M.B.: Learning Sequential Tasks by Incrementally Adding Higher Orders. In: Advances in Neural Information Processing Systems, pp. 115–122 (1992)
Puskorius, G.V., Feldkamp, L.A.: Neurocontrol of nonlinear dynamical systems with Kalman filtertrained recurrent networks. IEEE Transactions on Neural Networks 5(2), 279–297 (1994)
Watrous, R.L., Kuhn, G.M.: Induction of finite-state languages using second-order recurrent networks. Neural Computation 4(3), 406–414 (1992)
Schmidhuber, J., Hochreiter, S.: Guessing can outperform many long time lag algorithms (1996)
Schmidhuber, J.: Netzwerkarchitekturen, Zielfunktionen und Kettenregel. Habilitation, Technische Universitat Munchen 1(1), 1 (1993)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Johnson, K., MacNish, C. (2009). A Novel Connectionist Network for Solving Long Time-Lag Prediction Tasks. In: Nicholson, A., Li, X. (eds) AI 2009: Advances in Artificial Intelligence. AI 2009. Lecture Notes in Computer Science(), vol 5866. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-10439-8_56
Download citation
DOI: https://doi.org/10.1007/978-3-642-10439-8_56
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-10438-1
Online ISBN: 978-3-642-10439-8
eBook Packages: Computer ScienceComputer Science (R0)