Abstract
Recurrent neural networks unlike feed-forward networks are able to process inputs with time context. The key role in this process is played by the dynamics of the network, which transforms input data to the recurrent layer states. Several authors have described and analyzed dynamics of small sized recurrent neural networks with two or three hidden units. In our work we introduce techniques that allow to visualize and analyze the dynamics of large recurrent neural networks with dozens units, reveal both stable and unstable points (attractors and saddle points), which are important to understand the principles of successful task processing. As a practical example of this approach, dynamics of the simple recurrent network trained by two different training algorithms on context-free language a n b n was studied.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Cleeremans, A., Servan-Schreiber, D., McClelland, J.L.: Finite state automata and simple recurrent networks. Neural Computation 1(3), 372–381 (1989)
Tiňo, P., Šajda, J.: Laearning and extracting initial mealy automata with a modular neural network model. Neural Computation 7(4), 822–844 (1995)
Wiles, J., Elman, J.: Learning to count without a counter: A case study of dynamics and activation landscapes in recurrent networks. In: Proceedings of the Seventeenth Annual Conference of the Cognitive Science Society, pp. 482–487 (1995)
Christiansen, M.H., Chater, N.: Toward a connectionist model of recursion in human linguistic performance. Cognitive Science 23, 417–437 (1999)
Tiňo, P., Čerňanský, M., Beňušková, L.: Markovian architectural bias of recurrent neural networks. IEEE Transactions on Neural Networks 15(1), 6–15 (2004)
Hammer, B., Tiňo, P.: Recurrent neural networks with small weights implement definite memory machines. Neural Computation 15(8), 1897–1926 (2003)
Tiňo, P., Hammer, B.: Architectural bias in recurrent neural networks: Fractal analysis. Neural Computation 15(8), 1931–1957 (2003)
Tiňo, P., Horne, B.G., Giles, C.L., Collingwood, P.C.: Finite state machines and recurrent neural networks - automata and dynamical systems approaches. Neural Networks and Pattern Recognition, 171–220 (1998)
Kolen, J.F.: The origin of clusters in recurrent neural network state space. In: Proceedings from the Sixteenth Annual Conference of the Cognitive Science Society, pp. 508–513. Lawrence Erlbaum Associates, Hillsdale (1994)
Kolen, J.F.: Recurrent networks: state machines or iterated function systems? In: Mozer, M.C., Smolensky, P., Touretzky, D.S., Elman, J.L., Weigend, A. (eds.) Proceedings of the 1993 Connectionist Models Summer School, pp. 203–210. Erlbaum Associates, Hillsdale (1994)
Čerňanský, M., Makula, M., Beňušková, L.: Organization of the state space of a simple recurrent network before and after training on recursive linguistic structures. Neural Networks 20(2), 236–244 (2007)
Rodriguez, P., Wiles, J., Elman, J.L.: A recurrent neural network that learns to count. Connection Science 11, 5–40 (1999)
Boden, M., Wiles, J.: Context-free and context-sensitive dynamics in recurrent neural networks. Connection Science 12(3), 197–210 (2000)
Werbos, P.J.: Backpropagation through time; what it does and how to do it. Proceedings of the IEEE 78, 1550–1560 (1990)
Feldkamp, L., Prokhorov, D., Eagen, C., Yuan, F.: Enhanced multi-stream kalman filter training for recurrent networks. Nonlinear Modeling: Advanced Black-Box Techniques, 29–53 (1998)
Kuznetsov, Y.A.: Elements of applied bifurcation theory, 2nd edn. Springer, New York (1998)
Rodriguez, P.: Simple recurrent networks learn contex-free and contex-sensitive languages by counting. Neural Computation 13, 2093–2118 (2001)
Boden, M., Wiles, J.: On learning context-free and context-sensitive languages. IEEE Transactions on Neural Networks 13(2), 491–493 (2002)
Černaňský, M., Beňušková, L.: Simple recurrent network trained by rtrl and extended kalman filter algorithms. Neural Network World 13(2), 223–234 (2003)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Makula, M., Beňušková, Ľ. (2008). Analysis and Visualization of the Dynamics of Recurrent Neural Networks for Symbolic Sequences Processing. In: Kůrková, V., Neruda, R., Koutník, J. (eds) Artificial Neural Networks - ICANN 2008. ICANN 2008. Lecture Notes in Computer Science, vol 5164. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-87559-8_60
Download citation
DOI: https://doi.org/10.1007/978-3-540-87559-8_60
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-87558-1
Online ISBN: 978-3-540-87559-8
eBook Packages: Computer ScienceComputer Science (R0)