Abstract
In recurrent neural network (RNN) learning of finite state automata (FSA), we discuss how a neuro gain (β) influences the stability of the state representation and the performance of the learning. We formally show that the existence of the critical neuro gain (β 0): any β larger than β 0 makes an RNN maintain the stable representation of states of an acquired FSA. Considering the existence of β 0 and avoidance of local minima, we propose a new RNN learning method with the scheduling of β, called an annealed RNN learning. Our experiments show that the annealed RNN learning went beyond than a constant β learning.
Preview
Unable to display preview. Download preview PDF.
References
J. Hertz, A. Krogh and R.G. Palmer, Introduction to the Theory of Neural Computation, Addison-Wesley (1991)
Leong K.L., Fixed Point Analysis for Discrete-Time Recurrent Neural Networks, Proc. IJCNN IV (1992) 134–139
P. Tiňo, B.G. Horne, C.L. Giles and P.C. Collingwood, Finite State Machines and Recurrent Neural Networks — Automata and Dynamical System Approaches, UMIACS-TR-95-1, CS-TR-3396 (1995)
M.W. Goudreau, C.L. Giles, S.T. Chakradhar and D. Chen, First-Order Versus Second-Order Single-Layer Recurrent Neural Networks, IEEE Neural Networks, Vol. 5 No. 3 (1994) 511–111
P. Manolios and R. Fanelli, First-Order Recurrent Neural Networks and Deterministic Finite State Automata, Neural Computation, Vol. 6 (1994) 1155–1173
S. Das, C.L. Giles and G.Z. Sun, Using Prior Knowledge in a NNPDA to Learn Context-Free Languages, Advances in Neural Information Processing Systems, Vol. 5 (1993) 65–71.
D.E. Rumelhart, G.E. Hinton and R.J. Williams, Learning Internal Representations by Error Propagation, in Parallel Distributed Processing, D.E. Rumelhart and J.L. McClelland (1986) Vol. 1 318–362
N. Ueda and R. Nakano, Deterministic Annealing Variant of the EM Algorithm, Advances in Neural Information Processing Systems, Vol. 7 (1995) 545–552.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1996 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Arai, Ki., Nakano, R. (1996). Annealed RNN learning of finite state automata. In: von der Malsburg, C., von Seelen, W., Vorbrüggen, J.C., Sendhoff, B. (eds) Artificial Neural Networks — ICANN 96. ICANN 1996. Lecture Notes in Computer Science, vol 1112. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-61510-5_89
Download citation
DOI: https://doi.org/10.1007/3-540-61510-5_89
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-61510-1
Online ISBN: 978-3-540-68684-2
eBook Packages: Springer Book Archive