Skip to main content
SpringerLink
Log in

Recurrent neural network architectures: An overview

  • Chapter
  • First Online:
Book cover Adaptive Processing of Sequences and Data Structures (NN 1997)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 1387))

Included in the following conference series:

Abstract

In this paper, we have first considered a number of popular recurrent neural network architectures. Then, two subclasses of general recurrent neural network architectures are introduced. It is shown that all these popular recurrent neural network architectures can be grouped under either of these two subclasses of general recurrent neural network architectures. It is also inferred that these two subclasses of recurrent neural network architectures are distinct, in that it is not possible to transform from one form to the other. Two recently introduced recurrent neural network architectures specifically designed for special purposes, viz., for overcoming long term temporal dependency, and for data structure classifications are also considered.

Once the architectural aspects of the class of networks are settled, then one could consider the training aspects. This will be considered in a companion paper [31].

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Albertini, F., Sontag, E. “For neural networks, function determines form”. Neural Networks. Vol 6, pp 975–990, 1993.

    Article  Google Scholar 

  2. Back, A.D., Tsoi, A.C. “FIR and IIR synapses, a new neural network architecture for time series modelling”. Neural Computation. Vol. 3, No. 3, pp 375–385, 1991.

    Google Scholar 

  3. Baldi, P., Chauvin, Y. “Hybrid modelling, HMM/NN architectures, and protein modelling”. Neural Computation. Vol 8, No. 7, pp 1541–1565, 1996.

    Google Scholar 

  4. Bengio, Y., Simard, P., Frasconi, P. “Learning Long term dependencies with gradient descent is difficult”. IEEE Trans Neural Networks. Vol. 5, pp 157–166, 1994.

    Article  Google Scholar 

  5. Box, G. E. P., Jenkins, G. Time Series Analysis. Holden Day, 1967.

    Google Scholar 

  6. Calder, B., Grunwald, D., Jones, M., Lindsay, D., Martin, J., Mozer, M., Zorn, B. “Evidence-based static branch prediction using machine learning”. ACM Transaction on on Programming Languages and Systems, Vol. 19, pp 188–222, 1997.

    Article  Google Scholar 

  7. Chen, S., Billings, S., Grant, P. “Nonlinear system identification using neural networks”. International Journal of Control. Vol. 51, No. 6, pp. 1191–1214, 1990.

    MathSciNet  MATH  Google Scholar 

  8. Elman, J. “Finding structure in time”. Cognitive Science. Vol. 14, pp 179–211, 1990.

    Article  Google Scholar 

  9. Frasconi, P., Gori, M., Soda, G. “Local feedback multilayered networks”. Neural Computation. Vol. 4, pp 120–130, 1992.

    Google Scholar 

  10. Haykin, S. Neural Networks, A comprehensive foundation. MacMillan College Pub Co. 1994.

    Google Scholar 

  11. Hornik, K. “Approximation capabilities of multilayer feedforward neural networks”. Neural Networks. Vol. 4, pp 251–257, 1990.

    Article  Google Scholar 

  12. Hochreiter, S., Schmidhuber, J. “Long short-term memory”. Neural Computation. Vol 9, pp 1735–1780, 1997.

    Article  Google Scholar 

  13. Jordan, M. “Supervised learning and systems with excess degree of freedom”. Massachusetts Institute of Technology, COINS Technical Report 88-27, May, 1988.

    Google Scholar 

  14. Kailath, T. Linear Systems. Prentice Hall, Englewood Cliffs, N.J., 1980.

    MATH  Google Scholar 

  15. Lawrence, S., Giles, L., Back, A., Tsoi, A. C. “The gamma MLP — multiple temporal resolutions, the curse of dimensionality, and gradient descent learning”. Neural Computation To appear.

    Google Scholar 

  16. Lapedes, A., Farber, R. “Nonlinear signal processing using neural networks prediction and system modelling”. Los Alamos Natioanl Laboratory, Los Alamos, LA-UR-262, 1987.

    Google Scholar 

  17. Lin, T., Horne, B.G., Giles, L. “How embedding memory in recurrent neural network architecture helps learning long term temporal dependencies”. Technical Report, University of Maryland. Report Number UMIACS-TR-96-76, and CS-TR-3706, Institute for Advanced Computer Studies, University of Maryland, College Park, Maryland, 1996.

    Google Scholar 

  18. Marple, S.L. Digital spectral analysis and applications. Englewood, N.J.: Prentice Hall, 1987.

    Google Scholar 

  19. Narendra, K.P., Parthasarathy, K. “Identification and Control of Dynamical Systems using Neural Networks”. IEEE Trans Neural Networks. Vol 1, pp 4–27, 1990.

    Article  Google Scholar 

  20. Nerrand, O., Roussel-Ragot, P., Personnaz, L., Dreyfus, G., Marcos, S. “Neural Networks and nonlinear adaptive filtering: Unifying concepts and new algorithms”. Neural Computation. Vol 5, pp 165–197, 1993.

    Google Scholar 

  21. Pindea, F. “Dynamics and architecture for neural computation in recurrent neural networks”. Journal of Complexity. Vol 4., pp 216–245, 1988.

    Article  MathSciNet  Google Scholar 

  22. Principe, J., de Vries, B., Oliveira, P. “The gamma filter — a new class of adaptive IIR filters with restricted feedback”. IEEE Trans Signal Processing. Vol. 41, pp 649–656, 1993.

    Article  MATH  Google Scholar 

  23. Robinson, A., J. Dynamic error propagation networks. PhD thesis, University of Cambridge, Cambridge, U.K., 1989.

    Google Scholar 

  24. Scarselli, F. Tsoi, A.C. “Universal approximation using feedforward neural networks: A survey of some existing methods, and some results”. Neural Networks. To appear.

    Google Scholar 

  25. Siegelmann, H., Home, B., Giles, L. “Computational capabilites of recurrent NARX neural networks”. IEEE Trans System, Man and Cybernetics. Part B, Vol 27, pp 208–218, 1997.

    Article  Google Scholar 

  26. Sontag, E. “Neural networks for control”. In Essay on Control: Perspectives in the Theory and its applications. H. L. Trentelman, J. C. Willems, Ed. Boston: Birkhauser, pp. 339–380, 1993.

    Google Scholar 

  27. Sperduti, A. “Labelling RAAM”. Connection Science. Vol. 6, No. 4, pp 429–459, 1994.

    Google Scholar 

  28. Sperduti, A., Starita, A. “Supervised neural networks for the classification of structures”. IEEE Trans Neural Networks. Vol 8, pp 714–735, 1997.

    Article  Google Scholar 

  29. Tsoi, A.C., Back, A.D. “Locally recurrent globally feedforward networks: a critical review of architectures”. IEEE Trans on Neural Networks. Vol. 5, No. 2, pp 229–239, 1994.

    Article  Google Scholar 

  30. A C Tsoi, “Application of neural network methodology to the modelling of the yield strength in a steel rolling plate mill”, Advances in Neural Information Processing Systems, Vol 4. Ed. Moody, J, Hansen, S, Lippmann, R, Morgan Kaufmann Publishers, 1992.

    Google Scholar 

  31. Tsoi, A.C. “Gradient based learning methods”. This volume.

    Google Scholar 

  32. Tsoi, A.D., Back, A.D. “Discrete time recurrent neural network architectures: a unifying review”. Neurocomputing. Vol. 15, pp 183–224, 1997.

    Article  MATH  Google Scholar 

  33. Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., Lang, L. “Phonemic recognition using time delay neural networks” IEEE Trans Acoustic Speech and Signal Processing. Vol. 37, No. 3, pp 328–339, 1989.

    Article  Google Scholar 

  34. Wan, E. “Temporal backpropagation for FIR neural networks”. Proc Int Joint Conf Neural Networks. San Diego, June, 1990, pp 575–580, 1990.

    Google Scholar 

  35. Williams, R., Zipser, D. “A learning algorithm for continually running fully recurrent neural networks”. Neural Computation. Vol. 1, pp 270–280, 1989.

    Google Scholar 

  36. Zomaya, A., Mills, P.M., Tade, M.O. Neuron-adaptive process control, a practical approach. Wiley, 1996.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Informatics, University of Wollongong, Northfields Ave, 2522, Wollongong, NSW, Australia

    Ah Chung Tsoi

Authors
  1. Ah Chung Tsoi
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

C. Lee Giles Marco Gori

Rights and permissions

Reprints and permissions

Copyright information

© 1998 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Tsoi, A.C. (1998). Recurrent neural network architectures: An overview. In: Giles, C.L., Gori, M. (eds) Adaptive Processing of Sequences and Data Structures. NN 1997. Lecture Notes in Computer Science, vol 1387. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0053993

Download citation

  • DOI: https://doi.org/10.1007/BFb0053993

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-64341-8

  • Online ISBN: 978-3-540-69752-7

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation