Skip to main content
SpringerLink
Log in
Book cover

Handbook on Neural Information Processing pp 29–65Cite as

Recurrent Neural Networks

  • Chapter

Part of the book series: Intelligent Systems Reference Library ((ISRL,volume 49))

Abstract

This chapter presents an introduction to recurrent neural networks for readers familiar with artificial neural networks in general, and multi-layer perceptrons trained with gradient descent algorithms (back-propagation) in particular. A recurrent neural network (RNN) is an artificial neural network with internal loops. These internal loops induce recursive dynamics in the networks and thus introduce delayed activation dependencies across the processing elements (PEs) in the network.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Allen, R.B., Alspector, J.: Learning of stable states in stochastic asymmetric networks. Technical Report TM-ARH-015240, Bell Communications Research, Morristown, NJ (1989)

    Google Scholar 

  2. Atiya, A.F.: Learning on a general network. In: Neural Information Processing Systems, New York, pp. 22–30 (1988)

    Google Scholar 

  3. Back, A.D., Tsoi, A.C.: FIR and IIR synapses, a new neural network architecture for time series modeling. Neural Computation 3, 375–385 (1991)

    Article  Google Scholar 

  4. Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gadient descent is difficult. IEEE Transactions on Neural Networks 5, 157–166 (1994)

    Article  Google Scholar 

  5. Chen, L., Chua, H., Tan, P.: Grammatical inference using an adaptive recurrent neural network. Neural Processing Letters 8, 211–219 (1998)

    Article  Google Scholar 

  6. Chen, S., Billings, S., Grant, P.: Nonlinear system identification using neural networks. International Journal of Control 51(6), 1191–1214 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  7. Cohen, M.A., Grossberg, S.: Stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE Transactions on Systems, Man and Cybernetics 13, 815–826 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  8. Elman, J.L.: Finding structure in time. Cognitive Science 14, 179–211 (1990)

    Article  Google Scholar 

  9. Elman, J.L.: Distributed representations, simple recurrent networks and grammatical structure. Machine Learning 7, 195–225 (1991)

    Google Scholar 

  10. Fahlman, S.E., Lebiere, C.: The cascade-correlation learning architecture. Technical Report CMU-CS-90-100, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA (February 1990)

    Google Scholar 

  11. Forcada, M.L., Ñeco, R.P.: Recursive Hetero-Associative Memories for Translation. In: Mira, J., Moreno-Díaz, R., Cabestany, J. (eds.) IWANN 1997. LNCS, vol. 1240, pp. 453–462. Springer, Heidelberg (1997)

    Chapter  Google Scholar 

  12. Frasconi, P., Gori, M., Soda, G.: Local feedback multilayered networks. Neural Computation 4, 120–130 (1992)

    Article  Google Scholar 

  13. Galland, C.C., Hinton, G.E.: Deterministic Boltzman learning in networks with asymmetric connectivity. Technical Report CRG-TR-89-6, University of Toronto Department of Computer Science (1989)

    Google Scholar 

  14. Ge, H., Du, W., Qian, F., Liang, Y.: Identification and control of nonlinear systems by a time-delay recurrent neural network. Neurocomputing 72, 2857–2864 (2009)

    Article  Google Scholar 

  15. Giles, C., Kuhn, G., Williams, R.: Dynamic recurrent neural networks: theory and applications. IEEE Trans. Neural Netw. 5(2), 153–156 (1994)

    Google Scholar 

  16. Giles, C.L., Chen, D., Miller, C.B., Chen, H.H., Sun, G.Z., Lee, Y.C.: Second-order recurrent neural networks for grammatical inference. In: 1991 IEEE INNS International Joint Conference on Neural Networks, Seattle, Piscataway, NJ, vol. 2, pp. 271–281. IEEE Press (1991)

    Google Scholar 

  17. Giles, C.L., Horne, B.G., Lin, T.: Learning a class of large finite state machines with a recurrent neural network. Neural Networks 8, 1359–1365 (1995)

    Article  Google Scholar 

  18. Giles, C.L., Miller, C.B., Chen, D., Chen, H.H., Sun, G.Z., Lee, Y.C.: Learning and extracting finite state automata with second-order recurrent neural networks. Neural Computation 4, 395–405 (1992)

    Article  Google Scholar 

  19. Gori, M., Bengio, Y., Mori, R.D.: Bps: A learning algorithm for capturing the dynamic nature of speech. In: International Joint Conference on Neural Networks, vol. II, pp. 417–423 (1989)

    Google Scholar 

  20. Harigopal, U., Chen, H.C.: Grammatical inference using higher order recurrent neural networks. In: Proceedings of the Twenty-Fifth Southeastern Symposium on System Theory, SSST 1993, pp. 338–342 (1993)

    Google Scholar 

  21. Hinton, T.J., abd Sejnowski, G.E.: Optimal perceptual inference. In: Proceedines of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 448–453. IEEE Computer Society (1983)

    Google Scholar 

  22. Hochreiter, S.: Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis (1991)

    Google Scholar 

  23. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 9, 1735–1780 (1997)

    Article  Google Scholar 

  24. Hopcroft, J.E., Ullman, J.D.: Introduction to Automata Theory, Languages, and Computation. Addison-Wesley (1979)

    Google Scholar 

  25. Jordan, M.I.: Supervised learning and systems with excess degrees of freedom. Technical Report COINS Technical Report 88–27, Massachusetts Institute of Technology (1988)

    Google Scholar 

  26. Karakasoglu, A., Sudharsanan, S., Sundareshan, M.K.: Identification and decentralized adaptive control using dynamic neural networks with application to robotic manipulators. IEEE Trans. Neural Networks 4, 919–930 (1993)

    Article  Google Scholar 

  27. Karray, F.O., Silva, C.: Soft Computing and Intelligent Systems Design. Addison Wesley (2004)

    Google Scholar 

  28. Kilian, J., Siegelmann, H.T.: On the power of sigmoid neural networks. In: Proceedings of the Sixth ACM Workshop on Computational Learning Theory, pp. 137–143. ACM Press (1993)

    Google Scholar 

  29. Kolen, J.F., Kremer, S.C. (eds.): A Field Guide to Dynamical Recurrent Networks. Wiley-IEEE Press (2001)

    Google Scholar 

  30. Kuo, J., Celebi, S.: Adaptation of memory depth in the gamma filter. In: Acoustics, Speech and Signal Processing IEEE Conference, pp. 1–4 (1994)

    Google Scholar 

  31. Kuroe, Y.: Representation and Identification of Finite State Automata by Recurrent Neural Networks. In: Pal, N.R., Kasabov, N., Mudi, R.K., Pal, S., Parui, S.K. (eds.) ICONIP 2004. LNCS, vol. 3316, pp. 261–268. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  32. Lippmann, R.P.: An introduction to computing with neural nets. IEEE ASSP Magazine 4, 4–22 (1987)

    Article  Google Scholar 

  33. Mozer, M.: A focused background algorithm for temporal pattern recognition. Complex Systems 3 (1989)

    Google Scholar 

  34. Mozer, M.C.: Induction of multiscale temporal structure. In: Advances in Neural Information Processing Systems 4, pp. 275–282. Morgan Kaufmann (1992)

    Google Scholar 

  35. Nguyen, M., Cottrell, G.: A technique for adapting to speech rate. In: Kamm, C., Kuhn, G., Yoon, B., Chellapa, R., Kung, S. (eds.) Neural Networks for Signal Processing 3. IEEE Press (1993)

    Google Scholar 

  36. Omlin, C.W., Giles, C.L.: Constructing deterministic finite-state automata in recurrent neural networks. Journal of the ACM 43(6), 937–972 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  37. Patan, K.: Locally Recurrent Neural Networks. In: Patan, K. (ed.) Artificial. Neural Net. for the Model. & Fault Diagnosis. LNCIS, vol. 377, pp. 29–63. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  38. Pollack, J.B.: On Connectionist Models of Natural Language Processing. PhD thesis, Computer Science Department of the University of Illinois at Urbana-Champaign, Urbana, Illinois, Available as TR MCCS-87-100, Computing Research Laboratory, New Mexico State University, Las Cruces, NM (1987)

    Google Scholar 

  39. Principe, J.C., de Vries, B., de Oliveira, P.G.: The gamma filter - a new class of adaptive IIR filter with restricted feedback. IEEE Transactions on Signal Processing 41, 649–656 (1993)

    Article  MATH  Google Scholar 

  40. Renals, S., Rohwer, R.: A study of network dynamics. Journal of Statistical Physics 58, 825–848 (1990)

    Article  MathSciNet  MATH  Google Scholar 

  41. Robinson, A.J.: Dynamic Error Propagation Networks. Ph.d., Cambridge University Engineering Department (1989)

    Google Scholar 

  42. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: Parallel Distributed Processing. MIT Press, Cambridge (1986)

    Google Scholar 

  43. Schmidhuber, J.H.: A fixed size storage o(n 3) time complexity learning algorithm for fully recurrent continually running networks. Neural Computation 4(2), 243–248 (1992)

    Article  Google Scholar 

  44. Sejnowski, T.J., Rosenberg, C.R.: Parallel networks that learn to pronounce english text. Complex Syst. I, 145–168 (1987)

    Google Scholar 

  45. Shannon, C.E.: Communication in the presence of noise. Proc. Institute of Radio Engineers 37(1), 10–21 (1949); reprinted as classic paper in: Proc. IEEE 86(2) (February 1998)

    MathSciNet  Google Scholar 

  46. Shearer, J.L., Murphy, A.T., Richardson, H.H.: Introduction to System Dynamics. Addison-Wesley, Reading (1971)

    Google Scholar 

  47. Siegelmann, H.T., Sontag, E.D.: Turing computability with neural nets. Applied Mathematics Letters 4(6), 77–80 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  48. Silva, T.O.: Laguerre filters - an introduction. Revista do Detua 1(3) (1995)

    Google Scholar 

  49. Smith, J.O.: Delay lines. Physical Audio Signal Processing (2010), http://ccrma.stanford.edu/~jos/pasp/Tapped_Delay_Line_TDL.htm (cited November 28, 2010)

  50. Smith, S.W.: The scientist and engineer’s guide to digital signal processing. California Technical Publishing (2006), http://www.dspguide.com/ch15.htm (cited November 29, 2010)

  51. Tsoi, A.C., Back, A.D.: Locally recurrent globally feedforward networks: A critical review of architectures. IEEE Transactions on Neural Networks 5, 229–239 (1994)

    Article  Google Scholar 

  52. Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., Lang, L.: Phonemic recognition using time delay neural networks. IEEE Trans. Acoustic Speech and Signal Processing 37(3), 328–339 (1989)

    Article  Google Scholar 

  53. Werbos, P.: Beyond Regression: New Tools for Prediction and Analysis in the Behavioural Sciences. Phd thesis, Harvard University (1974)

    Google Scholar 

  54. Williams, R.J., Zipser, D.: A learning algorithm for continually running fully recurrent neural networks. Neural Computation 1, 270–289 (1989)

    Article  Google Scholar 

  55. Won, S.H., Song, I., Lee, S.Y., Park, C.H.: Identification of finite state automata with a class of recurrent neural networks. IEEE Transactions on Neural Networks 21(9), 1408–1421 (2010)

    Article  Google Scholar 

  56. Yan, P.F., Zhang, C.S.: Artificial Neural Network and Simulated Evolutionary Computation. Thinghua University Press, Beijing (2000)

    Google Scholar 

  57. Zamarreno, J.M., Vega, P.: State space neural network. Properties and application. Neural Networks 11, 1099–1112 (1998)

    Article  Google Scholar 

  58. Zeng, Z., Goodman, R.M., Smyth, P.: Learning finite state machines with self-clustering recurrent networks. Neural Computation 5(6), 977–990 (1993)

    Article  Google Scholar 

  59. Zeng, Z., Goodman, R.M., Smyth, P.: Discrete recurrent neural networks for grammatical inference. IEEE Transactions on Neural Networks 5(2), 320–330 (1994)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The School of Computer Science, University of Guelph, Guelph, Ontario, Canada

    Sajid A. Marhon, Christopher J. F. Cameron & Stefan C. Kremer

Authors
  1. Sajid A. Marhon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christopher J. F. Cameron
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan C. Kremer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sajid A. Marhon .

Editor information

Editors and Affiliations

  1. , Dipto. Ingegneria dell'Informazione, Università degli Studi di Siena, Via Roma 56, Siena, 53100, Italy

    Monica Bianchini

  2. Fac. Ingegneria, Dipto. Ingegneria dell'Informazione, Università Siena, Via Roma 56, Siena, 53100, Italy

    Marco Maggini

  3. University of Canberra, School of Electrical and Information, Adjunct Professor, Mawson Lakes Campus, ACT, 2601, Australia

    Lakhmi C. Jain

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Marhon, S.A., Cameron, C.J.F., Kremer, S.C. (2013). Recurrent Neural Networks. In: Bianchini, M., Maggini, M., Jain, L. (eds) Handbook on Neural Information Processing. Intelligent Systems Reference Library, vol 49. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-36657-4_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-36657-4_2

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-36656-7

  • Online ISBN: 978-3-642-36657-4

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation