Skip to main content
SpringerLink
Log in

Training recurrent neural networks using a hybrid algorithm

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This paper proposes a new hybrid approach for recurrent neural networks (RNN). The basic idea of this approach is to train an input layer by unsupervised learning and an output layer by supervised learning. In this method, the Kohonen algorithm is used for unsupervised learning, and dynamic gradient descent method is used for supervised learning. The performances of the proposed algorithm are compared with backpropagation through time (BPTT) on three benchmark problems. Simulation results show that the performances of the new proposed algorithm exceed the standard backpropagation through time in the reduction of the total number of iterations and in the learning time required in the training process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Nerrand O, Roussel-Ragot P, Urbani D, Personnaz L, Dreyfus G (1994) Training recurrent neural networks: why and How? An illustration in dynamical process modeling. IEEE Trans Neural Netw 5(2):178–184

    Article  Google Scholar 

  2. Narendra KS, Parthasarathy K (1990) Identification and control of dynamical systems using neural networks. IEEE Trans Neural Netw 1(1):4–27

    Article  Google Scholar 

  3. Li X, Yu W (2002) Dynamic system identification via recurrent multilayer perception. Inf Sci 147:45–63

    Article  MathSciNet  MATH  Google Scholar 

  4. Guler I, Ubeyli ED (2006) A recurrent neural networks classifier for doppler ultrasound blood flow signal. Pattern Recogn Lett 27:1560–1571

    Article  Google Scholar 

  5. Puskorius GV, Feldkamp LA (1998) A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification. Proc IEEE 86:2259–2277

    Article  Google Scholar 

  6. Chtourou S, Chtourou M, Hammami O (2008) A hybrid approach for training recurrent neural networks: application to multi-Step-ahead prediction of noisy and large data sets. Neural Comput Appl 17:245–254

    Article  Google Scholar 

  7. Connor T, Douglas R (1994) Recurrent neural networks and robust time series prediction. IEEE Trans Neural Netw 5(2):240–254

    Article  Google Scholar 

  8. Graves D, Pedrycz W (2009) Fuzzy prediction architecture using recurrent neural networks. Neurocomputing 72:1668–1678

    Article  Google Scholar 

  9. Funahashi KI, Nakamura Y (1993) Approximation of dynamical systems by continuous networks. Neural Netw 6:801–806

    Article  Google Scholar 

  10. Kosmatopoulos EA, Polycarpou MM, Christodoulou MA, Oannou PI (1995) High-order neural network structures for identification of dynamical systems. IEEE Trans Neural Netw 6:422–431

    Article  Google Scholar 

  11. Werbos PJ (1990) Backpropagation through time: What it does and how to do it. IEEE Proc 78(10):1550–1560

    Article  Google Scholar 

  12. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation. Parallel distributed processing: explorations in the microstructures of cognition, vol 1. MIT Press, Cambridge, pp 318–362

    Google Scholar 

  13. Williams RJ, Zipser D (1989) A learning algorithm for continually running fully recurrent neural networks. Neural Comput 1:270–280

    Article  Google Scholar 

  14. Toomarian N, Barben J (1991) Adjoint-function and temporal learning algorithms in neural networks. Adv Neural Inf Process Syst 3:113–120

    Google Scholar 

  15. Toomarian N, Barben J (1992) Learning a trajectory using adjoint functions and teacher forcing. Neural Netw 5(3):473–484

    Article  Google Scholar 

  16. Sim GZ, Chen HH, Lee YC (1992) Green’s function method for fast on-line learning algorithm of recurrent neural networks. Adv Neural Inf Process Syst 4:333–340

    Google Scholar 

  17. Schmidhuber J (1992) A fixed storage O(N3) time complexity learning algorithm for fully recurrent continually running networks. Neural Comput 4(2):243–248

    Article  Google Scholar 

  18. Williams R (1989) Complexity of exact gradient computation algorithms for recurrent neural networks. Northeasterm University, College Comp. Sci., Boston, MA, Tech. Rep. NU-CCS-89-27

  19. Feldkamp LA, Prokhorov DV (1998) Phased backpropagation: a hybrid of BPTT and temporal BP. Proc IEEE Int Joint Conf Neural Netw 3:2262–2267

    Google Scholar 

  20. Wan EA (1990) Temporal backpropagation for FIR neural networks. Proc IEEE Int Joint Conf Neural Netw 1:575–580

    Article  Google Scholar 

  21. Hochneiter S, Schmidhuber J (1997) Long short term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  22. Atiya AF, Parlos AG (2000) New results on recurrent network training: Unifying the algorithms and accelerating convergence. IEEE Trans Neural Netw 11(3):697–709

    Article  Google Scholar 

  23. De Jesus O, Hagan MT (2007) Backpropagation algorithms for a broad class of dynamics networks. IEEE Trans Neural Netw 18(1):14–27

    Article  Google Scholar 

  24. Song Q, Wu Y, Soh YC (2008) Robust adaptive gradient descent training algorithm for recurrent neural networks in discrete time domain. IEEE Trans Neural Netw 19(11):1841–1853

    Article  Google Scholar 

  25. Mandic DP, Chambers JA (2001) Recurrent neural networks for prediction: learning algorithm, architecture and stability. Wiley, Chichester, UK

    Book  Google Scholar 

  26. Williams R (1992) Training recurrent networks using the extended kalman filter, international joint conference on neural networks, Baltimore, ΙV, pp 241–246

  27. Puskorius GV, Feldkamp LA (1992) Recurrent networks training with decoupled extended Kalman Filter algorithm. In: Proceedings of the 1992 SPIE conference on the science of artificial neural networks, Orlando 1710, pp 461–473

  28. Vartak AA, Georgiopoulos M, Anagnostopoulos GC (2005) On line gauss- Newton—based learning for fully recurrent neural networks. Nonlinear Anal 63:867–876

    Article  Google Scholar 

  29. Ben Nasr M, Chtourou M (2006) A hybrid training algorithm for feedforward neural networks. Neural Process Lett 24(2):107–117

    Article  Google Scholar 

  30. Ben Nasr M, Chtourou M (2009) A fuzzy neighborhood-based training algorithm for feedforward neural networks. Neural Comput Appl 18(2):127–133

    Article  Google Scholar 

  31. Kohonen T (1990) The self- organizing map. Proc IEEE 78(9):1464–1480

    Article  Google Scholar 

  32. Box GE, Jenkins GM (1970) Time series analysis, forecasting and control, San Francisco, Holden Day

  33. Li M, Mechrotra K, Mohan C, Ranka S (1990) Sunspot numbers forecasting using neural network. In: Proc. IEEE Int. Symp. Intell. Contr., pp 524–528

  34. Kim J (1999) Adaptive neuro-fuzzy inference system and their application to non linear dynamical system. Neural Netw 12:1301–1319

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank all the anonymous reviewers for their useful suggestions, which improved the quality of this paper.

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Research Unit on Intelligent Control, Design and Optimization of Complex Systems (ICOS), University of Sfax, ENIS, BP W, 3038, Sfax, Tunisia

    Mounir Ben Nasr & Mohamed Chtourou

Authors
  1. Mounir Ben Nasr
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed Chtourou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mounir Ben Nasr.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nasr, M.B., Chtourou, M. Training recurrent neural networks using a hybrid algorithm. Neural Comput & Applic 21, 489–496 (2012). https://doi.org/10.1007/s00521-010-0506-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-010-0506-1

Keywords

Search

Navigation