Skip to main content
SpringerLink
Log in

Advanced RNN Based NARMA Predictors

  • Published:
Journal of VLSI signal processing systems for signal, image and video technology Aims and scope Submit manuscript

Abstract

An analysis of nonlinear time series prediction schemes, realised though advanced Recurrent Neural Network (RNN) techniques is provided. Due to practical constraints in using common RNNs, such as the problem of vanishing gradient, some other ways to improve RNN based prediction are analysed. This is undertaken for a simple RNN through to the Pipelined Recurrent Neural Network (PRNN), which consists of a number of nested small-scale RNNs. A Nonlinear AutoRegressive Moving Average (NARMA) nonlinear model is introduced in the context of RNN architectures, and an posteriori mode of operation within that framework. Moreover, it is shown that the basic a priori PRNN structure exhibits certain a posteriori features. The PRNN based predictor, is shown to exhibit nesting, and to be able to represent block cascaded stochastic models, such as the Wiener–Hammerstein model. Simulations undertaken on a speech signal support the analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. J. Makhoul, “Linear prediction: A Tutorial Overview,” Proceedings of the IEEE, vol. 63, no.4, 1975, pp. 561–580.

    Article  Google Scholar 

  2. G.E. Box and G.M. Jenkins, Time Series Analysis: Forecasting and Control, 2nd ed., Holden-Day, 1976.

  3. L. Ljung and T. Soderstrom, Theory and Practice of Recursive Identification Cambridge, Mass: MIT Press, 1983.

    MATH  Google Scholar 

  4. S. Haykin, Neural Networks–A Comprehensive Foundation, Prentice Hall, 1994.

  5. R.M. Dillon and C.N. Manikopoulos, “Neural Net Nonlinear Prediction for Speech Data,” Electronics Letters, vol. 27, no.10, 1991, pp. 824–826.

    Article  Google Scholar 

  6. B. Townshend, “Nonlinear Prediction of Speech,” in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP-91), 1991, pp. 425–428.

  7. C.R. Gent and C.P. Sheppard, “Predicting Time Series by a Fully Connected Neural Network Trained by Backpropagation,” Computing and Control Engineering Journal, 1992, pp. 109–112.

  8. L.K. Li, “Approximation Theory and Recurrent Networks,” in Proceedings of the International Joint Conference on Neural Networks, vol. II, 1992, pp. 266–271.

    Google Scholar 

  9. R. Williams and D. Zipser, “A Learning Algorithm for Continually Running Fully Recurrent Neural Networks,” Neural Computation, vol. 1, 1989, pp. 270–280.

    Article  Google Scholar 

  10. Y. Bengio, P. Simard, and P. Frasconi, “Learning Long-Term DependenciesWith Gradient Descent is Difficult,” IEEE Transactions on Neural Networks, vol. 5, no.2, 1994, pp. 157–166.

    Article  Google Scholar 

  11. J.T. Connor, R.D. Martin, and L.E. Atlas, “Recurrent Neural Networks and Robust Time Series Prediction,” IEEE Transactions on Neural Networks, vol. 5, no.2, 1994, pp. 240–254.

    Article  Google Scholar 

  12. H.T. Siegelmann, B.G. Horne, and C.L. Giles, “Computational Capabilities of Recurrent NARX Neural Networks,” IEEE Transactions on Systems, Man and Cybernetics-Part B: Cybernetics, vol. 27, no.2, 1997, pp. 208–215.

    Article  Google Scholar 

  13. S. Haykin and L. Li, “Nonlinear Adaptive Prediction of Non-Stationary Signals,” IEEE Transactions on Signal Processing, vol. 43, no.2, 1995, pp. 526–535.

    Article  Google Scholar 

  14. M.B. Priestley, Non-linear and Non-stationary Time Series Analysis, London: Academic Press, 1991.

    MATH  Google Scholar 

  15. P.-R. Chang and J.-T. Hu, “Optimal Nonlinear Adaptive Prediction and Modeling of MPEG Video in ATM Networks Using Pipelined Recurrent Neural Networks,” IEEE Journal on Selected Areas in Communications, vol. 15, no.6, 1997, pp. 1087–1100.

    Article  Google Scholar 

  16. D.P. Mandic, J. Baltersee, and J.A. Chambers, “Nonlinear Prediction of Speech With a Pipelined Recurrent Neural Network and Advanced Learning Algorithms,” in Signal Analysis and Prediction, A. Prochazka, J. Uhlir, P.J.W. Rayner, and N.G. Kingsbury (Eds.), Boston: Birkhauser, 1998, pp. 291–309.

    Chapter  Google Scholar 

  17. R.J. Williams and D. Zipser, “Gradient-Based Algorithms for Recurrent Networks and Their Computational Complexity,” in Backpropagation: Theory, Architectures, and Applications, Y. Chauvin and D.E. Rumelhart (Eds.), Lawrence Erlbaum Associates Inc., 1995.

  18. D.P. Mandic and J.A. Chambers, “A Posteriori Real Time Recurrent Learning Schemes for a Recurrent Neural Network Based Non-Linear Predictor,” IEE Proceedings–Vision, Image and Signal Processing, vol. 145, no.6, 1998, pp. 365–370.

    Article  Google Scholar 

  19. J.R. Treichler, C.R. Johnson, Jr., and M.G. Larimore, Theory and Design of Adaptive Filters, John Wiley & Sons, 1987.

  20. J.E. Dennis, Jr. and R.B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations Prentice-Hall 1983, Series in Computational Mathematics.

  21. K.Hwang and F.A. Briggs, Computer Architectures and Parallel Processing, McGraw-Hill, 1986.

  22. D.P. Mandic and J.A. Chambers, “From an A Priori RNN to an A Posteriori PRNN Nonlinear Predictor,” in Proceedings of the VIII IEEE Workshop on Neural Networks for Signal Processing (NNSP98), 1998, pp. 174–183.

  23. J.Baltersee and J.A. Chambers, “Non-linear Adaptive Prediction of Speech Signals Using a Pipelined Recurrent Neural Network,” IEEE Transactions on Signal Processing, vol. 46, no.8, 1998, pp. 2207–2216.

    Article  Google Scholar 

  24. D.G. Luenberger, Optimization by Vector Space Methods, Wiley, 1969.

  25. P.E. Gill, W. Murray, and M.H. Wright, Practical Optimization, London: Academic Press, 1981.

    MATH  Google Scholar 

  26. D.P. Mandic and J.A. Chambers, “Global Asymptotic Stability of Nonlinear Relaxation Equations Realised Through a Recurrent Perceptron,” in Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP-99), vol. 2, 1999, pp. 1037–1040.

    Google Scholar 

  27. D.P. Mandic and J.A. Chambers, “Advanced PRNN Based Nonlinear Prediction/System Identification,” in Digest of the IEE Colloquium on Non-Linear Signal and Image Processing, 1998, pp. 11/1–11/6.

  28. K.S. Narendra and K. Parthasarathy, “Identification and Control of Dynamical Systems Using Neural Networks,” IEEE Transactions on Neural Networks, vol. 1, no.1, 1990, pp. 4–27.

    Article  Google Scholar 

  29. A.D. Back and A.C. Tsoi, “Nonlinear System Identification Using Multilayer Perceptrons With Locally Recurrent Synaptic Structure,” in Proceedings of the IEEE-SP Workshop NNSP II, 1992, pp. 444–453.

    Google Scholar 

  30. T. Lin, B.G. Horne, and C.L. Giles, “How Embedded Memory in Recurrent Neural Network Architectures Helps Learning Long-Term Temporal Dependencies,” Neural Networks, vol. 11, 1998, pp. 861–868.

    Article  MATH  Google Scholar 

  31. S.-S. Kim, “Time-Delay Recurrent Neural Network for Temporal Correlations and Prediction,” Neurocomputing, vol. 20, 1998, pp. 253–263.

    Article  MATH  Google Scholar 

  32. T. Lin, B.G. Horne, P. Tino, and C.L. Giles, “Learning Long-Term Dependencies in NARX Recurrent Neural Networks,” IEEE Transactions on Neural Networks, vol. 7, no.6, 1996, pp. 1329–1338.

    Article  Google Scholar 

  33. D.P. Mandic and J.A. Chambers, “Toward an Optimal PRNN Based Nolinear Predictor,” IEEE Transactions on Neural Networks, vol. 10, no.6, 1999, pp. 1435–1442.

    Article  Google Scholar 

  34. A.J. Robinson and F. Fallside, “The Utility Driven Dynamic Error Propagation Network,” Technical Report CUED/F-INFENG/TR.1, Cambridge University, Engineering Department, 1987.

Download references

Author information

Authors and Affiliations

  1. Image, Speech and Signal Processing Research Group, School of Information Systems, University of East Anglia, Norwich, UK

    Danilo P. Mandic

  2. Communications and Signal Processing Research Group, Department of Electrical Engineering, Imperial College, London, UK

    Jonathon A. Chambers

Authors
  1. Danilo P. Mandic
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathon A. Chambers
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Mandic, D.P., Chambers, J.A. Advanced RNN Based NARMA Predictors. The Journal of VLSI Signal Processing-Systems for Signal, Image, and Video Technology 26, 105–117 (2000). https://doi.org/10.1023/A:1008151602135

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1008151602135

Keywords

Search

Navigation