Skip to main content
SpringerLink
Log in

A Dynamically Stabilized Recurrent Neural Network

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

This work proposes a novel recurrent neural network architecture, called the Dynamically Stabilized Recurrent Neural Network (DSRNN). The developed DSRNN includes learnable skip-connections across a specified number of time-steps, which allows for a state-space representation of the network’s hidden-state trajectory, and a regularization term is introduced in the loss function in the setting of Lyapunov stability theory. The regularizer enables the placement of eigenvalues of the (linearized) transfer function matrix to desired locations in the complex plane, thereby acting as an internal controller for the hidden-state trajectories. In this way, the DSRNN adjusts the weights of temporal skip-connections to achieve recurrent hidden-state stability, which mitigates the problems of vanishing and exploding gradients. The efficacy of the DSRNN is demonstrated on a forecasting task of a recorded double-pendulum experimental model. The results show that the DSRNN outperforms both the Long Short-Term Memory (LSTM) and vanilla recurrent neural networks, and the relative mean-squared error of the LSTM is reduced by up to \(\sim \)99.64%. The DSRNN also showed comparable results to the LSTM on a classification task of two Lorenz oscillator systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117

    Article  Google Scholar 

  2. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  3. Gers FA, Schmidhuber J, Cummins F (1999) Learning to forget: continual prediction with lstm

  4. Greff K, Srivastava RK, Koutník J, Steunebrink BR, Schmidhuber J (2016) Lstm: a search space odyssey. IEEE Trans Neural Netw Learn Syst 28(10):2222–2232

    Article  MathSciNet  Google Scholar 

  5. Lippi M, Montemurro MA, Degli Esposti M, Cristadoro G (2019) Natural language statistical features of lstm-generated texts. IEEE Trans Neural Netw Learn Syst 30(11):3326–3337

    Article  Google Scholar 

  6. Yu X, Wu L, Xu C, Hu Y, Ma C (2019) A novel neural network for solving nonsmooth nonconvex optimization problems. IEEE Trans Neural Netw Learn Syst

  7. Qin S, Xue X (2014) A two-layer recurrent neural network for nonsmooth convex optimization problems. IEEE Trans Neural Netw Learn Syst 26(6):1149–1160

    Article  MathSciNet  Google Scholar 

  8. Che H, Wang J (2018) A two-timescale duplex neurodynamic approach to biconvex optimization. IEEE Trans Neural Netw Learn Syst 30(8):2503–2514

    Article  MathSciNet  Google Scholar 

  9. Chen K, Yao L, Zhang D, Wang X, Chang X, Nie F (2019) A semisupervised recurrent convolutional attention model for human activity recognition. IEEE Trans Neural Netw Learn Syst

  10. Lipton ZC, Kale D, Wetzel R (2016) Directly modeling missing data in sequences with rnns: improved classification of clinical time series. In: Machine learning for healthcare conference, pp 253–270

  11. Miller J, Hardt M (2018) Stable recurrent models. arXiv preprint, arXiv:1805.10369

  12. Bao G, Peng Y, Zhou X, Gong S (2020) Region stability and stabilization of recurrent neural network with parameter disturbances. Neural Process Lett 52(3):2175–2188

    Article  Google Scholar 

  13. Chandran R, Balasubramaniam P (2013) Delay dependent exponential stability for fuzzy recurrent neural networks with interval time-varying delay. Neural Process Lett 37(2):147–161

    Article  Google Scholar 

  14. Ba JL, Kiros JR, Hinton GE (2016) Layer normalization. arXiv preprint, arXiv:1607.06450

  15. Bengio Y, Boulanger-Lewandowski N, Pascanu R (2013) Advances in optimizing recurrent networks. In: 2013 IEEE international conference on acoustics, speech and signal processing, pp 8624–8628, IEEE

  16. Jaeger H, Lukoševičius M, Popovici D, Siewert U (2007) Optimization and applications of echo state networks with leaky-integrator neurons. Neural Netw 20(3):335–352

    Article  Google Scholar 

  17. Zhang Y, Chen G, Yu D, Yaco K, Khudanpur S, Glass J (2016) Highway long short-term memory rnns for distant speech recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 5755–5759, IEEE

  18. Kim J, El-Khamy M, Lee J (2017) Residual lstm: design of a deep recurrent architecture for distant speech recognition. arXiv preprint, arXiv:1701.03360

  19. Haviv D, Rivkind A, Barak O (2019) Understanding and controlling memory in recurrent neural networks. In: International conference on machine learning, pp 2663–2671, PMLR

  20. Kalman RE, Bertram JE (1960) Control system analysis and design via the “second method" of lyapunov: I—continuous-time systems. J Basic Eng 82(2):371–393

    Article  MathSciNet  Google Scholar 

  21. Hauser M, Gunn S, Saab S Jr, Ray A (2019) State-space representations of deep neural networks. Neural Comput 31(3):538–554

    Article  MathSciNet  Google Scholar 

  22. Tay Y, Luu AT, Hui SC (2018) Recurrently controlled recurrent networks. In: Bengio S, Wallach H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R (eds) Advances in neural information processing systems 31, pp 4736–4748, Curran Associates, Inc

  23. Asseman A, Kornuta T, Ozcan A (2018) Learning beyond simulated physics

  24. de Jesús Serrano-Pérez J, Fernández-Anaya G, Carrillo-Moreno S, Yu W (2021) New results for prediction of chaotic systems using deep recurrent neural networks. Neural Process Lett, pp 1–18

  25. Bof N, Carli R, Schenato L (2018) Lyapunov theory for discrete time systems. arXiv preprint, arXiv:1809.05289

  26. Werbos PJ (1990) Backpropagation through time: what it does and how to do it. Proc IEEE 78(10):1550–1560

    Article  Google Scholar 

  27. Pascanu R, Mikolov T, Bengio Y (2013) On the difficulty of training recurrent neural networks. In: International conference on machine learning, pp 1310–1318

  28. Ouyang X, Luo Y, Liu J, Liu Y, Bi J, Qiu S (2018) Period analysis of chaotic systems under finite precisions. In: 2018 26th International conference on systems engineering (ICSEng), pp 1–5, IEEE

  29. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp 249–256

  30. Kingma DP, Ba J (2014) Adam: a method for stochastic optimization, arXiv preprint, arXiv:1412.6980

  31. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: a system for large-scale machine learning. OSDI 16:265–283

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, The Pennsylvania State University, State College, PA, 16802, USA

    Samer Saab Jr.

  2. Department of Mechanical Engineering, The Pennsylvania State University, State College, PA, 16802, USA

    Yiwei Fu, Asok Ray & Michael Hauser

  3. Department of Mathematics, The Pennsylvania State University, State College, PA, 16802, USA

    Asok Ray

Authors
  1. Samer Saab Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yiwei Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Asok Ray
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Hauser
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Samer Saab Jr..

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saab, S., Fu, Y., Ray, A. et al. A Dynamically Stabilized Recurrent Neural Network. Neural Process Lett 54, 1195–1209 (2022). https://doi.org/10.1007/s11063-021-10676-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-021-10676-7

Keywords

Search

Navigation