Skip to main content
Log in

Stability of Gated Recurrent Unit Neural Networks: Convex Combination Formulation Approach

  • Technical Note
  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

In this paper, a particular discrete-time nonlinear and time-invariant system represented as a vector difference equation is analyzed for its stability properties. The motivation for analyzing this particular system is that it models gated recurrent unit neural networks commonly used and well known in machine learning applications. From the technical perspective, the analyses exploit the systems similarities to a convex combination of discrete-time systems, where one of the systems is trivial, and thus, the overall properties are mostly dependent on the other one. Stability results are formulated for the nonlinear system and its linearization with respect to the systems, in general, multiple equilibria. To motivate and illustrate the potential of these results in applications, some particular results are derived for the gated recurrent unit neural network models and a connection between local stability analysis and learning is provided.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

References

  1. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  2. Graves, A.: Generating sequences with recurrent neural networks. arXiv:1308.0850 (2013)

  3. Greff, K., Srivastava, R.K., Koutnk, J., Steunebrink, B.R., Schmidhuber, J.: LSTM: a search space odyssey. IEEE Trans. Neural Netw. Learn. Syst. 28(10), 2222–2232 (2017)

    Article  MathSciNet  Google Scholar 

  4. Cho, K., Merrienboer, B.V., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv:1406.1078 (2014)

  5. Bertschinger, N., Natschläger, T.: Real-time computation at the edge of chaos in recurrent neural networks. Neural Comput. 16(7), 1413–1436 (2004)

    Article  Google Scholar 

  6. Leonov, G.A.: Strange Attractors and Classical Stability Theory. St. Petersburg University Press, St. Petersburg (2008)

    MATH  Google Scholar 

  7. Hirsch, M.W., Smale, S., Devaney, R.L.: Differential Equations, Dynamical Systems, and an Introduction to Chaos, 3rd edn. Elsevier Inc., Waltham (2013)

    MATH  Google Scholar 

  8. Stipanović, D.M., Murmann, B., Causo, M., Lekić, A., Royo, V.R., Tomlin, C.J., Beigne, E., Thuries, S., Zarudniev, M., Lesecq, S.: Some local stability properties of an autonomous long short-term memory neural network model. In: Proceedings of the 2018 IEEE International Symposium on Circuits and Systems (2018)

  9. Deka, S.A., Stipanović, D.M., Murmann, B., Tomlin, C.J.: Stabilization of long short-term memory neural networks with constant weights and biases. J. Optim. Theory Appl. 181, 231–243 (2019)

    Article  MathSciNet  Google Scholar 

  10. Deka, S.A., Stipanović, D.M., Murmann, B., Tomlin, C.J.: Long-short term memory neural network stability and stabilization using linear matrix inequalities. In: Proceedings of the 2019 IEEE International Symposium on Circuits and Systems (2019)

  11. Filippov, A.F.: Differential Equations with Discontinuous Righthand Sides. Kluwer Academic Publishers, Dordrecht (1988)

    Book  Google Scholar 

  12. Shevitz, D.S., Paden, B.E.: Lyapunov stability theory of nonsmooth systems. IEEE Trans. Autom. Control 39, 1910–1914 (1994)

    Article  MathSciNet  Google Scholar 

  13. Stipanović, D.M., Šiljak, D.D.: Connective stability of discontinuous dynamic systems. J. Optim. Theory Appl. 115, 711–726 (2002)

    Article  MathSciNet  Google Scholar 

  14. Stanković, S.S., Stanković, M.S., Stipanović, D.M.: Decentralized parameter estimation by consensus based stochastic approximation. IEEE Trans. Autom. Control 56, 531–543 (2011)

    Article  MathSciNet  Google Scholar 

  15. LaSalle, J.P.: The Stability and Control of Discrete Processes. Springer, New York (1986)

    Book  Google Scholar 

  16. Barmish, R.B.: New Tools for Robustness of Linear Systems. Macmilllan Publishing Company, New York (1994)

    MATH  Google Scholar 

  17. Stipanović, D.M., Šiljak, D.D.: Stability of polytopic systems via convex M-matrices and parameter-dependent Liapunov functions. Nonlinear Anal. 40, 589–609 (2000)

    Article  MathSciNet  Google Scholar 

  18. Kanai, S., Fujiwara, Y., Iwamura, S.: Preventing gradient explosions in gated recurrent units. In: Proceedings of the 2017 Neural Information Processing Systems (NIPS) Conference (2017)

  19. Ravanelli, M., Brakel, P., Omologo, M., Bengio, Y.: Light gated recurrent units for speech recognition. IEEE Trans. Emerg. Topics Computat. Intell. 2(2), 92–102 (2018)

    Article  Google Scholar 

  20. Michel, A.N., Liu, D.: Qualitative Analysis and Synthesis of Recurrent Neural Networks. Marcel Dekker, New York (2002)

    MATH  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Robotics Initiative grant titled “NRI: FND: COLLAB: Multi-Vehicle Systems for Collecting Shadow-Free Imagery in Precision Agriculture” (grant no. 2019-04791/project accession no. 1020285) from the USDA National Institute of Food and Agriculture and by the grant no. 451-03-68/2020-14/200156, titled “Innovative Scientific and Artistic Research from the FTS (activity) Domain” from the Serbian Ministry of Education, Science and Technological Development.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Dušan M. Stipanović.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stipanović, D.M., Kapetina, M.N., Rapaić, M.R. et al. Stability of Gated Recurrent Unit Neural Networks: Convex Combination Formulation Approach. J Optim Theory Appl 188, 291–306 (2021). https://doi.org/10.1007/s10957-020-01776-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-020-01776-w

Keywords

Mathematics Subject Classification

Navigation