Abstract
In this paper, a particular discrete-time nonlinear and time-invariant system represented as a vector difference equation is analyzed for its stability properties. The motivation for analyzing this particular system is that it models gated recurrent unit neural networks commonly used and well known in machine learning applications. From the technical perspective, the analyses exploit the systems similarities to a convex combination of discrete-time systems, where one of the systems is trivial, and thus, the overall properties are mostly dependent on the other one. Stability results are formulated for the nonlinear system and its linearization with respect to the systems, in general, multiple equilibria. To motivate and illustrate the potential of these results in applications, some particular results are derived for the gated recurrent unit neural network models and a connection between local stability analysis and learning is provided.
References
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Graves, A.: Generating sequences with recurrent neural networks. arXiv:1308.0850 (2013)
Greff, K., Srivastava, R.K., Koutnk, J., Steunebrink, B.R., Schmidhuber, J.: LSTM: a search space odyssey. IEEE Trans. Neural Netw. Learn. Syst. 28(10), 2222–2232 (2017)
Cho, K., Merrienboer, B.V., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv:1406.1078 (2014)
Bertschinger, N., Natschläger, T.: Real-time computation at the edge of chaos in recurrent neural networks. Neural Comput. 16(7), 1413–1436 (2004)
Leonov, G.A.: Strange Attractors and Classical Stability Theory. St. Petersburg University Press, St. Petersburg (2008)
Hirsch, M.W., Smale, S., Devaney, R.L.: Differential Equations, Dynamical Systems, and an Introduction to Chaos, 3rd edn. Elsevier Inc., Waltham (2013)
Stipanović, D.M., Murmann, B., Causo, M., Lekić, A., Royo, V.R., Tomlin, C.J., Beigne, E., Thuries, S., Zarudniev, M., Lesecq, S.: Some local stability properties of an autonomous long short-term memory neural network model. In: Proceedings of the 2018 IEEE International Symposium on Circuits and Systems (2018)
Deka, S.A., Stipanović, D.M., Murmann, B., Tomlin, C.J.: Stabilization of long short-term memory neural networks with constant weights and biases. J. Optim. Theory Appl. 181, 231–243 (2019)
Deka, S.A., Stipanović, D.M., Murmann, B., Tomlin, C.J.: Long-short term memory neural network stability and stabilization using linear matrix inequalities. In: Proceedings of the 2019 IEEE International Symposium on Circuits and Systems (2019)
Filippov, A.F.: Differential Equations with Discontinuous Righthand Sides. Kluwer Academic Publishers, Dordrecht (1988)
Shevitz, D.S., Paden, B.E.: Lyapunov stability theory of nonsmooth systems. IEEE Trans. Autom. Control 39, 1910–1914 (1994)
Stipanović, D.M., Šiljak, D.D.: Connective stability of discontinuous dynamic systems. J. Optim. Theory Appl. 115, 711–726 (2002)
Stanković, S.S., Stanković, M.S., Stipanović, D.M.: Decentralized parameter estimation by consensus based stochastic approximation. IEEE Trans. Autom. Control 56, 531–543 (2011)
LaSalle, J.P.: The Stability and Control of Discrete Processes. Springer, New York (1986)
Barmish, R.B.: New Tools for Robustness of Linear Systems. Macmilllan Publishing Company, New York (1994)
Stipanović, D.M., Šiljak, D.D.: Stability of polytopic systems via convex M-matrices and parameter-dependent Liapunov functions. Nonlinear Anal. 40, 589–609 (2000)
Kanai, S., Fujiwara, Y., Iwamura, S.: Preventing gradient explosions in gated recurrent units. In: Proceedings of the 2017 Neural Information Processing Systems (NIPS) Conference (2017)
Ravanelli, M., Brakel, P., Omologo, M., Bengio, Y.: Light gated recurrent units for speech recognition. IEEE Trans. Emerg. Topics Computat. Intell. 2(2), 92–102 (2018)
Michel, A.N., Liu, D.: Qualitative Analysis and Synthesis of Recurrent Neural Networks. Marcel Dekker, New York (2002)
Acknowledgements
This work is supported by the National Robotics Initiative grant titled “NRI: FND: COLLAB: Multi-Vehicle Systems for Collecting Shadow-Free Imagery in Precision Agriculture” (grant no. 2019-04791/project accession no. 1020285) from the USDA National Institute of Food and Agriculture and by the grant no. 451-03-68/2020-14/200156, titled “Innovative Scientific and Artistic Research from the FTS (activity) Domain” from the Serbian Ministry of Education, Science and Technological Development.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Stipanović, D.M., Kapetina, M.N., Rapaić, M.R. et al. Stability of Gated Recurrent Unit Neural Networks: Convex Combination Formulation Approach. J Optim Theory Appl 188, 291–306 (2021). https://doi.org/10.1007/s10957-020-01776-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10957-020-01776-w