Multistability of recurrent neural networks with time-varying delays and nonincreasing activation function

doi:10.1016/j.neucom.2016.07.032

Neurocomputing

Volume 216, 5 December 2016, Pages 135-142

https://doi.org/10.1016/j.neucom.2016.07.032 Get rights and content

Abstract

In this paper, we are concerned with a class of recurrent neural networks (RNNs) with nonincreasing activation function. First, based on the fixed point theorem, it is shown that under some conditions, such an n-dimensional neural network with nondecreasing activation function can have at least ${(4 k + 3)}^{n}$ equilibrium points. Then, it proves that there is only ${(4 k + 3)}^{n}$ equilibria under some conditions, among which ${(2 k + 2)}^{n}$ equilibria are locally stable. Besides, by analysis and study of RNNs with nondecreasing activation function, we can also obtain the same number of equilibria for RNNs with nonincreasing activation function. Finally, two simulation examples are given to show effectiveness of the obtained results.

Introduction

Dynamic behavior of equilibrium point has been one of the important contents of the Lyapunov theory, and has also been multistability concerned topic. Multistability study are beginning from multiple attractors analysis, which is closely related with the dynamic behavior of equilibrium point. In recent years, with the development of multistability and neural network, many efforts have been made on the applications such as signal processing, associative memories, image processing, pattern recognition, optimization problems, and so on (see [1], [2], [3], [4], [5], [6]). Such applications also rely heavily on the dynamical properties of neural network systems. With the development of the application, it remains to be further indepth study dynamics behavior of neural network.

For the existence of equilibrium in selecting subset, the existing literature often adopts two kinds of methods. The two methods are respectively based on Banach fixed point and Brouwer fixed point. Adopting the Banach fixed point method, there is a strong constraint for the subarea. If the selected area is too large, it may not satisfy contraction; however, if the selected area is too small, it may not satisfy the condition of self-mapping. Using the other fixed point method, the exact number of the fixed point is not clear. Therefore, in order to make up for these deficiencies, it is necessary to combine the two methods.

Based on the Banach fixed point, it was shown that cellular neural networks could have 3ⁿ memory patterns, of which 2ⁿ were locally exponentially stably in [7]. In order to increase storage capacity, a class of discontinuous activation functions were introduced in [8], the coexistence of ${(4 k - 1)}^{n}$ locally stably equilibrium points were derived. Multistability of recurrent neural networks (RNNs) with activation function symmetrical about the origin on the phase plane were also investigated in [9], in which new criteria on the multistability of neural networks were proposed. Also in [10], [11], [12], [13], [14], some multistability properties of neural networks were investigated, and some sufficient conditions were proposed to ensure system multistability.

Based on the Brouwer fixed point, in [15], the coexistence of multiple equilibrium points was investigated based on the geometrical configuration of the Fermi activation functions. In order to expand application scope, a class of nonsmooth activation functions were introduced in high-order neural networks in [16], and it was shown that the coexistence of 3ⁿ equilibrium points and the local stability of 2ⁿ equilibrium points. In addition, in [17], neural networks with discontinuous non-monotonic piecewise linear activation functions could have at least 5ⁿ equilibrium points, 3ⁿ of which were locally stable and the others were unstable. For more references, refer to [18], [19], [20] and so on.

In addition, the neural networks with concave-convex characteristics were investigated in [21], [22], and it was shown that method based on a kind of other types of fixed point is given in these paper. In [23], multistable property of neural networks were addressed and this provided a method based on dynamic analysis, which had been applied to study the convergence of equilibria. Of course, dynamics analysis method combined with the fixed point method, also could deal with the coexistence of multistability, for example in [24], [25], [26], [27], [28], [29] and so on. To explore multistability, these methods from [30], [31] were also very valuable.

As far as we know, the type of activation functions plays a crucial role in the multistability analysis of neural networks. In the above-mentioned works as well as most existing works, the activation functions applied to multistability analysis, including nondecreasing activation functions and nonmonotonic activation functions, were mainly focused on piecewise constants activation functions, sigmoidal activation functions, and nondecreasing saturated activation functions. For example, in [10], multistability for neural networks with nondecreasing saturated activation functions was addressed. The nondecreasing activation function in [10] was defined as the following form: $f (r) = {\begin{matrix} 4 k - 3, & r \in [4 k - 3, + \infty) \\ 2 r - (4 k - 3), & r \in [4 k - 5, 4 k - 3), \\ \dots, & \dots, \\ 2 r - 5, & r \in [3, 5), \\ 1 & r \in [1, 3), \\ r, & r \in (- 1, 1), \\ - 1, & r \in (- 3, 1], \\ 2 r + 5, & r \in (- 5, - 3], \\ \dots, & \dots, \\ 2 r + 4 k - 3, & r \in (3 - 4 k, 5 - 4 k], \\ 3 - 4 k, & r \in (- \infty, 3 - 4 k] \end{matrix}$

By the above continuous activation functions evolution, in this paper, we introduce a general class of activation functions, which are defined as follows: $g (r) = \sum_{s = 1}^{N} \frac{(m_{s - 1} - m_{s})}{2} (| r + E_{s} | - | r - E_{s} |)$ where $N = 2 k + 1, m_{0} = - 1, E_{s} = 3 s - 2, m_{s} = - 1 + {(- 1)}^{s + 1}$ and $s = 1, 2, \dots, 2 k + 1 .$

It is obvious that $g (r)$ is a nonincreasing and odd function. When $N = 2 k - 1, m_{0} = 1, E_{s} = 2 s - 1, m_{s} = 1 + {(- 1)}^{s}$ and $s = 1, 2, \dots, 2 k - 1$ in function (2), function (2) is equal to function (1) for $r \in R$ . That is, both functions have the same mathematical form in common. We surmise that the similar method from [10] may be adopted. However, we need to find more conditions to ensure the existence and uniqueness of equilibria for a given recurrent neural network in each subset. The method from existing literature have apparently unable to realize the goal. In order to further explore existence and uniqueness condition of multiple equilibria, it is necessary to provide a new method to realize the goal.

Inspired by the above discussion, this paper is devoted to investigate the dynamical behaviors of recurrent neural networks, including the total number of equilibria, their locations and local stability. It is worth noting that the main contributions of this paper are as follows:

•
This paper provides a method to deal with multistability of RNNs with nonincreasing activation function, which the nonincreasing function converts into the nondecreasing function. Meanwhile, the number of multiple equilibria in both of them is the same.
•
Rigorous mathematical analysis of the dynamical behaviors of some equilibrium points is presented for RNNs with time-varying delays and nonincreasing activation function. It is proof that RNNs can have ${(4 k + 3)}^{n}$ equilibrium points, and ${(2 k + 2)}^{n}$ of them are locally exponentially stable.
•
In terms of the coexistence and local stability of multiple equilibrium points for RNNs, this effect by the division of state space is revealed and we arrive at an important insight: there are many conditions to ensure the coexistence of multiple equilibria.

The following sections are arranged as follows. Section 2 describes system and preliminaries. Section 3 derives sufficient conditions of equilibrium points in RNNs with nondecreasing activation function. Section 4 is extension to sufficient conditions of equilibrium points in RNNs with nonincreasing activation function. Numerical examples are presented to verify the effectiveness of our results in Section 5. Finally, the conclusions are drawn in Section 6.

Section snippets

Notations

Let $C ([t_{0} - τ, t_{0}], D)$ be the Banach space of functions mapping $[t_{0} - τ, t_{0}]$ into $D \subseteq R^{n}$ with norm defined $∥ ϕ ∥_{\infty} = \max_{1 \leq i \leq n} {\sup_{r \in [t_{0} - τ, t_{0}]} | ϕ_{i} (r) |}$ , where $ϕ (s) = {(ϕ_{1} (s), ϕ_{2} (s), \dots, ϕ_{n} (s))}^{T} \in C ([t_{0} - τ, t_{0}], D)$ . Denote $∥ x ∥_{\infty} = \max_{1 \leq i \leq n} {| x_{i} |}$ as the vector norm of the vector $x = {(x_{1}, x_{2}, \dots, x_{n})}^{T}$ .

For the given integer $k \geq 1$ and the given constant $0 < E_{s} \in R$ $s = 1, 2, \dots, 2 k + 1$ , there exist $Z_{j}^{i -}, Z_{j}^{i +} \in R$ , $j = 1, 2, \dots, 4 k + 3$ , $\forall i \in {1, 2, \dots, n}$ , such that $Z_{1}^{i -} < Z_{1}^{i +} < - E_{2 k + 1} < Z_{2}^{i -} < Z_{2}^{i +} < - E_{2 k} < \dots < - E_{1} < Z_{2 k + 2}^{i -} < Z_{2 k + 2}^{i +} < E_{1} < Z_{2 k + 3}^{i -} < Z_{2 k + 3}^{i +} < E_{2} < \dots < E_{2 k} < Z_{4 k + 2}^{i -} < Z_{4 k + 2}^{i +} < E_{2 k + 1} < Z$

Main results

In this part, we mainly consider activation function $f (r) = - g (r)$ , $\forall r \in R$ . Obviously, the function f(r) is a nondecreasing odd function in R. We will explore new multistability conditions in the general delayed neural network system. Meanwhile, the new criterions and conclusions are established.

Theorem 1

Suppose that $i = 1, 2, \dots, n$ $Q_{i} (a_{ij}, b_{ij}, u_{i}) < 3 (a_{ii} + b_{ii} - 1) (1 - k) + \frac{1}{2} {3 - | (a_{ii} + b_{ii} - 1) (4 - 6 k) + 3 |}$ hold, then for $Λ \in Ω_{2}$ , RNN $(5^{'})$ has at least one equilibrium point in Λ, and RNN $(5^{'})$ has at least ${(4 k + 3)}^{n}$ equilibrium points

Extension to RNNs

We are now beginning to consider activation function $f (r) = g (r)$ , $\forall r \in R$ . Obviously, the function f(r) is a nonincreasing odd function in R. Due to the symmetrical structure of the function f(r), similar to lemma, theorem and corollary are presented as follows, and similar proof are omitted. Let $\begin{matrix} F_{i}^{+} (x_{i}) = - x_{i} + {\bar{γ}}_{i} g (x_{i}) \\ G_{3 i}^{+} = {x_{i} | F_{i}^{+} (x_{i}) + c_{i} = 0, | c_{i} | \leq β_{i}, x_{i} \in R}} \end{matrix}$ where ${\bar{γ}}_{i} < - 1$ .

Lemma 4

For the given integer $k \geq 1$ , if $0 \leq β_{i} < - 3 ({\bar{γ}}_{i} + 1) (1 - k) + \frac{1}{2} {3 - | - ({\bar{γ}}_{i} + 1) (4 - 6 k) + 3 |}$ , then $Card (G_{3 i}^{+}) = 4 k + 3$ .

Theorem 5

Suppose that $θ_{i} \geq 0, α_{i} \geq 0, η_{i} \geq 0$ for $i = 1, 2, \dots, n$

Numerical examples

In this section, two examples are provided to verify the effectiveness of results obtained in the previous section. We will choose $D_{i 1}^{-} = {[- \frac{1}{ϵ}, - (6 k + 1) - ϵ], [- 6 q + 8 + ϵ, - 6 q + 11 - ϵ], [(6 k + 1) + ϵ, \frac{1}{ϵ}], [6 q - 11 + ϵ, 6 q - 8 - ϵ], q = 2, 3, \dots, k + 1}, D_{i 2}^{-} = {[- 1 + ϵ, 1 - ϵ]}, D_{i 3}^{-} = {[6 q - 8 + ϵ, 6 q - 5 - ϵ], [- 6 q + 5 + ϵ, - 6 q + 8 - ϵ], q = 2, 3, \dots, k + 1}$ , then $Ω_{1}^{-} = {\prod_{i = 1}^{n} l_{j (i)}^{i} | l_{j (i)}^{i} \in D_{i 1}^{-}, j (i) = 1, 3, 5, \dots, or 4 k + 3} Ω_{2}^{-} = {\prod_{i = 1}^{n} l_{j (i)}^{i} | l_{j (i)}^{i} \in D_{i 1}^{-} \cup D_{i 2}^{-} \cup D_{i 3}^{-}, j (i) = 1, 2, \dots, 4 k + 3}$ where ϵ is small enough such that $0 < ϵ ≪ \frac{1}{6 k + 1}$ .

Example 1

Consider RNN (5) with activation function (6) ${\begin{matrix} {\dot{x}}_{1} = & - x_{1} (t) + \end{matrix}$

Conclusions

In this paper, the multistability has been investigated for RNNs with time-varying delays. By combining two kinds of fixed point method, new conditions are proposed to ensure some equilibrium points are exponential stability. Since the memory mode is often designed stable equilibrium (or attractors), we can use multistability of recurrent neural networks to design associative memory and the corresponding domain of attraction. Taking method in our paper is general and less conservative compared

Fanghai Zhang received the B.S. degree in Mathematics from Fuyang Normal University, Fuyang, China in 2012. He is currently pursuing the Ph.D. degree with the School of Automation, Huazhong University of Science and Technology, Wuhan, China. His current research interests include artificial neural networks, stability analysis of dynamical systems, switching control, and associative memories.

References (31)

L. Zhang et al.
Some multistability properties of bidirectional associative memory recurrent neural networks with unsaturating piecewise linear transfer functions
Neurocomputing
(2009)
Q. Li et al.
Event-triggered $H^{\infty}$ state estimation for discrete-time stochastic genetic regulatory networks with Markovian jumping parameters and time-varying delays
Neurocomputing
(2016)
R. Samli et al.
Global robust stability analysis of uncertain neural networks with time varying delays
Neurocomputing
(2015)
L.J. Pan et al.
Stability of bidirectional associative memory neural networks with Markov switching via ergodic method and the law of large numbers
Neurocomputing
(2015)
Z.G. Zeng et al.
Memory pattern analysis of cellular neural networks
Phys. Lett. A
(2005)
G. Bao et al.
Analysis and design of associative memories based on recurrent neural network with discontinuous activation functions
Neurocomputing
(2012)
J.D. Cao et al.
Multistability and multiperiodicity of delayed Cohen–Grossberg neural networks with a general class of activation functions
Physica D: Nonlinear Phenom.
(2008)
Y.J. Huang et al.
Dynamical stability analysis of multiple equilibrium points in time-varying delayed recurrent neural networks with discontinuous activation functions
Neurocomputing
(2012)
L.L. Wang et al.
Multistability and complete convergence analysis on high-order neural networks with a class of nonsmooth activation functions
Neurocomputing
(2015)
X.B. Nie et al.
Multistability of neural networks with discontinuous non-monotonic piecewise linear activation functions and time-varying delays
Neural Netw.
(2015)

L.L. Wang et al.

Multiple u-stability of neural networks with unbounded time-varying delays

Neural Netw.

(2014)

C.Y. Cheng et al.

Multistability and convergence in delayed neural networks

Physica D: Nonlinear Phenom.

(2007)

X.B. Nie et al.

Multistability of competitive neural networks with time-varying and distributed delays

Nonlinear Anal. Real World Appl.

(2009)

M. Di Marco et al.

Necessary and sufficient condition for multistability of neural networks evolving on a closed hypercube

Neural Netw.

(2014)

L.L. Wang et al.

Coexistence and local stability of multiple equilibria in neural networks with piecewise linear nondecreasing activation functions

Neural Netw.

(2010)

Cited by (23)

Multimode function multistability for Cohen-Grossberg neural networks with mixed time delays
2022, ISA Transactions
Citation Excerpt :
So, when this type of stability is achieved, the exponential stability, polynomial stability, logarithmic stability and asymptotic stability can be achieved simultaneously. In several engineering applications of NNs, for example, associative memory and image processing, it is crucial that the designed NNs are possessed of multiple equilibria [26–38], namely, the designed NNs are multiple stable. What should be pointed out is that multiple stability analysis for NNs is quite different from monotone stability for NNs.
In this paper, we are concerned with the multimode function multistability for Cohen-Grossberg neural networks (CGNNs) with mixed time delays. It is introduced the multimode function multistability as well as its specific mathematical expression, which is a generalization of multiple exponential stability, multiple polynomial stability, multiple logarithmic stability, and asymptotic stability. Also, according to the neural network (NN) model and the maximum and minimum values of activation functions, $n$ pairs of upper and lower boundary functions are obtained. Via the locations of the zeros of the $n$ pairs of upper and lower boundary functions, the state space is divided into $\prod_{i = 1}^{n} (2 H_{i} + 1)$ parts correspondingly. By virtue of the reduction to absurdity, continuity of function, Brouwer’s fixed point theorem and Lyapunov stability theorem, the criteria for multimode function multistability are acquired. Multiple types of multistability, including multiple exponential stability, multiple polynomial stability, multiple logarithmic stability, and multiple asymptotic stability, can be achieved by selecting different types of function $P (t)$ . Two numerical examples are offered to substantiate the generality of the obtained criteria over the existing results.
Multiple asymptotic stability of fractional-order quaternion-valued neural networks with time-varying delays
2021, Neurocomputing
In this paper, the multiple asymptotic stability is investigated for fractional-order quaternion-valued neural networks (FQVNNs) with time-varying delays. The activation function is a nonmonotonic piecewise nonlinear activation function. By applying the Hamilton rules, the FQVNNs are transformed into real-valued systems. Then, according to the Brouwer’s fixed point theorem, three new conditions are proposed to ensure that there exist $3^{4 n}$ equilibrium points. Moreover, by virtue of fractional-order Razumikhin theorem and Lyapunov function, a new condition is derived to guarantee the FQVNNs have $2^{4 n}$ locally asymptotic stable equilibrium points. For the first time, the multiple asymptotic stability of delayed FQVNNs is investigated. Contrast to multistability analysis of integer-order quaternion-valued neural networks, this paper present different conclusions. Finally, two numerical simulations demonstrate the validity of the results.
Multistability and robustness of complex-valued neural networks with delays and input perturbation
2021, Neurocomputing
This paper concentrates on the multistability and robustness of complex-valued neural networks (CVNNs) with delays and input perturbation. Firstly, several criteria on the multiple $ψ$ -type stability of CVNNs with time-varying delays are obtained by virtue of $ψ$ -type function and the analytical method. Secondly, several sufficient conditions of robustness have been derived to guarantee the $ψ$ -type stability and boundedness of delayed CVNNs with input perturbation. These obtained results improve and extend the previous results. Finally, one numerical example is provided to show the effectiveness of the theoretical results.
Multistability of delayed fractional-order competitive neural networks
2021, Neural Networks
This paper is concerned with the multistability of fractional-order competitive neural networks (FCNNs) with time-varying delays. Based on the division of state space, the equilibrium points (EPs) of FCNNs are given. Several sufficient conditions and criteria are proposed to ascertain the multiple $O (t^{- α})$ -stability of delayed FCNNs. The $O (t^{- α})$ -stability is the extension of Mittag-Leffler stability of fractional-order neural networks, which contains monostability and multistability. Moreover, the attraction basins of the stable EPs of FCNNs are estimated, which shows the attraction basins of the stable EPs can be larger than the divided subsets. These conditions and criteria supplement and improve the previous results. Finally, the results are illustrated by the simulation examples.
Multistability of delayed neural networks with monotonically nondecreasing linear activation function
2021, Neurocomputing
Citation Excerpt :
These two methods are very important methods with general applicability. And lots of existing works are established on them [26–33]. Unfortunately, as far as we know, the above two methods are no longer applicable to multistability analysis of neural networks with disturbance.
This paper investigates the multistability on a class of delayed neural networks with monotonically nondecreasing linear activation function. For n state neurons with $2 m + 1$ piecewise linear activation function, we prove the neural networks have ${(2 m + 1)}^{n}$ equilibrium points, ${(m + 1)}^{n}$ of which are locally exponentially stable. Different from the traditional multistability analysis method such as fixed point theory, this paper only uses the character of activation functions, and can also handle the neural networks with disturbance term. The research results of this paper generalize the previous related research works and are easy to test. A numerical example is given to shown the effectiveness of our theoretical results.
Multistability of switched neural networks with sigmoidal activation functions under state-dependent switching
2020, Neural Networks
Citation Excerpt :
As known widely nowadays, neural networks play important roles in many technical areas, such as pattern recognition (Kwan & Cai, 1994; Suganthan, Teoh, & Mital, 1995; Zeng, Huang, & Wang, 2005), associative memory (Isokawa, Nishimura, Kamiura, & Matsui, 2008; Zeng & Wang, 2008, 2009), and other areas (Bao, Cao, K, A, & B, 2018a; Bao, Park, & Cao, 2016). In recent decades, various neural network models, including Hopfield-type neural network model (Huang & Cao, 2010; Zeng, Huang, & Zheng, 2016), cellular neural networks with memristors (Bao & Zeng, 2013; Di, Forti, & Pancioni, 2017; Duan, Hu, Dong, Wang, & Mazumder, 2015), Cohen–Grossberg neural network model (Gang, Zhang, & Dong, 2007; Wang, Liu, Li, & Liu, 2006; Wang & Zou, 2002; Zhang & Wang, 2008), switched neural network model (Bao, Cao, & Kurths, 2018b; Li & Cao, 2007; Li, Zhao, Dimirovski, & Liu, 2009; Liu, Shen, & Zhao, 2013; Zhao, Yin, & Zheng, 2016; Zhao & Zhao, 2017), have been developed and analyzed. Numerous results are available for the stability analysis of the switched systems, e.g., Branicky (1998), Chen, Long, and Fu (2010), Guo, Liu, and Wang (2018, 2019), Guo, Wang, and Yan (2014), Hu, Fei, Ma, Wu, and Geng (2015), Kahloul and Sakly (2018), Li and Cao (2007), Lian and Wang (2015), Long and Wei (2007), Long, Xiao, Chen, and Zhang (2013), Nie and Cao (2015), Niu et al. (2017), Singh and Sukavanam (2012), Song, Liu, and Wang (2018), Song, Xiang, Chen, and Hu (2006), Tang and Zhao (2017) and Yu, Fei, Yu, and Fei (2011).
This paper presents theoretical results on the multistability of switched neural networks with commonly used sigmoidal activation functions under state-dependent switching. The multistability analysis with such an activation function is difficult because state–space partition is not as straightforward as that with piecewise-linear activations. Sufficient conditions are derived for ascertaining the existence and stability of multiple equilibria. It is shown that the number of stable equilibria of an $n$ -neuron switched neural networks is up to $3^{n}$ under given conditions. In contrast to existing multistability results with piecewise-linear activation functions, the results herein are also applicable to the equilibria at switching points. Four examples are discussed to substantiate the theoretical results.

View all citing articles on Scopus

Zhigang Zeng received the Ph.D. degree in systems analysis and integration from Huazhong University of Science and Technology, Wuhan, China, in 2003. He is currently a Professor with the School of Automation, Huazhong University of Science and Technology, Wuhan, China, and also with the Key Laboratory of Image Processing and Intelligent Control of the Education Ministry of China, Wuhan, China. He has been an Associate Editor of the IEEE Transactions on Neural Networks (2010–2011), IEEE Transactions on Cybernetics (since 2014), IEEE Transactions on Fuzzy Systems (since 2016), and a member of the Editorial Board of Neural Networks (since 2012), Cognitive Computation (since 2010), Applied Soft Computing (since 2013). He has published over 100 international journal papers. His current research interests include theory of functional differential equations and differential equations with discontinuous right-hand sides, and their applications to dynamics of neural networks, memristive systems, and control systems.

^☆: This work was supported by the Key Program of National Natural Science Foundation of China under Grant 61134012, the Doctoral Program of Higher Education of China under Grant 20130142130012, the Science and Technology Support Program of Hubei Province under Grant 2015BHE013, the Program for Science and Technology in Wuhan of China under Grant 2014010101010004.

View full text

Multistability of recurrent neural networks with time-varying delays and nonincreasing activation function☆

Abstract

Introduction

Section snippets

Notations

Main results

Extension to RNNs

Numerical examples

Conclusions

Neurocomputing

Neurocomputing

Neurocomputing

Neurocomputing

Phys. Lett. A

Neurocomputing

Physica D: Nonlinear Phenom.

Neurocomputing

Neurocomputing

Neural Netw.

Neural Netw.

Physica D: Nonlinear Phenom.

Nonlinear Anal. Real World Appl.

Neural Netw.

Neural Netw.