Multistability of state-dependent switching neural networks with discontinuous nonmonotonic piecewise linear activation functions

doi:10.1016/j.neucom.2021.01.046

Neurocomputing

Volume 437, 21 May 2021, Pages 300-311

https://doi.org/10.1016/j.neucom.2021.01.046 Get rights and content

Abstract

This paper presents the theoretical results on the multistability of state-dependent switching neural networks with discontinuous nonmonotonic piecewise linear activation functions. For n-neurons switching model, this paper shows that neural networks have 7ⁿ equilibrium points, 6ⁿ of which are located at the continuous points of activation functions and others are located at the discontinuous points of activation functions. Among these equilibrium points, 4ⁿ or 5ⁿ are stable and others are unstable, which depend on the relationship between the switching threshold and the discontinuous points of the activation functions. Compared with existing results, this paper reveals that switching threshold and discontinuous character are crucial in increasing the number of equilibrium points. Two examples are presented to verify the theoretical results.

Introduction

In recent years, neural networks are of significant importance due to their wide applications throughout science and engineering [1], [2]. Existing classical neural network models, including cellular neural network model, Hopfield-type neural network model, switching neural network model, have been introduced and applied in practice, such as associative memory [3], [4], [5], [6] and pattern classification [7], [8], [9]. The dynamic analysis plays a vital part in applications and designs of neural networks.

When applying neural networks in associative memory and pattern recognition, multiple stable equilibria is indispensable. Multistability has a strong correlation with storage capacity of neural networks. There are many differences between multistability analysis and monostability analysis. For the monostability analysis, we should pay attention to the global stability and uniqueness of equilibrium point [10], [11]. By comparison, for multistability analysis, the number of equilibria, multistability and domains of those equilibria should be investigated. On this account, many related results have been proposed [12], [13], [14], [15], [16], [17], [18], [19].

What is noteworthy is that multistability analysis mostly relies on the types of activation functions. In multistability analysis, the activation functions are mainly concentrated on nondecreasing saturated activation functions, sigmoid activation functions, Maxican–Hat-type activation functions and so on. In the existing works, it should be remarkable that the research on multistability basically concentrates on neural networks with continuous activation functions [20], [21], [22], [23], [24], [25], [26]. When handling dynamical systems with very high-slope nonlinear elements, we often to model them using discontinuous right sides, rather than using high slopes with limited values [27], [28], [29]. Thus, how to extend these results to the discontinuous activation functions has naturally become an important research topic.

When the system receives an external signal or the system operation meets certain conditions, the connection weight may vanish or change [30], [31]. To describe this switching phenomenon in the networks, switching system is introduced. According to the switching modes, we can divide switching systems into two categories [32], [33]. One is state-dependent switching system, namely the switching rule is determined by state. The other is time-dependent switching system, the corresponding switching rule is controlled by time. Compared with time-dependent switching system, state-dependent switching system may take on different initial values according to different initial status. The dynamical analysis of state-dependent switching is more complicated than that of conventional ones. Due to the great value of switching systems in both theory and practice, stability analysis of switching systems has been widely studied [34], [35], [36], [37], [38], [39], [40], [41].

Despite the stability of switching neural networks has been extensively studied, for all we know, the multistability of switching neural networks, especially state-dependent switching neural networks has been rarely investigated. The previous research mostly concentrated on continuous activation functions, so it cannot be directly applied to discontinuous activation functions. As a consequence, it is essential for us to study the multistability of discontinuous nonmonotonic piecewise linear activations.

Based on the above considerations, this paper discusses the multistability of state-dependent switching neural networks with discontinuous nonmonotonic piecewise linear activation functions. Compared with related existing works, the main contributions of this paper are as follows:

(1) For n-neurons state-dependent switching neural networks with discontinuous activation functions, this paper presents it have $7^{n}$ equilibrium points, $6^{n}$ of which are located at the continuous points of activation functions and $7^{n} - 6^{n}$ of which are located at the discontinuous points of activation functions. Among these equilibrium points, $4^{n}$ or $5^{n}$ are stable and others are unstable, which depend on the relationship between the switching threshold and the discontinuous points of activation functions.
(2) Compared with multistability of general discontinuous nonmonotonic activation functions [37], the total number of equilibrium points increased to $7^{n}$ from $5^{n}$ , and the number of stable equilibrium points increased to $4^{n}$ or $5^{n}$ from $3^{n}$ . Compared with multistability of state-dependent switching neural networks with piecewise-linear radial basis functions [35], we get the similar conclusions as above. It reveals that switching threshold and discontinuity can increase the number of equilibrium points.

The remaining parts of this paper are organized as follows: In Section 2, a state-dependent switching neural network model with discontinuous nonmonotonic activation functions is proposed. Then, some definitions and lemmas are presented. In Section 3, the multistability results of switching neural systems are addressed. In Section 4, two examples are given to substantiate the theoretical results. In Section 5, conclusion is given.

Section snippets

Preliminaries

Consider the state-dependent switching neural network model as follows: ${\dot{x}}_{i} (t) = - d_{i} (x_{i} (t)) x_{i} (t) + \sum_{j = 1}^{n} a_{ij} (x_{i} (t)) f_{j} (x_{j} (t)) + I_{i},$ where $i = 1, \dots, n, x (t) = {(x_{1} (t), \dots, x_{n} (t))}^{T} \in R^{n}$ is the state vector; $D (x) = diag$ $(d_{1} (x_{1} (t)), \dots, d_{n} (x_{n} (t)))$ is the neurons self-feedback connection weights matrix with $d_{i} > 0$ , $A = {[a_{ij} (x_{i} (t))]}_{n \times n}$ is the inner-neuron connection weight matrix; $I = {(I_{1}, \dots, I_{n})}^{T} \in R^{n}$ is the input or bias vector, and $f (x) = {(f_{1} (x_{1}), \dots, f_{n} (x_{n}))}^{T}$ is the activation function.

In this paper, a class of discontinuous nonmonotonic

Main results

In this section, the multistability of state-dependent switching neural networks with discontinuous nonmonotonic piecewise linear activation functions is analysed. Since $x = 2$ is the discontinuous point of the activation function, for switching threshold $ξ$ , there are two main situations for us to discuss: Situation $A : 0 < ξ < 2$ , Situation $B : ξ > 2$ . ( $ξ = 2$ can be seen in Remark 3).

For Situation $A : 0 < ξ < 2$ , in the continuous points of activation function (3), we have the following conclusions.

Theorem 1

If the following

Numerical simulations

In this section, two illustrative examples are given to verified the theoretical results. Neural network model (1) with two neurons is described by: $\{\begin{matrix} \dot{x_{1}} (t) = & - d_{1} (x_{1} (t)) x_{1} (t) + a_{11} (x_{1} (t)) f_{1} (x_{1} (t)) \\ + a_{12} (x_{1} (t)) f_{2} (x_{2} (t)) + I_{1}, \\ \dot{x_{2}} (t) = & - d_{2} (x_{2} (t)) x_{2} (t) + a_{21} (x_{2} (t)) f_{1} (x_{1} (t)) \\ + a_{2} (x_{2} (t)) f_{2} (x_{2} (t)) + I_{2}, \end{matrix}$ where the activation function is defined as (3) that is $f_{1} (x) = f_{2} (x) = f (x)$ .

First, Let us consider the Situation A, that is the switching threshold $ξ \in (0, 2)$ .

Example 1

Neural networks (44) with the input term $I_{1} = 3 / 2, I_{2} = 3$ .

\begin{matrix} d_{1} (t) = \{\begin{matrix} 1 / 2, | x_{1} (t \end{matrix} \end{matrix}

Conclusion

This paper study the multistability of a state-dependent switching neural network model with discontinuous nonmonotonic piecewise linear activation function. By analyzing the discontinuous points and switching threshold, sufficient conditions are derived for ascertaining the coexistence and stability of equilibrium points. We can conclude that switching threshold and discontinuity can increase the number of equilibrium points. In addition, the relationship between the discontinuous points of

CRediT authorship contribution statement

Jiahui Zhang: Conceptualization, Methodology, Software, Writing - original draft, Visualization, Validation, Writing - review & editing. Song Zhu: Conceptualization, Methodology, Writing - review & editing. Nannan Lu: Software. Shiping Wen: Conceptualization, Methodology, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 61873271, 62006233, the Fundamental Research Funds for the Central Universities 2018XKQYMS15 and the Double-First-Rate Special Fund for Construction of China University of Mining and Technology No. 2018ZZCX14.

Jiahui Zhang was born in 1993. She received the B.Admin. degree in information management and information system from Chang’an University, Xi’an, China, in 2015. She is currently pursuing the M.S. degree in operational research and cybernetics with the China University of Mining and Technology, Xuzhou, China. She current research interests include neural networks and stochastic control.

References (41)

K. Hunt et al.
Neural networks for control systems—a survey
Automatica
(1992)
Z. Zeng et al.
Associative memories based on continuous-time cellular neural networks designed using space-invariant cloning templates
Neural Netw.
(2009)
P. Suganthan et al.
Pattern recognition by homomorphic graph matching using Hopfield neural networks
Image Vision Comput.
(1995)
Z. Zeng et al.
Memory pattern analysis of cellular neural networks
Phys. Lett. A
(2005)
G. Zhang et al.
Global exponential stability of a class of memristor-based recurrent neural networks with time-varying delays
Neurocomputing
(2012)
R. Rakkiyappan et al.
Multiple μ)stability analysis of complex-valued neural networks with unbounded time-varying delays
Neurocomputing
(2015)
F. Zhang et al.
Multistability and instability analysis of recurrent neural networks with time-varying delays
Neural Netw.
(2018)
P. Wan et al.
Multistability and attraction basins of discrete-time neural networks with nonmonotonic piecewise linear activation functions
Neural Netw.
(2020)
L. Wang et al.
Coexistence and local stability of multiple equilibria in neural networks with piecewise linear nondecreasing activation functions
Neural Netw.
(2010)
Z. Guo et al.
Multistability of switched neural networks with sigmoidal activation functions under state-dependent switching
Neural Netw.
(2020)

G. Bao et al.

Analysis and design of associative memories based on recurrent neural network with discontinuous activation functions

Neurocomputing

(2012)

J. Liang et al.

Multistability of complex-valued neural networks with discontinuous activation functions

Neural Netw.

(2016)

J. Zhao et al.

Synchronization of complex dynamical networks with switching topology: a switched system point of view

Automatica

(2009)

C. Long et al.

Novel results on finite-time stabilization of state-based switched chaotic inertial neural networks with distributed delays

Neural Netw.

(2020)

J. Cai et al.

Decentralized event-triggered control for interconnected systems with unknown disturbances

J. Franklin Inst.

(2020)

L. Chua et al.

Cellular neural networks: applications

IEEE Trans. Circuits Syst.

(1988)

Z. Zeng et al.

design and analysis of high-capacity associative memories based on a class of discrete-time recurrent neural networks

IEEE Trans. Syst. Man Cybern.

(2008)

G. Bao et al.

Memory analysis for memristors and memristive recurrent neural networks

IEEE/CAA J. Automatica Sin.

(2020)

B. Hu et al.

Multistability of delayed hybrid impulsive neural networks with application to associative memories

IEEE Trans. Neural Netw. Learn Syst.

(2019)

S. Lee et al.

A new recurrent neural-network architecture for visual pattern recognition

IEEE Trans. Neural Netw.

(1997)

Cited by (10)

Multistability analysis of complex-valued recurrent neural networks with sine and cosine activation functions
2024, Neurocomputing
This paper is devoted to the multistability problem for complex-valued recurrent neural networks (CVRNNs) with a specific class of piecewise nonlinear activation functions. Firstly, a general class of piecewise nonlinear activation functions is presented to facilitate further analysis. Then, by resorting to the fixed point theorem and Lagrange’s mean value theorem, sufficient criteria are established which ascertain that the existence of ${(2 k + 1)}^{2 n}$ equilibrium points. Meanwhile, it also verifies that ${(k + 1)}^{2 n}$ equilibrium points of the considered CVRNNs with a piecewise nonlinear activation function are stable, where $k$ stands for a positive real number, which is associated with the frequency of sine and cosine functions. Moreover, the utilization of the activation functions provides a larger storage capacity in associative memory application. In the end of paper, three numerical examples are presented to demonstrate the feasibility and validity of the achieved theoretical results.
Passivity and robust passivity of impulsive inertial neural networks with proportional delays under the non-reduced order approach
2024, Neurocomputing
This paper primarily addresses the problems of passivity and robust passivity of impulsive inertial neural networks (IINNs) with proportional delays. By constructing the Lyapunov functional directly on the original system using the non-reduced order approach, some passivity and robust passivity criteria for IINNs are addressed. In comparison with the order reduction method utilized in the existing articles, the non-reduced order method is more in line with real requirements and can better analyze the dynamic behavior of inertial neural networks (INNs). Meanwhile, the results in this paper are all in algebraic form, which are easy to verify. To demonstrate the effectiveness of the results derived, numerical examples are presented at the end.
Coexistence and local stability of multiple equilibrium points for fractional-order state-dependent switched competitive neural networks with time-varying delays
2023, Neural Networks
This paper investigates the coexistence and local stability of multiple equilibrium points for a class of competitive neural networks with sigmoidal activation functions and time-varying delays, in which fractional-order derivative and state-dependent switching are involved at the same time. Some novel criteria are established to ensure that such $n$ -neuron neural networks can have $5^{m_{1}} \cdot 3^{m_{2}}$ total equilibrium points and $3^{m_{1}} \cdot 2^{m_{2}}$ locally stable equilibrium points with $m_{1} + m_{2} = n$ , based on the fixed-point theorem, the definition of equilibrium point in the sense of Filippov, the theory of fractional-order differential equation and Lyapunov function method. The investigation implies that the competitive neural networks with switching can possess greater storage capacity than the ones without switching. Moreover, the obtained results include the multistability results of both fractional-order switched Hopfield neural networks and integer-order switched Hopfield neural networks as special cases, thus generalizing and improving some existing works. Finally, four numerical examples are presented to substantiate the effectiveness of the theoretical analysis.
Finite-time dissipative control for bidirectional associative memory neural networks with state-dependent switching and time-varying delays
2022, Knowledge-Based Systems
Citation Excerpt :
Recently there have been a number of researchers working on the dynamic behavior of BAMNNs, and significant findings have been obtained in stability analysis, synchronization control, and other dynamical behaviors (see [9–11] and references therein). Switching systems, as a special type of hybrid system, have been an important subject of engineering and physical research [12–15], widely used in robot control, automotive industry, aircraft control and other applications. It is well known that switching systems can be divided into two categories under different switching modes.
This paper focuses on finite-time exponential dissipative analysis and control problems for bidirectional associative memory neural networks (BAMNNs) with state-dependent switching and time-varying delays. Firstly, the state-dependent switching parameters of BAMNNs are interpreted as interval parameters by the interval matrix method instead of differential inclusion theory and set-value map. Under the framework of Filippov’s solution and differential inclusions, some sufficient conditions of finite-time bounded (FTB) and finite-time exponential $(Q, S, R) - γ$ dissipative (FTED) for BAMNNs are obtained based on Lyapunov–Krasovskii functional (LKF) and some inequality techniques. Then, the finite-time dissipative PI controllers are designed by solving linear matrix inequality (LMI). Finally, a numerical example is given to illustrate the correctness of the proposed results and the effectiveness of the designed controllers.
THREE DECADES OF ACTIVATIONS: A COMPREHENSIVE SURVEY OF 400 ACTIVATION FUNCTIONS FOR NEURAL NETWORKS
2024, arXiv
Almost periodic solutions of memristive multidirectional associative memory neural networks with mixed time delays
2024, International Journal of Biomathematics

View all citing articles on Scopus

Song Zhu was born in 1982. He received the B.S. degree in mathematics from Jiangsu Normal University, Xuzhou, China, in 2004, and the M.S. degree in probability and mathematical statistics and the Ph.D. degree in system engineering from the Huazhong University of Science and Technology, Wuhan, China, in 2007 and 2010, respectively. He is currently a Professor with the School of Mathematics, China University of Mining and Technology, Xuzhou. He has published over 60 international journal papers. His current research interests include neural networks and stochastic control.

Nannan Lu received the B.E. degree in electronic science and technology from PLA Information Engineering University, China, in 2006, the M.E. degree in communication and information system from China University of Mining and Technology (CUMT) in 2009, and the Ph. D. degree from Waseda University in 2013. Currently, she is a Lecturer with the School of Information and Control Engineering, CUMT. Her research interests include transfer learning, deep learning and sensor data fusion.

Shiping Wen received the M.Eng. degree in control science and engineering from the School of Automation, Wuhan University of Technology, Wuhan, China, in 2010, and the Ph.D. degree in control science and engineering from the School of Automation, Huazhong University of Science and Technology, Wuhan, in 2013. He is currently a Professor with the Australian AI Institute, Faculty of Engineering and Information Technology, University of Technology Sydney, Ultimo, NSW, Australia. His current research interests include memristor-based circuits and systems, neural networks, and deep learning. Dr. Wen serves as an Associate Editor for the IEEE Access and Neural Processing Letters.

View full text

Multistability of state-dependent switching neural networks with discontinuous nonmonotonic piecewise linear activation functions

Abstract

Introduction

Section snippets

Preliminaries

Main results

Numerical simulations

Conclusion

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgments

Automatica

Neural Netw.

Image Vision Comput.

Phys. Lett. A

Neurocomputing

Neurocomputing

Neural Netw.

Neural Netw.

Neural Netw.

Neural Netw.

Neurocomputing

Neural Netw.

Automatica

Neural Netw.

J. Franklin Inst.

Cellular neural networks: applications

IEEE Trans. Circuits Syst.

design and analysis of high-capacity associative memories based on a class of discrete-time recurrent neural networks

IEEE Trans. Syst. Man Cybern.

Memory analysis for memristors and memristive recurrent neural networks

IEEE/CAA J. Automatica Sin.

Multistability of delayed hybrid impulsive neural networks with application to associative memories

IEEE Trans. Neural Netw. Learn Syst.

A new recurrent neural-network architecture for visual pattern recognition

IEEE Trans. Neural Netw.