Analysis and design of associative memories based on recurrent neural network with discontinuous activation functions

doi:10.1016/j.neucom.2011.08.026

Neurocomputing

Volume 77, Issue 1, 1 February 2012, Pages 101-107

https://doi.org/10.1016/j.neucom.2011.08.026 Get rights and content

Abstract

This paper considers a recurrent neural network (RNN) with a special class of discontinuous activation function which is piecewise constants in the state space. One sufficient condition is established to ensure that the novel recurrent neural networks can have $(4 k - 1)^{n}$ locally exponential stable equilibrium points. Such RNN is suitable for synthesizing high-capacity associative memories. The design procedure is presented with the method of singular value decomposition. Finally, the validity and performance of the results are illustrated by use of two numerical examples.

Introduction

Associative memories are brain-style devices designed to memorize a set of patterns such that the stored patterns can be retrieved with the initial probes containing sufficient information about the patterns. Associative memories include autoassociative memory and heteroassociative memory.

As models of human brains, neural networks have been used as associative memory. In the past decades, dynamics of recurrent neural network (RNN) with their various generalizations, including cellular neural network [1] (CNN) as its special case, have been extensively investigated (see, e.g., [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23] and their references) for their applications to associative memory, pattern recognition, optimization computation, etc. It has been shown that n-neuron RNN with one step (k steps) piecewise linear activation function can have 3ⁿ ( $(4 k - 1)^{n}$ ) equilibrium points and 2ⁿ ( $(2 k)^{n}$ ) points of them located in saturation regions are locally exponentially stable [9], [20].

In the application, neural network with discontinuous activation function [24] is frequently encountered. Global stability and global exponential convergence of the state and output solutions of neural network with monotone nondecreasing discontinuous activation function were discussed by Forti et al. in [25], [26], [27], [28]. It is worth to note that some conditions were obtained to ensure global convergence in finite time in [25], [28]. Wang et al. [29], [30], [31] generalized Forti's work by discharging the boundedness and monotonicity of the discontinuous activation function. Huang and Cao [7] introduced a class of neural networks with a special discontinuous activation function which has n² isolated equilibrium points which are locally exponentially stable.

Many researchers used RNN with several kinds of continuous activation functions for associative memory. Two methods have been proposed for designing associative memories based on cellular neural networks in [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52]. In the first method, RNN with multiple locally stable equilibrium points are directly regarded as associative memories [32], [33], [34], [35], [48], [49], [50], [51], [52]. In this case, the addressable memories or patterns are stored as stable equilibria or periodic orbits. While the second method is to design RNN with unique equilibrium points and ensure that each trajectory globally converges to a unique equilibrium point which depends on the external input instead of initial state [45], [46], [47]. To the best of our knowledge, RNN with discontinuous activation function is not considered as associative memory. Motivated by the above discussion, we investigate a class of recurrent neural networks with a special discontinuous activation function which is piecewise constant in the state space. One sufficient condition is obtained to ensure that the novel neural networks can have $(4 k - 1)^{n}$ locally exponentially stable equilibrium points. And then, synthesis procedure of associative memories is presented by just modifying the method described in [42].

The rest of this paper is organized as the following. In Section 2, stability analysis for associative memories is presented. Design procedure of neural network for associative memories is given in Section 3. Simulation results of two examples are shown in Section 4. Finally, the study in this paper is concluded in Section 5.

Section snippets

Stability analysis for associative memories

In this section, stability analysis of RNN with discontinuous activation function is presented. By dividing the state space into $(4 k - 1)^{n}$ subspaces, one sufficient condition is obtained to guarantee that there exists one equilibrium point in each subspace.

Synthesis of neural network

In this section, we present a design procedure for system (1) by modifying the synthesis procedure developed in [39].

For a given positive integer k, let $M = {3 - 4 k, 5 - 4 k, \dots, - 3, - 1,0, 1,3, \dots, 4 k - 5,4 k - 3}$ and $M^{n} = {ξ \in R^{n}, ξ = (ξ_{1}, ξ_{2}, \dots, ξ_{n})^{T} ξ_{i} \in M, i = 1,2, \dots, n} .$

Hence, M and Mⁿ composed of $4 k - 1$ and $(4 k - 1)^{n}$ elements, respectively.

Illustrative examples

In this section, two examples are given for illustrating the obtained results and the synthesis procedure, respectively.

Consider the following RNN with discontinuous activation function: $\{\begin{matrix} {\dot{x}}_{1} (t) = - x_{1} (t) + 0.7 f (x_{1} (t)) + 0.5 f (x_{2} (t)) + 0.5 f (x_{1} (t - 0.3)) \\ - 0.51 f (x_{2} (t - 0.3)) + 0.1; \\ {\dot{x}}_{2} (t) = - x_{2} (t) - 0.62 f (x_{1} (t)) + 0.6 f (x_{2} (t)) + 0.6 f (x_{1} (t - 0.1)) \\ + 0.7 f (x_{2} (t - 0.1)) - 0.2 . \end{matrix}$ Example 1 is almost the same as the example in [20] by just replacing the piecewise linear function $f (z) = \{\begin{matrix} 5, & z \in [5, + \infty), \\ 2 z - 5, & z \in [3,5), \\ 1, & z \in [1,3), \\ z, & z \in (- 1,1), \\ - 1, & z \in (- 3, - 1], \\ 2 z \end{matrix}$

Concluding remarks

In the present paper, we investigated a class RNN with discontinuous activation function. It has been shown that such n dimensional RNN can have $(4 k - 1)^{n}$ locally exponentially stable isolated equilibrium points. The local number of local exponentially stable equilibrium points is more than that of the existing neural network model. Multi-stability in recurrent neural networks is an crucial issue when it is used to associative memories. Increasing storage capacity is a fundamental problem, hence,

Acknowledgments

The authors would like to thank the associate editor and the referees for their detailed comments and valuable suggestions which considerably improved the presentation of the paper. The work is supported by the Natural Science Foundation of China under Grant 60974021, the 973 Program of China under Grant 2011CB710606, the Fund for Distinguished Yong Scholars of Hubei Province under Grant 2010CDA081, the Specialized Research Fund for the Doctoral Program of Higher Education of China under Grant

Gang Bao received the B.S. degree in mathematics from Hubei Normal University, Huangshi, China, the M.S. degree in applied mathematics from Beijing University of Technology, Beijing, China, in 2000 and 2004, respectively. At present, he is working hard for Ph.D. degree Department of Control Science and Engineering, Huazhong University of Science and Technology, and Image Processing and Intelligent Control Key Laboratory of Education Ministry of China, Wuhan, Hubei 430074, China.

References (54)

L. Zhang et al.
Some multistability properties of bidirectional associative memory recurrent neural networks with unsaturating piecewise linear transfer functions
Neurocomputing
(2009)
Q.K. Song
Exponential stability of recurrent neural networks with both time-varying delays and general activation functions via LMI approach
Neurocomputing
(2008)
C.D. Zheng et al.
Less conservative results of state estimation for delayed neural networks with fewer LMI variables
Neurocomputing
(2011)
G. Huang et al.
Multi-stability of neural networks with discontinuous activation function
Commun. Nonlinear Sci. Numer. Simulation
(2008)
C.D. Zheng et al.
Novel delay-dependent criteria for global robust exponential stability of delayed cellular neural networks with norm-bounded uncertainties
Neurocomputing
(2009)
O.M. Kwon et al.
A new augmented Lyapunov–Krasovski functional approach to exponential passivity for neural networks with time-varying delays
Appl. Math. Comput.
(2011)
D.H. Ji et al.
Passivity-based control for Hopfield neural networks using convex representation
Appl. Math. Comput.
(2011)
S.M. Lee et al.
A novel delay-dependent criterion for delayed neural networks of neural type
Phys. Lett. A
(2010)
M. Forti et al.
Generalized Lyapunov approach for convergence of neural networks with discontinuous or nonLipschitz activations
Physica D
(2006)
L.P. Li et al.
Dynamical behaviors of a class of recurrent neural networks with discontinuous neuron activations
Appl. Math. Modell.
(2009)

M. Gonzlez et al.

Block attractor in spatially organized neural networks

Neurocomputing

(2009)

C.Y. Cheng et al.

Multistability in recurrent neural networks

SIAM J. Appl. Math.

(2006)

Cited by (92)

Analysis of Th1/Th2 response pattern with Treg cell inhibition and stochastic effect
2021, Chaos, Solitons and Fractals
T cells differentiate into Th1 or Th2 cells upon maturation to influence different patterns of the immune response. Th1 and Th2 cells regulate each other and their responses are inhibited by Treg cells. With noisy external stimulation, Th1/Th2 cell differentiation can be dynamically balanced. The underlying mechanisms of Th cell differentiation under Treg cell inhibition and the extrinsic noise effects are not yet completely understood. In this paper, a mathematical model of the interactions between Th1 and Th2 cells with Treg cell inhibition and stochastic effects is developed to study the preference of outcomes and the noise-induced hopping among different states. First, we provide the conditions for different asymptotic phases of Th1 and Th2 responses under Treg cell regulation. Numerical simulations are applied to calculate the switching probability and the mean residence time to study how the noise affects the attractiveness of different states. Our results support that due to the stronger inhibitory effect of Treg cells on Th1 cell development, the high-Th2-low-Th1 state is more attractive under small noise effects. Additionally, we show that the attractiveness of the states is affected mainly by the extrinsic noise in Th2 cell signaling.
Multistability of Hopfield neural networks with a designed discontinuous sawtooth-type activation function
2021, Neurocomputing
In this paper, a class of discontinuous sawtooth-type activation function is designed and the multistability of Hopfield neural networks (HNNs) with such kind of activation function is studied. By virtue of the Brouwer’s fixed point theorem and the property of strictly diagonally dominant matrix (SDDM), some sufficient conditions are presented to ensure that the $n$ -neuron HNN can have at least $7^{n}$ equilibria, among which $4^{n}$ equilibria are locally exponentially stable and the remaining $7^{n} - 4^{n}$ equilibria are unstable. Then, the obtained results are extended to a more general case. We continue to increase the number of the peaks of the sawtooth-type activation function and we find that the $n$ -neuron HNN can have ${(2 k + 3)}^{n}$ equilibria, ${(k + 2)}^{n}$ of them are locally exponentially stable and the remaining equilibria are unstable. Therein, $k$ denotes the total number of the peaks in the designed activation function. That is to say, there is a quantitative relationship between the number of the peaks and the number of the equilibria. It implies that one can improve the storage capacity of a HNN by increasing the number of the peaks of the activation function in theory and in practice. To some extent, this method is convenient and flexible. Compared with the existing results, HNN with the designed sawtooth-type activation function can have more total equilibria as well as more locally stable equilibria. Finally, two examples are presented to demonstrate the validity of the obtained results.
Multistability of state-dependent switching neural networks with discontinuous nonmonotonic piecewise linear activation functions
2021, Neurocomputing
This paper presents the theoretical results on the multistability of state-dependent switching neural networks with discontinuous nonmonotonic piecewise linear activation functions. For n-neurons switching model, this paper shows that neural networks have 7ⁿ equilibrium points, 6ⁿ of which are located at the continuous points of activation functions and others are located at the discontinuous points of activation functions. Among these equilibrium points, 4ⁿ or 5ⁿ are stable and others are unstable, which depend on the relationship between the switching threshold and the discontinuous points of the activation functions. Compared with existing results, this paper reveals that switching threshold and discontinuous character are crucial in increasing the number of equilibrium points. Two examples are presented to verify the theoretical results.
Multistability of delayed neural networks with monotonically nondecreasing linear activation function
2021, Neurocomputing
This paper investigates the multistability on a class of delayed neural networks with monotonically nondecreasing linear activation function. For n state neurons with $2 m + 1$ piecewise linear activation function, we prove the neural networks have ${(2 m + 1)}^{n}$ equilibrium points, ${(m + 1)}^{n}$ of which are locally exponentially stable. Different from the traditional multistability analysis method such as fixed point theory, this paper only uses the character of activation functions, and can also handle the neural networks with disturbance term. The research results of this paper generalize the previous related research works and are easy to test. A numerical example is given to shown the effectiveness of our theoretical results.
A new fixed-time stability theorem and its application to the fixed-time synchronization of neural networks
2020, Neural Networks
In this paper, we derive a new fixed-time stability theorem based on definite integral, variable substitution and some inequality techniques. The fixed-time stability criterion and the upper bound estimate formula for the settling time are different from those in the existing fixed-time stability theorems. Based on the new fixed-time stability theorem, the fixed-time synchronization of neural networks is investigated by designing feedback controller, and sufficient conditions are derived to guarantee the fixed-time synchronization of neural networks. To show the usability and superiority of the obtained theoretical results, we propose a secure communication scheme based on the fixed-time synchronization of neural networks. Numerical simulations illustrate that the new upper bound estimate formula for the settling time is much tighter than those in the existing fixed-time stability theorems. Moreover, the plaintext signals can be recovered according to the new fixed-time stability theorem, while the plaintext signals cannot be recovered according to the existing fixed-time stability theorems.
A multi-layer memristive recurrent neural network for solving static and dynamic image associative memory
2019, Neurocomputing
Citation Excerpt :
At the same time, a RNN based on memory can be constructed by using memristive neuron synaptic connections [23]. The literatures [24–26] use different activation functions to demonstrate its stability by derivation and experimental simulation. Therefore, the activation function is the core of the RNN associative memory.
Recent years have seen increased attention being given to recurrent neural networks in associative memory applications. The activation function is the core of the recurrent neural network for associative memory, but its hardware implementation is quite complicated. A novel multi-layer memristive recurrent neural network (MMRNN) is proposed, in which nano-scaled element memristor is employed as activation function. Its stability is proved by numerical analysis, which provides the theoretical premise for its application. There are three advantages: (i) Since the memristor is a passive device, the activation function circuit will be low power consumption; (ii) It can not only simplify the hardware circuit, but also facilitate the implementation of VLSI; (iii) MMRNN can realize the associative memory of both static and dynamic images. A series of theoretical analysis and numerical simulations demonstrated the effectiveness of the proposed MMRNN. The study helps to explore the dynamic image associative memory, and provides a new effective way to realize the associative memory of video.

View all citing articles on Scopus

Zhigang Zeng received the B.S. degree in mathematics from Hubei Normal University, Huangshi, China, in 1993, the M.S. degree in ecological mathematics from Hubei University, Wuhan, China, in 1996, and the Ph.D. degree in systems analysis and integration from Huazhong University of Science and Technology, Wuhan, China, in 2003.

Currently, he is a Professor at the Department of Control Science and Engineering, Huazhong University of Science and Technology. His research interests include neural networks, switched systems, computational intelligence, stability analysis of dynamical systems, pattern recognition, and associative memories.

View full text

Analysis and design of associative memories based on recurrent neural network with discontinuous activation functions

Abstract

Introduction

Section snippets

Stability analysis for associative memories

Synthesis of neural network

Illustrative examples

Concluding remarks

Acknowledgments

Neurocomputing

Neurocomputing

Neurocomputing

Commun. Nonlinear Sci. Numer. Simulation

Neurocomputing

Appl. Math. Comput.

Appl. Math. Comput.

Phys. Lett. A

Physica D

Appl. Math. Modell.

Neurocomputing

Neural Networks

Neurocomputing

Cellular neural networks: theory

IEEE Trans. Circuits Syst.

Multistability and new attraction basins of almost-periodic solutions of delayed neural networks

IEEE Trans. Neural Network

Multistability analysis for recurrent neural networks with unsaturating piecewise linear transfer functions

Neural Comput.

Stability analysis of delayed cellular neural networks described using cloning templates

IEEE Trans. Circuits Syst. I

Global asymptotic stability of recurrent neural networks with multiple time varying delays

IEEE Trans. Neural Network

Robust stability analysis for interval Cohen–Grossberg neural networks with unknown time varying delays

IEEE Trans. Neural Network

Global asymptotic stability and robust stability of a general class of Cohen–Grossberg neural networks with mixed delays

IIEEE Trans. Circuits Syst. I

Activity invariant sets and exponentially stable attractors of linear threshold discrete-time recurrent neural networks

IEEE Trans. Autom. Control

Nontrivial global attractors in 2-D multistable attractor neural networks

IEEE Trans. Neural Network

Cellular neural networks with output function having multiple constant regions

IEEE Trans. Circuits Syst. I

Multiperiodicity and exponential attractivity evoked by periodic external inputs in delayed cellular neural networks

Neural Comput.

Multiperiodicity and attractivity of delayed recurrent neural networks with unsaturating piecewise linear transfer functions

IEEE Trans. Neural Network

Block attractor in spatially organized neural networks

Neurocomputing

Multistability in recurrent neural networks

SIAM J. Appl. Math.