Elsevier

Neurocomputing

Volume 186, 19 April 2016, Pages 44-53
Neurocomputing

A unified associative memory model based on external inputs of continuous recurrent neural networks

https://doi.org/10.1016/j.neucom.2015.12.079Get rights and content

Abstract

A unified associative memory model with a novel method for designing associative memories is presented in this paper. Based on continuous recurrent neural networks, bipolar patterns inputted from external can cause the output of neural networks to be memorized patterns. In the method, two conditions relevant to external inputs are derived to ensure the network states converge to a stable interval, and an exponential stable criterion is proposed for the network being a bipolar associative memory with higher recall speed. By introducing a tunable slope activation function and considering time delay, the proposed model is general and can recall the memorized patterns in auto-associative and hetero-associative way, while higher robust and more flexible memory can be obtained through the proposed method. Experimental verification demonstrates the effectiveness and generalization of the proposed method.

Introduction

Recently, a number of research papers have dealt with dynamical systems, such as neural networks and fuzzy control systems [1], [2], [3], [4], [5], [6]. Feedback control is often taken to fuzzy control systems to guarantee the stability of the systems, and controllers are often designed to track reference signals [4], [5], [6], [7], [8], [9]. There has been increasing interest in neural networks and fuzzy control systems, and their potential applications in many areas such as classification, object tracking [7], [8], [9], signal processing [10], [11], [12], pattern recognition, associative memory [13], [14], [15], [16], [17], [18], [19], [20], [21], [22].

Associative memories are content-addressable memories, which are often considered as brain-like devices for memorizing prototype patterns, such that the prototype patterns can be recalled with the retrieval patterns that contain information about the contents of the prototype patterns. In associative memory processes, the prototype patterns are associated with their retrieval patterns internally in a fault-tolerant and robust way under different conditions. Based on this observation, data-driven approaches are often used to solve such issues, which are receiving considerably increasing attention nowadays [23], [24], and can improve process monitoring performance in industrial processes monitoring [25], [26], [27]. In [25], a data-driven fault diagnosis method is proposed for vehicle suspension systems, and handled the function attenuation of springs only by adopting the measurements of accelerometers fixed on the four corners of vehicle suspensions. In [26], a data-driven is presented for the task of fault detection in nonlinear systems, and locally weighted projection regression (LWPR) is employed to serve as a powerful tool for modeling the nonlinear process with locally linear models. In [27], an improved partial least squares (PLS) approach is proposed to decompose the measurable process variables into the key performance indicator (KPI)-related and unrelated parts, respectively.

A unified associative memory based on neural networks is presented in this paper, in which data-driven method is applied and neural networks are regarded as tools for approximating the memorizing prototype patterns under different retrieval patterns with fault or perturbation. The associative memory is implemented based on the adjustment of parameters in neural networks, and it is with potential applications in the fields of fault diagnosis, image restoration and communication security.

In the past few decades, many of studies have focused on associative memories based on neural networks [13], [14], [15], [16], [17], [18], [19], [20], [21], [22]. The stability of recurrent neural networks is an important issue for associative memories and has been studied extensively [28], [29], [30], [31], [32], [33]. Among electronic units of analog neural networks, time-delay is not avoidable due to the signal transmission, and it may cause instability and oscillation of a neural network. The researches on the stability analysis of recurrent neural networks with time delay have raised researchers’ interests [34], [35], [36], [37], [38], [39]. Therefore, it should be a significant topic when the stability analysis of recurrent neural networks with time delay applied to associative memory.

In general, there are two ways in designing associative memories based on recurrent neural networks. One tries to obtain proper weights for making the networks’ states be stable at some equilibriums that are regarded as memorized points. Hebb learning rules, as an earliest learning method, has low memory capacity and may bring some spurious equilibrium. In order to overcome these shortcomings, some other methods appeared, such as Gram–Schmidt orthogonalization method, pseudo inverse method and Sparse encoding method [14], [15], [16], Gram–Schmidt orthogonalization and pseudo inverse methods may cause lower robust and smaller domain of attraction. Sparse encoding method may occupy a lot of resources compared with non-sparse encoding method, although it can cause higher capacity. Through this way, the weights are often dependent on the initial states of the networks and may bring some spurious equilibrium [40], [41], [42]. To deal with the spurious equilibriums, the study on multi-stability of recurrent neural networks appears in the literature [43], [44], [45], [46], [47] and is still in need of further exploration. In order to overcome this deficiency, another way is brought forth recently, which tries to memorize the external inputs of recurrent neural networks as memorized targets, and they will be designed as the global equilibriums of the networks. In this way, it is expected that the spurious equilibriums could be avoided and the initial states of neural networks could be given randomly.

As for the second way, many results are obtained for designing associative memories based on recurrent neural networks [17], [18], [19], [20], [21], [22]. In [17], [18], discrete-time cellular neural networks (CNNs) are applied to design associative memory for memorizing external inputs, and some globally stability criteria are derived. In [19], a novel method is presented for designing associative memories based on discrete recurrent neural networks to accurately memorize the networks׳ external inputs. In the method, a generalized model is proposed for bipolar auto-associative memory and establishing an exponential stable criteria of the networks. In [20], two new design procedures for auto-associative memory and hetero-associative memory are separately given based on external inputs of continuous recurrent neural networks, and some sufficient criteria are established to guarantee the global exponential stability for recurrent neural networks with mixed delays. In [21], with continuous-time CNNs, associative memories are designed based on some stability criteria, and the criteria put some extra constraints on the relationship among parameters of CNNs, in which the initial values of neuron states are fixed to zero. In [22], based on continuous-time CNNs with time delays, associative memories are designed by solving a set of linear inequalities with few parameters and retrieval targets fed from external inputs, and making the CNNs guaranteed to be globally exponentially stable, in which the self-regulating parameter is restricted to be 1.

Recently, researches have been drawing more and more attention on the associative memories that could store a set of external input patterns as stable states of recurrent neural networks [19], [20], [21], [22]. However, the research results have by now the following shortcomings:

  • 1.

    There is a lack of unified model for the design of bipolar associative memory with continuous recurrent neural networks.

  • 2.

    As for continuous recurrent neural networks, the distance between memory point and the stable equilibrium is not fully used in designing associative memories due to fixed unity gain activation function. Moreover, when the activation function is disturbed, it is difficult to ensure the recall of memorized patterns.

  • 3.

    The relationship between the robustness and the self-regulating parameter of the continuous recurrent neural networks is not considered in designing associative memories.

With the inspiration of the methods proposed in [19], [20], [21], [22] and the motivation of overcoming the above shortcomings, we propose in this paper a unified associative memory model, namely a general continuous recurrent neural network with time delay and a tunable slope activation function. To the best of our knowledge, our proposed model is original in the establishment of new stable criteria and the introduction of a tunable activation function. The main contributions of this paper are listed as follows:

  • 1.

    A unified bipolar associative memory model based on continuous recurrent neural networks is proposed to design auto-associative memory and hetero-associative memory, which is different from the work of [19] that only discusses on auto-associative memory based on discrete recurrent neural networks.

  • 2.

    A new method is presented for designing hetero-associative memory, which makes the design process of neural networks simpler and more applicable than the method proposed in [20].

  • 3.

    Global exponential stable criterion of the proposed model is established which is helpful for reducing computational complexity of associative memory.

  • 4.

    The model has no limit on initial state of the networks, while [21] requires the initial state should be zero. And a self-regulating parameter is introduced to improve the performance of the designed associative memory, which is not considered in other existing models like [22].

The remainder of the paper is arranged as follows. In Section 2, problem description is given with some definitions and preliminaries used in later sections. Section 3 presents our method for solving the problem with lemmas, theorems and procedures. Then, in Section 4, some experimental verifications on numerical examples are presented. Finally, Section 5 concludes the paper.

Section snippets

Problem descriptions and preliminaries

Problem: Given m(m2n) pairs of vectors (u(1),v(1)), (u(2),v(2)),…, (u(m),v(m)), where u(j)=(u1,u2,un)T{q,q}n, v(j)=(v1,v2,vn)T{q,q}n, j{1,2,,m},q>0, design an associative memory based on a neural network such that if u(j)is fed as an external input to the network, then the output of the network converges to corresponding pattern v(j).

The following definitions and theorem are the bases of later discussions.

Definition 1

The neural network is an auto-associative memory when v(j)=u(j),j{1,2,,m},

Problem solutions

This section first proposes a general associative memory model, and then discusses how to design a specific auto-associative memory and hetero-associative memory by the model respectively.

Experimental verification

In this section, six numerical examples are chosen in our experiments to demonstrate the effectiveness of the proposed method.

Example 1

The purpose of the experiments on this example is to verify the correction of our method by organizing neural network (2) as an auto-associative memory.

Based on neural network (4), where the activation function is f(xi)=q2r(|xi+r||xir|) with q=1, and r=1. Let vector u(1)=v(1)=(1,1,1,1,1)Tand u(2)=v(2)=(1,1,1,1,1)T be the external inputs as memorized point, from (5)

Conclusions

In this paper, a further investigation on designing associative memories which improved on our previous research work is presented, in which a continuous recurrent neural network model is proposed instead of discrete one for accurately memorizing restored patterns. The model is unified and general with a tunable activation function and time delay, which can generalize auto-associative and hetero-associative memories and establish an exponential stable criterion. With the tunable slope

Acknowledgments

This work is partially supported by the National Natural Science Foundation of China (Grant no. 60971088, 61273106, 11202180, and 11402224) and Natural Science Foundation of Yancheng Teachers University (Grant no.14YCKL002).

Caigen Zhou received his B.S. and M.S degrees in Computer Science from Nanjing Normal University and Yangzhou University, respectively. He is currently pursuing his Ph.D. degree at Hohai University. His research interests include computational intelligence, neural networks, and associative memory.

References (48)

  • Y. Huang et al.

    Dynamical stability analysis of multiple equilibrium points in time-varing delayed recurrent neural networks with discontinuous activation functions

    Neurocomputing

    (2012)
  • L. Wang

    Dynamical analysis on the multistability of high-order neural networks

    Neurocompting

    (2013)
  • T. Wang, H. Gao, J. Qiu, A combined adaptive neural network and nonlinear model predictive control for multirate...
  • H. Li et al.

    Filtering of interval type-2 fuzzy systems with intermittent measurements

    IEEE Trans. Cybern.

    (2015)
  • X. Xie et al.

    Control synthesis of discrete-time T-S fuzzy systems via a multi-instant homogenous polynomial approach

    IEEE Trans. Cybern.

    (2015)
  • H. Li et al.

    Output-feedback control for T-S fuzzy delta operator systems with time-varying delays via an input-output approach

    IEEE Trans. Fuzzy Syst.

    (2015)
  • H. Li et al.

    State and output feedback control of a class of fuzzy systems with mismatched membership functions

    IEEE Trans. Fuzzy Syst.

    (2014)
  • Q. Zhou et al.

    Decentralized adaptive fuzzy tracking control for robot finger dynamics

    IEEE Trans. Fuzzy Syst.

    (2015)
  • Q. Zhou et al.

    Approximation-based adaptive tracking control for MIMO nonlinear systems with input saturation

    IEEE Trans. Cybern.

    (2015)
  • H. Li et al.

    Observer-based adaptive sliding mode control of Nonlinear markovian jump systems

    Automatica

    (2015)
  • J. Napoles et al.

    Selective harmonic mitigation technique for cascaded h-bridge converters with nonequal dc link voltages

    IEEE Trans. Ind. Electron.

    (2013)
  • E. Romero-Cadaval et al.

    Grid-connected photovoltaic generation plants components and operation

    IEEE Ind. Electron. Mag.

    (2013)
  • D. Liu et al.

    Dynamical neural networks with saturation nonlinearities: analysis and design

    (1994)
  • A.D. Maruani et al.

    Information retrieval in neural networks. - I. Eigenproblems in neural networks

    Rev. Phys. Appl. (Paris)

    (1987)
  • Cited by (0)

    Caigen Zhou received his B.S. and M.S degrees in Computer Science from Nanjing Normal University and Yangzhou University, respectively. He is currently pursuing his Ph.D. degree at Hohai University. His research interests include computational intelligence, neural networks, and associative memory.

    Xiaoqin Zeng received the Ph.D. degree from the Hong Kong Polytechnic University, the M.S. degree from Southeast University, and the B.S. degree from Nanjing University, all in Computer Science. He currently is a professor, Ph.D. student supervisor, and the director of the Institute of Intelligence Science and Technology, Hohai University, China. Prof. Zeng is an associate editor of IEEE Transactions on Cybernetics. As a principal investigator, he has been taking charge of several research projects awarded by Natural Science Foundation of China. His current research interests include computational Intelligence, machine learning, machine vision, and graph grammars.

    Jianjiang Yu received his Ph.D. degree in control theory and control engineering from Southeast University, China, in 2010. He was a Postdoctoral Researcher at the University of Portsmouth, UK, in 2010/2011. He currently is an associate professor in School of Information Engineering, Yancheng Teachers University. His research interests include time-delay systems, neural networks, and fuzzy control.

    Haibo Jiang received his Ph.D. degree in solod mechanics from Jiangsu University, China in 2012. He was an academic visitor at the Center of Applied Dynamics Research (CADR) of University of Aberdeen, UK, in 2014. His research interests include fuzzy control, impulsive control, nonlinear dynamics and multi-agent systems.

    View full text