Elsevier

Neural Networks

Volume 12, Issue 2, March 1999, Pages 281-297
Neural Networks

Improved bidirectional retrieval of sparse patterns stored by Hebbian learning

https://doi.org/10.1016/S0893-6080(98)00125-7Get rights and content

Abstract

The Willshaw model is asymptotically the most efficient neural associative memory (NAM), but its finite version is hampered by high retrieval errors. Iterative retrieval has been proposed in a large number of different models to improve performance in auto-association tasks. In this paper, bidirectional retrieval for the hetero-associative memory task is considered: we define information efficiency as a general performance measure for bidirectional associative memory (BAM) and determine its asymptotic bound for the bidirectional Willshaw model. For the finite Willshaw model, an efficient new bidirectional retrieval strategy is proposed, the appropriate combinatorial model analysis is derived, and implications of the proposed sparse BAM for applications and brain theory are discussed. The distribution of the dendritic sum in the finite Willshaw model given by Buckingham and Willshaw [Buckingham, J., & Willshaw, D. (1992). Performance characteristics of associative nets. Network, 3, 407–414] allows no fast numerical evaluation. We derive a combinatorial formula with a highly reduced evaluation time that is used in the improved error analysis of the basic model and for estimation of the retrieval error in the naive model extension, where bidirectional retrieval is employed in the hetero-associative Willshaw model. The analysis rules out the naive BAM extension as a promising improvement. A new bidirectional retrieval algorithm — called crosswise bidirectional (CB) retrieval — is presented. The cross talk error is significantly reduced without employing more complex learning procedures or dummy augmentation in the pattern coding, as proposed in other refined BAM models [Wang, Y. F., Cruz, J. B., & Mulligan, J. H. (1990). Two coding strategies for bidirectional associative memory. IEEE Trans. Neural Networks, 1(1), 81–92; Leung, C.-S., Chan, L.-W., & Lai, E. (1995). Stability, capacity and statistical dynamics of second-order bidirectional associative memory. IEEE Trans. Syst. Man Cybern., 25(10), 1414–1424]. The improved performance of CB retrieval is shown by a combinatorial analysis of the first step and by simulation experiments: it allows very efficient hetero-associative mapping, as well as auto-associative completion for sparse patterns — the experimentally achieved information efficiency is close to the asymptotic bound. The different retrieval methods in the hetero-associative Willshaw matrix are discussed as Boolean linear optimization problems. The improved BAM model opens interesting new perspectives, for instance, in information retrieval it allows efficient data access providing segmentation of ambiguous user input, relevance feedback and relevance ranking. Finally, we discuss BAM models as functional models for reciprocal cortico–cortical pathways, and the implication of this for a more flexible version of Hebbian cell-assemblies.

Introduction

In the late fifties, Steinbuch (1961) proposed one of the first hardware realizations of a neural associative memory (NAM) model, his so called `Lernmatrix', where binary synapses are formed by Hebbian local learning from binary memory patterns and hetero-associative one-step retrieval is performed. This model is now referred to as the Willshaw model, since Willshaw et al. (1969) first determined its asymptotic1 information efficiency as ln[2]=0.69 bit/synapse. Such high efficiencies are only achieved in models with sparse memory patterns where the ratio between active and passive elements is far below 0.5 (Palm and Sommer, 1995), and asymptotically, the Willshaw model is the most efficient NAM, see the model comparison in Sommer and Dayan (1998). Unfortunately, at maximum memory load, the finite Willshaw network retrieves with high error rates (see Section 5). Error reduction in the Willshaw model can only be achieved by reducing the memory load that leads to information efficiencies far below the asymptotic value. For the Hopfield task, namely auto-associative pattern completion (Hopfield, 1982), iterative retrieval has been introduced in the Willshaw model (Gardner-Medwin, 1976; Gibson and Robinson, 1992; Hirase and Recce, 1996; Schwenker et al., 1996). For auto-association of finite sparse patterns, it is now well understood that the original retrieval process is the limiting bottleneck (Palm and Sommer, 1992), and what kind of retrieval modifications are most promising in the light of probabilistic reasoning (Sommer and Dayan, 1998).

This paper considers bidirectional retrieval in the hetero-associative Willshaw model. Hetero-associative iterative retrieval schemes have been proposed in bidirectional associative memory (BAM) models for dense patterns (Kosko, 1987), where two sets of threshold units are bidirectionally connected by a synaptic weight matrix. An improved theoretical analysis for the Willshaw model is derived and applied to the straightforward BAM extension of the Willshaw model (Haines and Hecht-Nielsen, 1988), referred to as standard bidirectional (SB) retrieval in the following, where standard retrieval is performed alternatingly for the two disjunct sets of units, the x and the y layers. By this analysis, the SB model can be ruled out as a promising model variant. Standard retrieval discriminates active neurons by thresholding the overlap between the vector of synaptic values and the activity pattern in the other layer. We propose a completely new retrieval method, crosswise bidirectional (CB) retrieval, based on dynamic virtual connections between units within a layer called conditional links. The conditional link between two neurons depends on the activity pattern in the other layer: it is defined by the overlap between the parts of their synaptic vectors that receive input from active units in the other layer. By the structure of the Hebbian synaptic matrix in the Willshaw model, the conditional link has a probabilistic interpretation: a high value indicates a high probability that the corresponding unit pair belongs to the memory pattern that should be associated with the activity pattern in the other layer. CB retrieval employs bidirectional activity propagation to determine the clique of units that are connected by the highest conditional links. CB retrieval uses dendritic sums weighted by conditional links that can be computed in parallel — by bidirectional propagation through the synaptic matrix. Thus, each CB update step involves evaluation of columns and rows as well, a fact that led to the name of this retrieval strategy. The advantage of CB retrieval is shown by the analysis of a single step and by simulation experiments.

The paper is organized into six sections. Section 2 revisits briefly the Willshaw NAM, its biological motivation and some basic definitions; it also explains the relations and differences between the hetero- and auto-association task and what memory models can combine both functionalities. In Section 3, we define bidirectional extensions of the model, namely SB retrieval — the classical BAM scheme, and CB retrieval based on conditional links. We present two versions of CB retrieval that differ in the iteration scheme and in the means to limit activity in the network. The theoretical background of the proposed retrieval strategies is developed in Section 4. In Section 4.1, the information efficiency is defined as a general performance measure for BAM and in Section 4.2, the asymptotic efficiency bound for the Willshaw model is determined. Section 4.3 contains a refined analysis of the finite size Willshaw model: an improved combinatorial calculation of retrieval error probabilities is derived (Proposition 4.2) that allows much faster numerical evaluation than the previously given formula (Buckingham and Willshaw, 1993). We analyze pattern part retrieval (Proposition 4.3), a method using the standard model to extract a part of the 1-elements in the memory pattern with higher accuracy. The analysis of the standard model is used in Section 4.4 to estimate the retrieval error of SB retrieval (Proposition 4.4). Section 4.5 analyzes the first step of CB retrieval (Proposition 4.5). Section 4.6 identifies the problem of optimal retrieval (POOR) as a Boolean linear optimization problem. In this framework, we point out the differences between SB and CB retrieval in Section 4.7. Section 5 presents some typical simulation results with the CB retrieval method and compares it with the Willshaw model. The conclusion section resumes the implications of the presented analysis (Section 6.1) and the properties of the proposed new BAM model (Section 6.2). The perspectives of applying the CB model in information retrieval are discussed in Section 6.3. We close with some speculations about the biological realization of BAM in reciprocal cortico–cortical pathways and the impact of such a cortex model on concept formation and processing (Section 6.4).

Section snippets

Motivations for associative memory models

In one of the first attempts of computational brain modeling, McCulloch and Pitts (1943) proposed a neural network of binary `all or none' units and showed the computational universality of such systems. The psychologist Hebb (1949) speculated that psychological concepts could be represented by simultaneous activity of many nerve cells distributed throughout the brain, which he called a cell-assembly. He postulated that cell-assemblies are formed by an amplification process taking place in all

Improved retrieval strategies

This section describes the model modifications that we introduce in the Willshaw model. The simplest idea is the straight-forward BAM extension which, however, turns out as an inefficient model. The literature about BAM concentrates on improving the learning prescription: either multiple training schemes have to be employed (Hassoun, 1989), or a higher number than nm weights have to be stored, for instance, if dummy augmentation (adding subsidiary components to the original memory patterns) or

Definitions

In the following, we assume that the memory patterns are randomly generated, i.e. each component in the x- and y-patterns has been set to `one' with probability p:=a/n and q:=b/m respectively. (a, b, m, and n as defined in Section 2.2).

We consider the retrieval with an initial pattern x(0)=x̃μ which is the noisy version of the x pattern in a learning pattern pair (xμ, yμ). For the occurrence probabilities of `false alarm' and `miss' errors, the following notation will be used:For r odd:

Retrieval errors and capacity

Both versions of CB retrieval have been tested in simulation experiments with random patterns, and compared with the standard retrieval model. Here, we show results for variant II with a parameter setting that has not been particularly optimized to maximum capacity. For more detailed experimental results with variant I, see Sommer and Palm (1998) and Sommer et al. (1998). SB retrieval is not investigated experimentally, because it can be ruled out as a promising modification by the arguments in

Implications of the analysis

In this paper, we have analyzed bidirectional retrieval from hetero-associative memories. The search capacity introduced in Section 4.1 describes the information balance of BAM, taking into account both pattern mapping and pattern completion. As a general performance measure for BAM models, we propose the information efficiency which is based on the search capacity, but also takes into account the required synaptic depth. For memories with binary synapses both quantities coincide. We calculate

Acknowledgements

The authors would like to thank N. Palomero-Gallagher and T. Wennekers for their comments and suggestions which helped in improving the manuscript. This work was supported by grants SO352/3-1 and PA268/8-3 from the Deutsche Forschungsgemeinschaft.

References (43)

  • D.J. Felleman et al.

    Distributed hierarchical processing in the primate cerebral cortex

    Cerebral Cortex

    (1991)
  • P. Földiak

    Forming sparse representations by local anti-hebbian learning

    Biol. Cybern.

    (1990)
  • A. Gardner-Medwin

    The recall of events through the learning of associations between their parts

    Proc. R. Soc. Lond. B

    (1976)
  • W. Gibson et al.

    Statistical analysis of the dynamics of a sparse associative memory

    Neural Networks

    (1992)
  • B. Graham et al.

    Improving recall from an associative memory

    Biol. Cybern.

    (1995)
  • Haines, K., & Hecht-Nielsen, R. (1988). A BAM with increased information storage capacity. In Proceedings of the...
  • Hebb, D. O. (1949). The organization of behaviour. New York:...
  • H. Hirase et al.

    A search for the optimal thresholding sequence in an associative memory

    Network

    (1996)
  • Hopfield, J. (1982). Neural networks and physical systems with emergent collective computational abilities. In...
  • Kohonen, T. (1977). Associative memory. New York:...
  • G.A. Kohring

    Performance enhancement of Willshaw type networks through the use of limit cycles

    J. Phys. Fr.

    (1990)
  • Cited by (57)

    • An associative memory approach to medical decision support systems

      2012, Computer Methods and Programs in Biomedicine
      Citation Excerpt :

      Hopfield models are also appealing to many cognitive modelers because of their apparent similarity to human episodic memory: they can recall patterns after a single exposure using a Hebbian learning rule, and they are capable of retrieval from partial or noisy patterns [19]. This model is one of the most popular work that use Hebbian learning and owes some of its advantages regarding learning and recalling of altered patterns to Hebbian learning rules [20]. Among the myriad contributions and innovations in the field of associative memories, an associative memory approach for pattern recognition termed as Distributed Hierarchical Graph Neuron (DHGN) was presented by Khan and Muhamad Amin [21].

    View all citing articles on Scopus
    View full text