Improved bidirectional retrieval of sparse patterns stored by Hebbian learning
Introduction
In the late fifties, Steinbuch (1961) proposed one of the first hardware realizations of a neural associative memory (NAM) model, his so called `Lernmatrix', where binary synapses are formed by Hebbian local learning from binary memory patterns and hetero-associative one-step retrieval is performed. This model is now referred to as the Willshaw model, since Willshaw et al. (1969) first determined its asymptotic1 information efficiency as ln[2]=0.69 bit/synapse. Such high efficiencies are only achieved in models with sparse memory patterns where the ratio between active and passive elements is far below 0.5 (Palm and Sommer, 1995), and asymptotically, the Willshaw model is the most efficient NAM, see the model comparison in Sommer and Dayan (1998). Unfortunately, at maximum memory load, the finite Willshaw network retrieves with high error rates (see Section 5). Error reduction in the Willshaw model can only be achieved by reducing the memory load that leads to information efficiencies far below the asymptotic value. For the Hopfield task, namely auto-associative pattern completion (Hopfield, 1982), iterative retrieval has been introduced in the Willshaw model (Gardner-Medwin, 1976; Gibson and Robinson, 1992; Hirase and Recce, 1996; Schwenker et al., 1996). For auto-association of finite sparse patterns, it is now well understood that the original retrieval process is the limiting bottleneck (Palm and Sommer, 1992), and what kind of retrieval modifications are most promising in the light of probabilistic reasoning (Sommer and Dayan, 1998).
This paper considers bidirectional retrieval in the hetero-associative Willshaw model. Hetero-associative iterative retrieval schemes have been proposed in bidirectional associative memory (BAM) models for dense patterns (Kosko, 1987), where two sets of threshold units are bidirectionally connected by a synaptic weight matrix. An improved theoretical analysis for the Willshaw model is derived and applied to the straightforward BAM extension of the Willshaw model (Haines and Hecht-Nielsen, 1988), referred to as standard bidirectional (SB) retrieval in the following, where standard retrieval is performed alternatingly for the two disjunct sets of units, the x and the y layers. By this analysis, the SB model can be ruled out as a promising model variant. Standard retrieval discriminates active neurons by thresholding the overlap between the vector of synaptic values and the activity pattern in the other layer. We propose a completely new retrieval method, crosswise bidirectional (CB) retrieval, based on dynamic virtual connections between units within a layer called conditional links. The conditional link between two neurons depends on the activity pattern in the other layer: it is defined by the overlap between the parts of their synaptic vectors that receive input from active units in the other layer. By the structure of the Hebbian synaptic matrix in the Willshaw model, the conditional link has a probabilistic interpretation: a high value indicates a high probability that the corresponding unit pair belongs to the memory pattern that should be associated with the activity pattern in the other layer. CB retrieval employs bidirectional activity propagation to determine the clique of units that are connected by the highest conditional links. CB retrieval uses dendritic sums weighted by conditional links that can be computed in parallel — by bidirectional propagation through the synaptic matrix. Thus, each CB update step involves evaluation of columns and rows as well, a fact that led to the name of this retrieval strategy. The advantage of CB retrieval is shown by the analysis of a single step and by simulation experiments.
The paper is organized into six sections. Section 2 revisits briefly the Willshaw NAM, its biological motivation and some basic definitions; it also explains the relations and differences between the hetero- and auto-association task and what memory models can combine both functionalities. In Section 3, we define bidirectional extensions of the model, namely SB retrieval — the classical BAM scheme, and CB retrieval based on conditional links. We present two versions of CB retrieval that differ in the iteration scheme and in the means to limit activity in the network. The theoretical background of the proposed retrieval strategies is developed in Section 4. In Section 4.1, the information efficiency is defined as a general performance measure for BAM and in Section 4.2, the asymptotic efficiency bound for the Willshaw model is determined. Section 4.3 contains a refined analysis of the finite size Willshaw model: an improved combinatorial calculation of retrieval error probabilities is derived (Proposition 4.2) that allows much faster numerical evaluation than the previously given formula (Buckingham and Willshaw, 1993). We analyze pattern part retrieval (Proposition 4.3), a method using the standard model to extract a part of the 1-elements in the memory pattern with higher accuracy. The analysis of the standard model is used in Section 4.4 to estimate the retrieval error of SB retrieval (Proposition 4.4). Section 4.5 analyzes the first step of CB retrieval (Proposition 4.5). Section 4.6 identifies the problem of optimal retrieval (POOR) as a Boolean linear optimization problem. In this framework, we point out the differences between SB and CB retrieval in Section 4.7. Section 5 presents some typical simulation results with the CB retrieval method and compares it with the Willshaw model. The conclusion section resumes the implications of the presented analysis (Section 6.1) and the properties of the proposed new BAM model (Section 6.2). The perspectives of applying the CB model in information retrieval are discussed in Section 6.3. We close with some speculations about the biological realization of BAM in reciprocal cortico–cortical pathways and the impact of such a cortex model on concept formation and processing (Section 6.4).
Section snippets
Motivations for associative memory models
In one of the first attempts of computational brain modeling, McCulloch and Pitts (1943) proposed a neural network of binary `all or none' units and showed the computational universality of such systems. The psychologist Hebb (1949) speculated that psychological concepts could be represented by simultaneous activity of many nerve cells distributed throughout the brain, which he called a cell-assembly. He postulated that cell-assemblies are formed by an amplification process taking place in all
Improved retrieval strategies
This section describes the model modifications that we introduce in the Willshaw model. The simplest idea is the straight-forward BAM extension which, however, turns out as an inefficient model. The literature about BAM concentrates on improving the learning prescription: either multiple training schemes have to be employed (Hassoun, 1989), or a higher number than nm weights have to be stored, for instance, if dummy augmentation (adding subsidiary components to the original memory patterns) or
Definitions
In the following, we assume that the memory patterns are randomly generated, i.e. each component in the x- and y-patterns has been set to `one' with probability p:=a/n and q:=b/m respectively. (a, b, m, and n as defined in Section 2.2).
We consider the retrieval with an initial pattern which is the noisy version of the x pattern in a learning pattern pair (xμ, yμ). For the occurrence probabilities of `false alarm' and `miss' errors, the following notation will be used:For r odd:
Retrieval errors and capacity
Both versions of CB retrieval have been tested in simulation experiments with random patterns, and compared with the standard retrieval model. Here, we show results for variant II with a parameter setting that has not been particularly optimized to maximum capacity. For more detailed experimental results with variant I, see Sommer and Palm (1998) and Sommer et al. (1998). SB retrieval is not investigated experimentally, because it can be ruled out as a promising modification by the arguments in
Implications of the analysis
In this paper, we have analyzed bidirectional retrieval from hetero-associative memories. The search capacity introduced in Section 4.1 describes the information balance of BAM, taking into account both pattern mapping and pattern completion. As a general performance measure for BAM models, we propose the information efficiency which is based on the search capacity, but also takes into account the required synaptic depth. For memories with binary synapses both quantities coincide. We calculate
Acknowledgements
The authors would like to thank N. Palomero-Gallagher and T. Wennekers for their comments and suggestions which helped in improving the manuscript. This work was supported by grants SO352/3-1 and PA268/8-3 from the Deutsche Forschungsgemeinschaft.
References (43)
- et al.
Information storage and effective data retrieval in sparse matrices
Neural Networks
(1989) Text signatures by superimposed coding of letter triplets and quadruplets
Info. Syst.
(1987)Dynamic heteroassociative neural memories
Neural Networks
(1989)- et al.
Iterative retrieval of sparsely coded associative memory patterns
Neural Networks
(1996) - et al.
Increasing the efficiency of a neural network through unlearning
Physika A
(1990) The Hebbian paradigm reintegrated: local reverberations as internal representations
Behav. Brain Sci.
(1995)- Braitenberg, V., & Schüz, A. (1991). The anatomy of the cortex. Statistics and geometry. New York:...
- et al.
Performance characteristics of associative nets
Network
(1992) - et al.
On setting unit thresholds in an incompletely connected associative net
Network
(1993) - et al.
REM sleep and neural nets
J. Mind Behav.
(1986)
Distributed hierarchical processing in the primate cerebral cortex
Cerebral Cortex
Forming sparse representations by local anti-hebbian learning
Biol. Cybern.
The recall of events through the learning of associations between their parts
Proc. R. Soc. Lond. B
Statistical analysis of the dynamics of a sparse associative memory
Neural Networks
Improving recall from an associative memory
Biol. Cybern.
A search for the optimal thresholding sequence in an associative memory
Network
Performance enhancement of Willshaw type networks through the use of limit cycles
J. Phys. Fr.
Cited by (57)
Neural associative memories and sparse coding
2013, Neural NetworksAn associative memory approach to medical decision support systems
2012, Computer Methods and Programs in BiomedicineCitation Excerpt :Hopfield models are also appealing to many cognitive modelers because of their apparent similarity to human episodic memory: they can recall patterns after a single exposure using a Hebbian learning rule, and they are capable of retrieval from partial or noisy patterns [19]. This model is one of the most popular work that use Hebbian learning and owes some of its advantages regarding learning and recalling of altered patterns to Hebbian learning rules [20]. Among the myriad contributions and innovations in the field of associative memories, an associative memory approach for pattern recognition termed as Distributed Hierarchical Graph Neuron (DHGN) was presented by Khan and Muhamad Amin [21].
Neural coding in graphs of bidirectional associative memories
2012, Brain ResearchRegarding the temporal requirements of a hierarchical Willshaw network
2012, Neural NetworksTree-like hierarchical associative memory structures
2011, Neural NetworksExponential attractor of κ-almost periodic sequence solution of discrete-time bidirectional neural networks
2010, Simulation Modelling Practice and Theory