A local and neurobiologically plausible method of learning correlated patterns

Abstract

The problem of learning correlated patterns in a neurobiologically plausible way without multiple iterations is discussed. Three guiding principles are outlined for solving this problem in a manner suggestive of how the memory modules of the human brain do it. The first, already known, principle of minimum disturbance is applied quantitatively to minimise the number of learning iterations. The second guiding principle involves a self-induced remedy to undo any damage caused to the stabilities of the old stored patterns after a new one has been learnt. The third involves localising connectivities between neurons depending on the structure of the input patterns. A neurobiologically plausible network and a learning rule are constructed based on these principles. Satisfactory use of this network in learning English words as sequences of their letters is demonstrated by means of computer simulation.

Introduction

Modelling the pattern learning capabilities of the human brain is an active area of research in recent times. It is now generally understood that patterns are stored in the synapses between the cortical neurons in the form of connection strengths and each pattern corresponds to a stable or fixed mode of activities of the neurons. The finite basin of attraction around a fixed mode of activity is supposed to provide the required noise tolerance in recalling an intended pattern. Based on a model proposed by Hopfield (1982), much work has been done subsequently (van Hemmen and Kuhn, 1991, Hertz et al., 1991) in trying to understand the storage of patterns in the human brain. Many Hebbian and local rules have been proposed for learning extensively many random and uncorrelated patterns. Both analytical and numerical studies have shown their efficacy (Amit et al., 1987, Amit, 1989) beyond doubt. However, when the patterns to be stored are correlated, most known rules are unable to learn them in just one presentation. Invariably many iterations of learning are required before all the patterns are stored as fixed modes of activity of the network (Krauth and Mezard, 1987, Gardner, 1988). With only one presentation most correlated patterns are not stored correctly; moreover, undesirable linear combinations of them get stored as spurious patterns.

The Hopfield model and many of its variants deal with high activity patterns. Experimental studies in neurobiology suggest that the average activity of any module of the brain is very low (Abeles, 1982), certainly well below the 50% assumed in the Hopfield model. Taking this into account Tsodyks and Feigel'man proposed a model for learning low activity patterns (Tsodyks & Feigel'man, 1988). Their model uses 0/1 coding for representing passive and active neurons. Its storage capacity turns out to be far greater than that of the Hopfield model when the activity levels of the patterns to be stored are well below 50%. However, it also cannot learn correlated patterns; specifically, patterns having correlations among their active elements. The same is true of a model proposed by Willshaw et al. (Golomb et al., 1990, Willshaw et al., 1969).

It is, of course, straightforward to construct an iterative learning rule requiring multiple presentations of low activity and correlated patterns. Locality or Hebbian condition can also be enforced. However, an external supervisory mechanism is needed to check if all the patterns are correctly learnt and if the iterations can stop. Patterns encountered in practical life are seldom uncorrelated with each other and a biologically plausible learning mechanism must be able to do with one presentation of a pattern to be learnt. Often, a second presentation of the pattern may not be feasible except in ideal classroom situations. Many times we are able to recall a pattern, say a face, on being presented with a clue long after learning it for the first time. In the intervening period we might not have had a chance to explicitly learn it again nor might we have learnt anything associated with it. Moreover, we might have learnt other patterns of a similar nature. In spite of all this, we do recall an old pattern, learnt only once, correctly when we need to. For practical applications too a rule capable of learning patterns in a single presentation would be useful. Learning, then, would become automatic and extra components on the learning system such as a teacher and pattern lookup storage, etc. could be dispensed with.

One of the reasons why most learning rules call for multiple presentations of correlated patterns is that while trying to ensure the correct level of stability of a new pattern, they either undercorrect or overcorrect the synaptic strengths. If the corrections to the synaptic strengths were less than what are required, obviously multiple presentations of the new pattern would become necessary. On the other hand, if the corrections are overdone, they are bound to disturb the stability of some old stored pattern. As a result, that old pattern would have to be relearnt. Correlated patterns are more likely to disturb the stabilities of each other than uncorrelated ones. Therefore, the corrections to the synaptic strengths have to be carefully worked out. As part of a numerical study being reported in this paper a learning rule, which can incorporate the right amount of corrections was worked out and experimented. It has been found that additional principles and mechanisms are needed if correlated patterns have to be learnt in one presentation. One of them is a method of self-induced relearning to cure the damages caused to the stabilities of the old patterns when learning a new one. Another is to localise the connectivities of the network in such a way as to suit the structure of the patterns to be stored. A neural network model and a learning rule based on these principles are described in the following sections. Experimental verification of the efficiency of the network is provided by means of learning sets of English words in one presentation.

The paper is organised as follows. Firstly, the three guiding principles are presented one of which has already been proposed in the literature. Based on these principles a new network for associative memory and a learning rule are constructed. Experimental verification of the performance of this network in learning correlated patterns in one presentation is presented next. The paper concludes with a discussion on the generality of the guiding principles for constructing biologically plausible associative memory networks.