Abstract
Associative matrix memories with real-valued synapses have been studied in many incarnations. We consider how the signal/noise ratio for associations depends on the form of the learning rule, and we show that a covariance rule is optimal. Two other rules, which have been suggested in the neurobiology literature, are asymptotically optimal in the limit of sparse coding. The results appear to contradict a line of reasoning particularly prevalent in the physics community. It turns out that the apparent conflict is due to the adoption of different underlying models. Ironically, they perform identically at their co-incident optima. We give details of the mathematical results, and discuss some other possible derivations and definitions of the signal/noise ratio.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bienenstock E, Cooper LN, Munro P (1982) Theory for the development of neuron selectivity: Orientation specificity and binocular interaction in visual cortex. J Neurosci 2:32–48
Bliss TVP, Lømo T (1973) Long-lasting potentiation of synaptic transmission in the dentate area of the anæsthetized rabbit following stimulation of the perforant path. J Physiol (London) 232:331–356
Buckingham JT (1991) Computation in the archicortex. PhD Thesis, Department of Artificial Intelligence, University of Edinburgh
Buhman J, Divko R, Schulten K (1989) On sparsely coded associative memories. In: Personnaz L, Dreyfus G (eds) Neural networks: from models to applications. nEURO88, Paris
Duda RO, Hart PE (1973) Pattern classification and scene analysis. Wiley, New York
Gaffan D (1974) Recognition impaired and association intact in the memory of monkeys after transection of the fornix. J Comp Physiol Psychol 86:1100–1109
Gardner E (1988) The space of interactions in neural network models. J Phys A: Math Gen 21:257
Hancock PJB, Smith LS, Phillips WA (1991) A biologically supported error-correcting learning rule. Neural Computation (in press)
Hebb DO (1949) The organization of behavior: a neuropsychological theory. Wiley, New York
Hopfield JJ (1982) Neural networks and physical systems with emergent computational abilities. Proc Natl Acad Sci 79:2554–2558
Ito M, Sakurai M, Tongroach P (1982) Climbing fibre induced depression of both mossy fibre responsiveness and glutamate sensitivity of cerebellar Purkinje cells. J Physiol 324:113–134
Kanerva P (1988) Sparse distributed memory. MIT Press, Cambridge
Kohonen T (1972) Correlation matrix memories. IEEE Trans Comput C-21:353–359
Marr D (1971) Simple memory: a theory for archicortex. Philos Trans Soc London B 262:23–81
Palm G (1988a) On the asymptotic information storage capacity of neural networks. In: Eckmiller R, von der Malsburg C (eds) Neural computers. NATO ASI Series F41. Springer, Berlin, Heidelberg New York, pp 271–280
Palm G (1988b) Local synaptic rules with maximal information storage capacity. In: Haken H (eds) Neural and synergetic computers Springer Series in Synergetics, vol 42. Springer, Berlin, Heidelberg New York, pp 100–110
Perez-Vincente CJ, Amit DJ (1989). Optimised network for sparsely coded patterns. J Phys A: Math Gen 22:559–569
Racine RJ, Milgram NW, Hafner S (1983) Long-term potentiation phenomena in the rat limbic forebrain. Brain Res 260:217–231
Sejnowski TJ (1977a) Storing covariance with nonlinearly interacting neurons. J Math Biol 4:303–321
Sejnowski TJ (1977b) Statistical constraints on synaptic plasticity. J Theor Biol 69:385–389
Singer W (1985) Activity-dependent self-organization of synaptic connections as a substrate of learning. In: Changeux JP, Konishi M (eds) The neural and molecular bases of learning. Wiley, New York, pp 301–335
Stanton P, Sejnowski TJ (1989) Associative long-term depression in the hippocampus: Induction of synaptic plasticity by Hebbian covariance. Nature 339:215–18
Stent GS (1973) A physiological mechanism for Hebb's postulate of learning. Proc Natl Acad Sci 70:997–1001
Tsodyks MV, Fiegel'man MV (1988) The enhanced storage capacity in neural networks with low activity level. Europhys Lett 6:101–105
Willshaw DJ (1971) Models of distributed associative memory. PhD Thesis, University of Edinburgh
Willshaw DJ, Dayan P (1990) Optimal plasticity in matrix memories: what goes up MUST come down. Neural Comput 2:85–93
Willshaw DJ, Buneman OP, Longuet-Higgins HC (1969) Nonholographic associative memory. Nature 222:960–962
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Dayan, P., Willshaw, D.J. Optimising synaptic learning rules in linear associative memories. Biol. Cybern. 65, 253–265 (1991). https://doi.org/10.1007/BF00206223
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/BF00206223