Abstract
The classical binary Willshaw model of associative memory has an asymptotic storage capacity of ln 2 ≈ 0.7 which exceeds the capacities of other (e.g., Hopfield-like) models by far. However, its practical use is severely limited, since the asymptotic capacity is reached only for very large numbers n of neurons and for sparse patterns where the number k of one-entries must match a certain optimal value k opt(n) (typically kopt = log n). In this work I demonstrate that optimal compression of the binary memory matrix by a Huffman or Golomb code can increase the asymptotic storage capacity to 1. Moreover, it turns out that this happens for a very broad range of k being either ultra-sparse (e.g., k constant) or moderately-sparse (e.g., k = 325-01). A storage capacity in the range of ln 2 is already achieved for practical numbers of neurons.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Bentz, H.J., Hagstroem, M., Palm, G.: Information storage and effective data retrieval in sparse matrices. Neural Networks 2 (1989) 289–293
Cover, T.M, Thomas, J.A.: Elements of information theory. Wiley, New York, 1991
Golomb, S.W.: Run-length encodings. IEEE Transactions on Information Theory 12 (1966) 399–401
Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA 79 (1982) 2554–2558
Hopfield, J.J., Tank, D.W.: Computing with neural circuits. Science 233 (1986) 625–633
Huffman, D.A.: A method for the construction of minimum redundancy codes. Proceedings of the Institute of Radio Engineers 40 (1952) 1098–1101
Knoblauch, A., Palm, G.: Pattern separation and synchronization in spiking associative memories and visual areas. Neural Networks 14 (2001) 763–780
Knoblauch, A., Palm, G.: Scene segmentation by spike synchronization in reciprocally connected visual areas. II. Global assemblies and synchronization on larger space and time scales. Biological Cybernetics 87 (2002) 168–184
Knoblauch, A., Sommer, F.T., Palm, G.: Willshaw associative memory and matrix compression. in preparation
Kosko, B.: Bidirectional associative memories. IEEE Transactions on Systems, Man, and Cybernetics 18 (1988) 49–60
Palm, G.: On associative memories. Biological Cybernetics 36 (1980) 19–31
Palm, G.: Memory capacities of local rules for synaptic modification. A comparative review. Concepts in Neuroscience 2 (1991) 97–128
Rachkovskij, D.A., Kussul, E.M.: Binding and normalization of binary sparse distributed representations by context-dependent thinning. Neural Computation 13 (2001) 411–452
Schwenker, F., Sommer, F.T., Palm, G.: Iterative retrieval of sparsely coded associative memory patterns. Neural Networks 9 (1996) 445–455
Sommer, F.T., Palm, G.: Improved bidirectional retrieval of sparse patterns stored by Hebbian learning. Neural Networks 12 (1999) 281–297
Steinbuch, K.: Die Lernmatrix. Kybernetik 1 (1961) 36–45
Willshaw, D.J., Buneman, O.P., Longuet-Higgins, H.C.: Non-holographic associative memory. Nature 222 (1969) 960–962
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Knoblauch, A. (2003). Optimal Matrix Compression Yields Storage Capacity 1 for Binary Willshaw Associative Memory. In: Kaynak, O., Alpaydin, E., Oja, E., Xu, L. (eds) Artificial Neural Networks and Neural Information Processing — ICANN/ICONIP 2003. ICANN ICONIP 2003 2003. Lecture Notes in Computer Science, vol 2714. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44989-2_39
Download citation
DOI: https://doi.org/10.1007/3-540-44989-2_39
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40408-8
Online ISBN: 978-3-540-44989-8
eBook Packages: Springer Book Archive