Convergence results in an associative memory model

doi:10.1016/0893-6080(88)90029-9

Neural Networks

Volume 1, Issue 3, 1988, Pages 239-250

https://doi.org/10.1016/0893-6080(88)90029-9 Get rights and content

Abstract

This paper presents rigorous mathematical proofs for some observed convergence phenomena in an associative memory model introduced by Hopfield (based on Hebbian rules) for storing a number of random $n-bit$ patterns. The capability of the model to correct a linear number of random errors in a bit pattern has been established earlier, but the existence of a large domain of attraction (correcting a linear number of arbitrary errors) has not been proved.

We present proofs for the following:

•
• When $m$ , the number of patterns stored, is less than $n /(4 log n)$ , the fundamental memories have a domain of attraction of radius ρn with $ρ = 0.024$ , and the algorithm converges in time $O (log log n)$ .
•
• When $m = α n$ (with α small), all patterns within a distance ρn from a fundamental memory end up, in constant time, within a distance ϵn from the fundamental memory, where ϵ is about $e^{−1/4α}$

We also extend somewhat Newman's description of the “energy landscape,” and prove the existence of an exponential number of stable states (extraneous memories) with convergence properties similar to those of the fundamental memories.

References (18)

D.J. Amit et al.
Statistical mechanics of neural networks near saturation
Annals of Physics
(1987)
W.A. Little et al.
Analytic study of the memory storage capacity of a neural network
Mathematical Biosciences
(1978)
C.M. Newman
Memory capacity in neural network models: Rigorous lower bounds
Neural Networks
(1988)
D.J. Amit et al.
Spin-glass models of neural networks
Physical Review A
(1985)
M.A. Cohen et al.
Absolute stability of global pattern formation and parallel memory storage by competitive neural networks
IEEE Transactions on Systems, Man, and Cybernetics
(1983)
E. Gardner
Structure of metastable states in the Hopfield model
Journal of Physics A
(1986)
S. Grossberg
Studies of mind and brain
(1982)
D.O. Hebb
The organization of behavior
(1949)

There are more references available in the full text version of this article.

Cited by (68)

The capacity of the dense associative memory networks
2022, Neurocomputing
This paper revisits the dense associative memory (DAM) networks and studies rigorously the capacity of the DAM networks. We present the capacity theorem of the DAM networks with an attraction radius or a noise level from the messages and prove that the probe can converge to the targeted message just after the one-step update. Under this convergence, the capacity of DAM networks is between a lower bound and an upper bound. Although when the attraction radius is $0.0$ away from the messages, i.e. noiseless, previous literature provides an approximate result. However, a rigorous proof is not given in this study. In addition, we consider a more general notion of capacity which allows the retrieval of messages from noisy probes (the attraction radius is not $0.0$ ). We demonstrates that the convergence result can be acquired just after the one-step update when the probe is a corrupted version with a Gaussian noise from one message. We further provide simulated experiments to validate theorems herein.
On the capacity of an associative memory model based on neural cliques
2015, Statistics and Probability Letters
Fluctuations of the free energy in the high temperature Hopfield model
2004, Stochastic Processes and their Applications
We consider the Hopfield model of size N and with p∼tN patterns, in the whole high temperature (paramagnetic) region. Our result is that the partition function has log-normal fluctuations. It is obtained by extending to the present model the method of the interpolating Brownian motions used by Comets (Comm. Math. Phys. 166 (1995) 549–564) for the Sherrington–Kirkpatrick model. We view the load t of the memory as a dynamical parameter, making the partition function a nice stochastic process. Then we write some semi-martingale decomposition for the logarithm of the partition function, and we prove that all the terms in this decomposition converge. In particular, the martingale term converges to a Gaussian martingale.
Covering Cubes by Random Half Cubes, with Applications to Binary Neural Networks
1998, Journal of Computer and System Sciences
LetQ_nbe the (hyper)cube {−1, 1}ⁿ. This paper is concerned with the following question: How many vectors must be chosen uniformly and independently at random fromQ_nbefore every vector inQ_nitself has negative inner product with at least one of the random vectors? For any fixedε>0, a simple expectation argument shows that for all sufficiently largen, (1+ε) nrandom vectors suffice with high probability. In this paper we prove that there areε, ρ>0 such that (1−ε) nrandom vectors are also enough and such that at leastρnrandom vectors are necessary. This problem is partially motivated by neural network problems. Neural networks are being used to solve a growing number of difficult problems such as speech recognition, handwriting recognition, and protein structure prediction. Recently, for both theoretical and practical reasons, neural networks with binary weights (binary neural networks) have attracted much attention. In spite of considerable analysis based on statistical mechanics, the following two basic questions about binary neural networks have remained unanswered. Q1. Is there a positive constantρsuch that for all sufficiently largenthere is a binary neural network ofnneurons which can separateρn(unbiased) random patterns with probability close to 1? Q2. Is it possible for a binary neural network ofnneurons to separate (1−o(1)) nrandom patterns with probability greater than some positive constant? (Hereo(1) goes to 0 asngoes to infinity.) This question is raised because no binary neural network ofnneurons can separate more than n random patterns with probability close to 1. Our results yield the answers “YES” to Q1 and “NO” to Q2.
Learning pattern classification-A survey
1998, IEEE Transactions on Information Theory
Typical error pattern recovery of the hopfield memory under error-tolerant conditions
1998, IEEE Transactions on Information Theory

View all citing articles on Scopus

View full text

Original contributionConvergence results in an associative memory model

Abstract

Annals of Physics

Mathematical Biosciences

Neural Networks

Spin-glass models of neural networks

Physical Review A

Absolute stability of global pattern formation and parallel memory storage by competitive neural networks

IEEE Transactions on Systems, Man, and Cybernetics

Structure of metastable states in the Hopfield model

Journal of Physics A

Studies of mind and brain

The organization of behavior

Original contribution
Convergence results in an associative memory model