Original contribution
Associative memories with short-range, higher order couplings

https://doi.org/10.1016/0893-6080(93)90008-KGet rights and content

Abstract

A study of recurrent associative memories with exclusively short-range connections is presented. To increase the capacity, higher order couplings are used. We study capacity and pattern completion ability of networks consisting of units with binary (±1) output. Results show that perfect learning of random patterns is difficult for very short coupling ranges, and that the average expected capacities (allowing small errors) in these cases are much smaller than the theoretical maximum, 2 bits per coupling. However, it is also shown that by choosing ranges longer than certain limit sizes, depending on network size and order, we can come close to the theoretical capacity limit. We indicate that these limit sizes increase very slowly with net size. Thus, couplings to at least 28 and 36 neighbors suffice for second order networks with 400 and 90,000 units, respectively. From simulations it is found that even networks with coupling ranges below the limit size are able to complete input patterns with more than 10% errors. Especially remarkable is the ability to correct inputs with large local errors (part of the pattern is masked). We present a local learning algorithm for heteroassociation in recurrent networks without hidden units. The algorithm is used in a multinet system to improve pattern completion ability on correlated patterns.

References (22)

  • D.J. Amit et al.

    Statistical mechanics of neural networks near saturation

    Annals of Physics

    (1987)
  • Y.C. Lee et al.

    Machine learning using a higher order correlation network

    Physica D

    (1986)
  • L.F. Abbott et al.

    Optimal learning in neural network memories

    Journal of Physics A

    (1989)
  • J.K. Anlauf et al.

    The AdaTron: An adaptive perception algorithm

    Europhysics Letters

    (1989)
  • J. Austin

    ADAM: A distributed associative memory for scene analysis

  • P. Baldi et al.

    Number of stable points for spin-glasses and neural networks of higher orders

    Physiological Review Letters

    (1987)
  • A. Canning et al.

    Partially connected models of neural networks

    Journal of Physics A

    (1988)
  • T.M. Cover

    Geometrical and statistical properties of systems of linear inequalities with application in pattern completion

    IEEE Transactions on Electronic Computers

    (1965)
  • B. Derrida et al.

    An exactly soluble asymmetric neural network model

    Europhysics Letters

    (1987)
  • E. Gardner

    Multiconnected neural network models

    Journal of Physics A

    (1987)
  • E. Gardner

    The space of interactions in neural networks models

    Journal of Physics A

    (1988)
  • Cited by (8)

    • Dual-mode space-varying self-designing cellular neural networks for associative memory

      1999, IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications
    • Poles and zeros of bandpass filters for high‐performance sigma‐delta modulators

      1995, International Journal of Circuit Theory and Applications
    View all citing articles on Scopus

    Work carried out at Computational Neural Network Center (CONNECT) and NORDITA, Copenhagen.

    View full text