Elsevier

Neural Networks

Volume 9, Issue 5, July 1996, Pages 837-844
Neural Networks

CONTRIBUTED ARTICLE
Using Features for the Storage of Patterns in a Fully Connected Net

https://doi.org/10.1016/0893-6080(95)00113-1Get rights and content

Abstract

One of the many possible conditions for pattern storage in a Hopfield net is to demand that the local field vector be a pattern reconstruction. We use this criterion to derive a set of weights for the storage of correlated biased patterns in a fully connected net. The connections are built from the eigenvectors or principal components of the pattern correlation matrix. Since these are often identified with the features of a pattern set we have named this particular set of weights as the feature matrix. We present simulation results that show the feature matrix to be capable of storing up to N random patterns in a network of N spins. Basins of attraction are also investigated via simulation and we compare them with both our theoretical analysis and those of the pseudo-inverse rule. A statistical mechanical investigation using the replica trick confirms the result for storage capacity. Finally we discuss a biologicaly plausible learning rule capable of realising the feature matrix in a fully connected net. Copyright © 1996 Elsevier Science Ltd

Section snippets

INTRODUCTION

The fully connected net (Hopfield, 1982) is a model of a neural network which exhibits associative memory (Amit, 1989). It consists of a connected system of spins (neurons), si = ±1, with adaptable internal connections (synapses) wij, i, j = 1, 2, 3, …, N. The synapses are chosen such that a prescribed set of states become fixed point attractors of the network dynamics. These states {ξμ|μ = 1, 2, 3, …, P} are the patterns memorised by the network. For the memory to be truly associative we also

THE MODEL

We consider a fully connected net with dynamics:si(t+1) sgn(hi(s(t))hi(s(t))=∑jwijsj(t). The state of the ith neuron in the network is represented by si and may take only the values ±1. The function sgn(x) returns the sign of x and hi(s(t)) is simply the weighted sum of inputs at the ith neuron. It contains contributions from all neurons in the network. In the language of spin-glasses it is the local field at the ith spin.

The dynamics, decribed by Eq. (1), are not governed by the energy

A STATISTICAL MECHANICAL DESCRIPTION OF THE NETWORK

In this section we investigate the equilibrium properties of the feature matrix (without self-interactions) in the thermodynamic limit. Furthermore we calculate the basin of attraction for networks with constant stabilities in the limit of strong synaptic dilution. This description is relevant to both the feature matrix and the pseudo-inverse rule.

LEARNING THE FEATURE MATRIX

The feature matrix is easily obtainable with traditional numerical routines that extract eigenvectors [see for example Press et al. (1994)]. One may exploit the properties of the pattern correlation matrix and perform a Householder reduction on it (requiring O(4 N3/3) operations). An algorithm such as QL may be used to obtain the eigenvectors and eigenvalues of such a real symmetric, tri-diagonal matrix (requiring O(3 N3) operations).

In this section we present a neural mechanism for the

CONCLUSION

In this paper we have demonstrated that a certain matrix built from the principal components of a pattern set may act as a projection operator on the pattern subspace. Consequently it has many similarities with the pseudo-inverse rule, including the ability to store up to N patterns in a fully connected network of N spins. In fact the free-energies of the two systems were shown to differ only by a constant. Unlike the pseudo-inverse rule, however, the feature matrix is not restricted to

Acknowledgements

This work was supported by a grant from the Engineering and Physical Sciences Research Council (UK).

References (24)

  • D.H. Ackley et al.

    A learning algorithm for Boltzmann machines

    Cognitive Science

    (1985)
  • P. Peretto

    On the dynamics of memorization processes

    Neural Networks

    (1988)
  • T.D. Sanger

    Optimal unsupervised learning in a single layer linear feedforward neural network

    Neural Networks

    (1989)
  • Amit, D.J. (1989) Modelling brain function. Cambridge: Cambridge University...
  • D.J. Amit et al.

    Storing infinite numbers of patterns in a spin-glass model of neural networks

    Phys. Rev. Lett.

    (1985)
  • Coolen, T., & Sherrington, D. (1993). Dynamics of attractor neural networks. In J.G. Taylor (Ed.), Mathematical...
  • S. Coombes et al.

    Using generalised principal component analysis to achieve associative memory in a Hopfield net

    Network

    (1993)
  • S. Coombes et al.

    The storage and stabilization of patterns in a Hopfield net

    Neural Network World

    (1995)
  • B. Derrida et al.

    An exactly solvable asymmetric neural network model

    Europhys. Letters

    (1987)
  • E. Gardner

    The space of interactions in neural network models

    J. Phys. A

    (1988)
  • J.J. Hopfield

    Neural networks and physical systems with emergent collective computational abilities

    Proc. Nat. Acad. Sci. USA

    (1982)
  • I. Kanter et al.

    Associative recall of memory without errors

    Phys. Rev. A

    (1987)

    • Cited by (0)

    View full text
    Copyright © 1996 Elsevier Science Ltd. All rights reserved.