Capacity analysis in multi-state synaptic models: a retrieval probability perspective

Abstract

We define the memory capacity of networks of binary neurons with finite-state synapses in terms of retrieval probabilities of learned patterns under standard asynchronous dynamics with a predetermined threshold. The threshold is set to control the proportion of non-selective neurons that fire. An optimal inhibition level is chosen to stabilize network behavior. For any local learning rule we provide a computationally efficient and highly accurate approximation to the retrieval probability of a pattern as a function of its age. The method is applied to the sequential models (Fusi and Abbott, Nat Neurosci 10:485–493, 2007) and meta-plasticity models (Fusi et al., Neuron 45(4):599–611, 2005; Leibold and Kempter, Cereb Cortex 18:67–77, 2008). We show that as the number of synaptic states increases, the capacity, as defined here, either plateaus or decreases. In the few cases where multi-state models exceed the capacity of binary synapse models the improvement is small.

Appendix

1.1 A.1 Kronecker product

If A is an m×n matrix and B is a p×q matrix, then the Kronecker product A ⊗ B is the mp×nq block matrix

$$ A\otimes B = \begin{bmatrix} a_{11} B & \cdots & a_{1n}B \\ \vdots & \ddots & \vdots \\ a_{m1} B & \cdots & a_{mn} B \end{bmatrix}. $$

The Kronecker product is bilinear and associative

• A ⊗ (B + C) = A ⊗ B + A ⊗ C,

• (A + B) ⊗ C = A ⊗ C + B ⊗ C,

• (kA) ⊗ B = A ⊗ (kB) = k(A ⊗ B),

• (A ⊗ B) ⊗ C = A ⊗ (B ⊗ C),

where A, B and C are matrices and k is a scalar. If A, B, C and D are matrices of such size that one can form the matrix products AC and BD, then

$$ (A \otimes B)(C \otimes D) = AC \otimes BD. $$

This is called the mixed-product property because it mixes the ordinary matrix product and the Kronecker product.

1.2 A.2 Transition matrices and covariances

As the synaptic modification rule is local, the evolution of the state of a synapse is a Markov chain. Summing over the possible firing states of the pre- and postsynaptic neuron, the probability a synapse transits from α _m to α _m′ is

$$ f^2 q^{11}_{m m'} + f(1\!-\!f) q^{10}_{m m'}+ (1\!-\!f) f q^{01}_{m m'} + (1\!-\!f)^2 q^{00}_{m m'}, $$

so the transition matrix can then be written as

$$ P = f^2Q^{11}+f(1\!-\!f) Q^{10}+f(1\!-\!f) Q^{01}+(1\!-\!f)^2Q^{00}. $$

Similarly, a pair of synapses with a common postsynaptic neuron $(J^{(p)}_{ij}, J^{(p)}_{ik})$ is also a Markov chain, where the state space is the cross product {α ₁, α ₂,...,α _M }×{α ₁, α ₂,...,α _M }. Again summing over the firing status of the three neurons i, j, k, the transition probability from (α _l ,α _m ) to (α _l′,α _m′) is

$$\begin{array}{lll} &&{\kern-6pt}f[fq^{11}_{l l'}+(1\!-\!f) q^{10}_{l l'}][fq^{11}_{m m'}+(1\!-\!f) q^{10}_{m m'}]\\ &&+\;(1\!-\!f) [fq^{01}_{l l'}+(1\!-\!f) q^{00}_{l l'}][fq^{01}_{m m'}+(1\!-\!f) q^{00}_{m m'}] \end{array}$$

so the transition matrix is

$$\begin{array}{rll} S&=&f [fQ^{11}+(1\!-\!f) Q^{10}]\otimes[fQ^{11}+(1\!-\!f) Q^{10}]\\ &&+\;(1\!-\!f) [fQ^{01}\!+\!(1\!-\!f) Q^{00}]\otimes[fQ^{01}\!+\!(1\!-\!f) Q^{00}] \end{array}$$

Suppose the network is initialized at its stationary state. $\pi_x^{(0)}=\pi$ and $\gamma^{(0)}_x=\gamma$. To obtain the mean and variance after the first step of learning (Eq. (6)), given $\xi^{(1)}_i=x$, since the presynaptic neuron is on $\xi^{(1)}_j=\xi^{(1)}_k=1$, the transition probability for $J^{(p)}_{ij}$ from α _m to α _m′ is $q^{x1}_{m m'}$, and the transition probability for $(J^{(p)}_{ij}, J^{(p)}_{ik})$ from (α _l ,α _m ) to (α _l′,α _m′) is $q^{x1}_{l l'}q^{x1}_{m m'}$. This explains Eq. (6).

Since $J^{(p)}_{ij}$ is a indicators variable, $\mathsf{E}[J^{(p)}_{ij}|\xi^{(1)}_i=x,\xi^{(1)}_j=1]$ and $\mathsf{E}[J^{(p)}_{ij}\otimes J^{(p)}_{ik}|\xi^{(1)}_i=x,\xi^{(1)}_j=\xi^{(1)}_k=1]$ are exactly the p-step distribution of the Markov chains $\pi^{(p)}_x$ and $\gamma^{(p)}_x$, and Eq. (7) comes from the Kolmogorov equations.

Equation (8) is straightforward since $W^{(p)}_{ij}=J^{(p)}_{ij}\mathbf{w}^T$. One can verify that $\mathsf{Var}(J^{(p)}_{ij}|\xi^{(1)}_i=x,\xi^{(1)}_j=1)=\mathrm{Diag}(\pi_x^{(p)}) -\pi_x^{(p)}(\pi_x^{(p)})^T$ and deduce Eq. (9), where $\mathrm{Diag}(\pi_x^{(p)})$ is the diagonal matrix with $\pi_x^{(p)}$ on the diagonal. To obtain Eq. (10) we use:

$$\begin{array}{rll} \mathsf{Cov}(J_{ij}\mathbf{w}^T,J_{ik}\mathbf{w}^T)&=&\mathsf{E}[(J_{ij}\mathbf{w}^T)^T J_{ik}\mathbf{w}^T] - \mu^2\\ &=& \mathbf{w}\mathsf{E}[J_{ij}^T J_{ik}]\mathbf{w}^T - \mu^2\\&=&\gamma(\mathbf{w}\otimes\mathbf{w})^{T} -\mu^2. \end{array}$$

where the last equality comes from the mixed-product property of the Kronecker product (See Appendix A.1).

1.3 A.3 Decorrelating the synapses

If the four Q matrices are of the form $Q^{11}=I_{2m}+q_+D_+$, $Q^{01}=Q^{10}=I_{2m}+q_-D_-$, $Q^{00}=I_{2m}+q_0D_+$, where $q_0=\frac{f^2q_+}{(1-f)^2}$, $q_-=\frac{\tau f q_+}{1-f}$, then

$$\begin{array}{lll} P_1=I_{2m}+fq_+(D_++\tau D_-),\\ P_0=I_{2m}+\frac{f^2q_+}{1-f}(D_++\tau D_-),\\ P=I+2f^2q_+(D_++\tau D_-) \end{array}$$

If π is the stationary distribution of P, i.e. πP = π, then π(D ₊ + τD ₋) = 0, which implies πP ₁ = πP ₀ = π. Hence, γ = π ⊗ π of is the stationary distribution of S since

$$\begin{array}{rll} (\pi\otimes\pi) S &=& (\pi\otimes\pi) [f P_1\otimes P_1 + (1-f) P_0\otimes P_0]\\ &=&f (\pi P_1\otimes \pi P_1) + (1-f) (\pi P_0)\otimes (\pi P_0)\\ &=&f (\pi\otimes \pi) + (1-f) (\pi\otimes \pi)=(\pi\otimes \pi). \end{array}$$

Thus the stationary synaptic covariance ρ defined in Eq. (14) must be 0.

This learning rule only works for single-level coded stimuli, but not for multi-level coded stimuli.

Table 2 Summary of properties of different multi-state models

1.4 A.4 Bias in sparseness measurement

In Rolls and Tovee (1995) and Sato et al. (2007) coding level is defined as

$$a = \frac{(\sum_i^n r_i/n)^2}{\sum_i^n r_i^2/n} $$

(36)

where r _i is the firing rate of the neuron to the ith stimulus in a set of n stimuli. Say the baseline firing rate of the neuron is r, and if the neuron is selective to the stimulus, the firing rate is Lr (L > 1), and assume the true sparseness is f. Then on average ∑  _i r _i /n ≈ fLr + (1 − f)r, $\sum_i r_i^2/n \approx fL^2r^2 + (1-f)r^2$.

$$ a \approx \frac{(fL+1-f)^2}{fL^2+1-f} > f $$

Though a→f as L→ ∞, when f→0, a ≈ 1 − f. The relationship between a and f is not monotone and when L = 30, a is always above 0.1 whatever f is (Fig. 11). If the baseline firing rate is subtracted from each r _i , then L will be larger and make a closer to f.

Fig. 11

The sparseness measure a (Eq. (36)) versus the actual coding level f for L = 10 and 30