Neural coding of categories: information efficiency and optimal population codes

Abstract

This paper deals with the analytical study of coding a discrete set of categories by a large assembly of neurons. We consider population coding schemes, which can also be seen as instances of exemplar models proposed in the literature to account for phenomena in the psychophysics of categorization. We quantify the coding efficiency by the mutual information between the set of categories and the neural code, and we characterize the properties of the most efficient codes, considering different regimes corresponding essentially to different signal-to-noise ratio. One main outcome is to find that, in a high signal-to-noise ratio limit, the Fisher information at the population level should be the greatest between categories, which is achieved by having many cells with the stimulus-discriminating parts (steepest slope) of their tuning curves placed in the transition regions between categories in stimulus space. We show that these properties are in good agreement with both psychophysical data and with the neurophysiology of the inferotemporal cortex in the monkey, a cortex area known to be specifically involved in classification tasks.

Appendices

Appendix A: Derivation of Eq. (23)

The goal of this appendix and the following one is to derive Eqs. (18), (23) and (28). Remark: given the simplicity and the underlying identity of these results, we do not expect our derivations to be the simplest ones.

When N goes to ∞, we expect the mutual information I(μ, r) to converge towards I(μ, x), and we are interested in the first non trivial correction to this asymptotic limit. We thus compute for large N the difference

$$\Delta \equiv I(\mu, \mathbf{x}) - I(\mu, \mathbf{r}) \; \geq 0.$$

(33)

One can write

$$\Delta = - \int d^K\mathbf{x} \, d^N\mathbf{r} \, P(\mathbf{r}|\mathbf{x}) \, \phi(\mathbf{x}|\mathbf{r})$$

(34)

where

$$\phi(\mathbf{x}|\mathbf{r}) \equiv \sum_{\mu=1}^M p(\mathbf{x}) P(\mu|\mathbf{x}) \ln \frac{P(\mu|\mathbf{r})}{P(\mu|\mathbf{x})}$$

(35)

We follow the same approach as in Brunel and Nadal (1998). The first step consists in integrating over x. Taking the large N limit, we show that the leading order of the right term of Eq. (34) is zero. We then seek for the first correction using Laplace/steepest descent method. The last step eventually consists in integrating over r.

We introduce G(r|x) defined as :

$$G(\mathbf{r}|\mathbf{x}) \equiv \frac{1}{N} \ln P(\mathbf{r}|\mathbf{x})$$

(36)

and assume that it has a single global maximum at x = x _m (r). We can rewrite Eq. (34) in the following way:

$$\Delta = - \int d^K\mathbf{x} \, d^N\mathbf{r} \, e^{N G(\mathbf{r}|\mathbf{x})} \, \phi(\mathbf{x}|\mathbf{r})$$

(37)

Integration over x. In order to integrate Eq. (34) over x, let us first show that

$$\phi \big(\mathbf{x}_m(\mathbf{r})|\mathbf{r}\big) = 0$$

(38)

We begin by evaluating

$$\begin{array}{*{20}c} P(\mu|\mathbf{r}) \!-\! P\big(\mu|\mathbf{x}_m(\mathbf{r})\big) &=& \frac{P(\mathbf{r}|\mu) q_{\mu}}{P(\mathbf{r})} \!-\! P\big(\mu|\mathbf{x}_m(\mathbf{r})\big) \end{array} $$

(39)

$$\begin{array}{*{20}c} &=& \frac{1}{P(\mathbf{r})}\! \int{\kern-2pt} d^K\mathbf{x} \; \, \exp \big(N G(\mathbf{r}|\mathbf{x}) \big) \varphi_{\mu}(\mathbf{x}|\mathbf{r})\end{array}$$

(40)

with

$$ \varphi_{\mu}(\mathbf{x}|\mathbf{r}) = p(\mathbf{x}) \big[P(\mu|\mathbf{x}) - P\big(\mu|\mathbf{x}_m(\mathbf{r})\big) \big] $$

(41)

By using saddle-point method and assuming that N ≫ 1 and K ≪ N, we find :

$$\begin{array}{*{20}l} &&{\kern-6.5pt}P(\mu|\mathbf{r}) - P\big(\mu|\mathbf{x}_m(\mathbf{r})\big)\\ &&{\kern.8pc} =\frac{1}{P(\mathbf{r})} \exp \big(N G_m(\mathbf{r})\big) \sqrt{\frac{(2\pi)^K}{\det \mathbb{H}\big(-NG(\mathbf{r}|\mathbf{x})\big)\big|_{\mathbf{x}_m(\mathbf{r})}}}\\ &&{\kern1.7pc} \times \frac{1}{2} \operatorname{tr} \left( \mathbb{H}\big(\varphi_\mu(\mathbf{x}|\mathbf{r})\big)\big|_{\mathbf{x}_m(\mathbf{r})} {\kern3pt} \mathbb{H}\big(-NG(\mathbf{r}|\mathbf{x})\big)^{-1}\big|_{\mathbf{x}_m(\mathbf{r})} \right)\\ \end{array}$$

(42)

where ℍis the hessian matrix of ξ,

$$ \mathbb{H}_{kl}\big(\xi(\mathbf{x})\big) = \frac{\partial^2\xi(\mathbf{x})}{\partial x_k \partial x_l} $$

(43)

evaluated at x _m (r).

Since

$$ P(\mathbf{r}) = p\big(\mathbf{x}_m(\mathbf{r})\big) \exp \big(N G_m(\mathbf{r})\big) \sqrt{\frac{(2\pi)^K}{\det \mathbb{H}\big(-NG(\mathbf{r}|\mathbf{x})\big)\big|_{\mathbf{x}_m(\mathbf{r})}}} $$

(44)

we have

$$ \begin{array}{*{20}l} &&{\kern-6pt} P(\mu|\mathbf{r}) - P\big(\mu|\mathbf{x}_m(\mathbf{r})\big)\\ &&{\kern4pt} ={\kern-2pt} \frac{1}{2 p\big(\mathbf{x}_m(\mathbf{r})\big)}{\kern-1.5pt} \operatorname{tr}{\kern-2.5pt} \left({\kern-1pt} \mathbb{H}\big(\varphi_\mu(\mathbf{x}|\mathbf{r})\big)\big|_{\mathbf{x}_m(\mathbf{r})} \mathbb{H}\big({\kern-3pt}-{\kern-2pt}NG(\mathbf{r}|\mathbf{x})\big)^{-1}{\kern-1pt} \big|_{\mathbf{x}_m(\mathbf{r})}{\kern-1pt} \right)\\ \end{array}$$

(45)

As a result,

$$ \begin{array}{*{20}l} \phi \big(\mathbf{x}_m(\mathbf{r})|\mathbf{r}\big) \\ &&{\kern.3pc} \equiv {\kern-3pt} \displaystyle\sum_{\mu=1}^M p \big(\mathbf{x}_m(\mathbf{r})\big) P\big(\mu| \mathbf{x}_m(\mathbf{r})\big) \!\ln \frac{P(\mu|\mathbf{r})}{P \big(\mu|\mathbf{x}_m(\mathbf{r})\big)} \end{array}$$

(46)

$$ \begin{array}{*{20}c} &&{\kern.3pc}={\kern-3pt} \sum_{\mu=1}^M{\kern-2pt} p \big(\mathbf{x}_m(\mathbf{r})\big)\! P\big(\mu| \mathbf{x}_m(\mathbf{r})\big) {\kern-2pt}\ln{\kern-3pt} \left({\kern-3pt} 1 {\kern-2pt}+{\kern-2pt} \frac{P(\mu|\mathbf{r}){\kern-2pt}-{\kern-2pt}P\big(\mu|\mathbf{x}_m(\mathbf{r})\big)}{P\big(\mu|\mathbf{x}_m(\mathbf{r})\big)}{\kern-2pt} \right)\end{array}$$

(47)

According to Eq. (45), $P(\mu|\mathbf{r})-P\big(\mu|\mathbf{x}_m(\mathbf{r})\big)$ is of order 1/N, which entails, as ln (1 + z) ≈ z when z ≪ 1, that

$$ \begin{array}{*{20}l} \phi \big(\mathbf{x}_m(\mathbf{r})|\mathbf{r}\big)\\ &&{\kern.3pc} = \frac{1}{2} \sum_{\mu=1}^M \operatorname{tr} \left( \mathbb{H}\big(\varphi_\mu(\mathbf{x})\big)\big|_{\mathbf{x}_m(\mathbf{r})} {\kern3pt} \mathbb{H}\big(-NG(\mathbf{r}|\mathbf{x})\big)^{-1}\big|_{\mathbf{x}_m(\mathbf{r})} \right) &&{\kern.3pc} = \frac{1}{2} {\kern-2pt}\operatorname{tr}{\kern-2pt} \left( \mathbb{H}\left(\sum_{\mu=1}^M \varphi_\mu(\mathbf{x})\right)\big|_{\mathbf{x}_m(\mathbf{r})} \mathbb{H}\big({\kern-2pt}-{\kern-2pt}NG(\mathbf{r}|\mathbf{x})\big)^{-1}\big|_{\mathbf{x}_m(\mathbf{r})} \right) \end{array}$$

(48)

$$ \begin{array}{*{20}c} &&{\kern.3pc} = \frac{1}{2} {\kern-2pt}\operatorname{tr}{\kern-2pt} \left( \mathbb{H}\left(\sum_{\mu=1}^M \varphi_\mu(\mathbf{x})\right)\big|_{\mathbf{x}_m(\mathbf{r})} \mathbb{H}\big({\kern-2pt}-{\kern-2pt}NG(\mathbf{r}|\mathbf{x})\big)^{-1}\big|_{\mathbf{x}_m(\mathbf{r})} \right) \end{array}$$

(49)

Now, as $\sum_{\mu=1}^M \varphi_\mu(\mathbf{x}) = 0$ we have demonstrated that $\phi \big(\mathbf{x}_m(\mathbf{r})|\mathbf{r}\big) = 0$.

We can return to Eq. (34), and apply saddle-point method knowing that $\phi \big(\mathbf{x}_m(\mathbf{r})|\mathbf{r}\big) = 0$. This leads to :

$$ \begin{array}{*{20}c} \Delta = &-& \int d^N\mathbf{r} \exp \big(N G_m(\mathbf{r})\big) \sqrt{\frac{(2\pi)^K}{\det \mathbb{H}\big(-NG(\mathbf{r}|\mathbf{x})\big)\big|_{\mathbf{x}_m(\mathbf{r})}}}\\ &\times& \frac{1}{2} \operatorname{tr} \left( \mathbb{H}\big(\phi(\mathbf{x})|\mathbf{r}\big)\big|_{\mathbf{x}_m(\mathbf{r})} {\kern3pt} \mathbb{H}\big(-NG(\mathbf{r}|\mathbf{x})\big)^{-1}\big|_{\mathbf{x}_m(\mathbf{r})} \right)\\ \end{array}$$

(50)

Recalling that $P(\mu|\mathbf{r})\!-\!P\big(\mu|\mathbf{x}_m(\mathbf{r})\big) \!\sim\! O(\frac{1}{N})$, it is straightforward to show that :

$$ \begin{array}{*{20}l}&& {\kern-6.5pt} \frac{1}{p\big(\mathbf{x}_m(\mathbf{r})\big)} \frac{\partial \phi(\mathbf{x}|\mathbf{r}) }{\partial x_k \partial x_l} \Big|_{\mathbf{x}_m(\mathbf{r})}\\ &&{\kern.3pc} = - \sum_{\mu=1}^M \frac{1}{ P\big(\mu| \mathbf{x}_m(\mathbf{r})\big)} \frac{\partial P(\mu|\mathbf{x})}{\partial x_k} \Big|_{\mathbf{x}_m(\mathbf{r})} \frac{\partial P(\mu|\mathbf{x})}{\partial x_l} \Big|_{\mathbf{x}_m(\mathbf{r})} \end{array}$$

(51)

ie

$$ \begin{array}{*{20}l} &&{\kern-6.5pt} \frac{1}{p\big(\mathbf{x}_m(\mathbf{r})\big)} \mathbb{H}\big(\phi(\mathbf{x}|\mathbf{r})\big)\big|_{\mathbf{x}_m(\mathbf{r})}\\ &&{\kern.4pc} = - \sum_{\mu=1}^M \frac{1}{ P\big(\mu| \mathbf{x}_m(\mathbf{r})\big)} \nabla P(\mu|\mathbf{x})\big|_{\mathbf{x}_m(\mathbf{r})} \nabla P(\mu|\mathbf{x})\big|_{\mathbf{x}_m(\mathbf{r})}^{\top} \\ \end{array}$$

(52)

where $\nabla \xi(\mathbf{x})\big|_{\mathbf{x}_m(\mathbf{r})}$ is the column vector of the partial derivatives of ξ

$$ \nabla \xi(\mathbf{x}) = (\ldots, \partial\xi(\mathbf{x})/\partial x_k ,\ldots)^{\top} $$

(53)

evaluated at x _m (r).

Putting Eqs. (44), (50) and (52) together eventually leads to:

$$ \Delta = \frac{1}{2} \int d^N\mathbf{r} P(\mathbf{r}) \left(\sum_{\mu=1}^M \frac{1}{ P(\mu| \mathbf{x})} \nabla P(\mu|\mathbf{x}) \nabla P(\mu|\mathbf{x})^{\top} \right) \bigg|_{\mathbf{x}_m(\mathbf{r})} : \mathbb{H}\big(-NG(\mathbf{r}|\mathbf{x})\big)^{-1}\Big|_{\mathbf{x}_m(\mathbf{r})} $$

(54)

where ‘:’ denotes the Frobenius inner product on $\mathcal{M}_K(\mathbb{R})$, defined as follows:

$$ \forall A,B \in \mathcal{M}_K(\mathbb{R}), \; A:B = tr(A^{\top}B) = \sum_{k,l} A_{kl}B_{kl}. $$

(55)

Integration over r. By proceeding as Brunel and Nadal (1998, pp.1753–1754) in order to integrate over r, we get:

$$ \begin{array}{*{20}l} &&{\kern-6.5pt} I(\mu,\mathbf{x}) - I(\mu,\mathbf{r})\\ &&{\kern-6pt} = {\kern-2pt}\frac{1}{2}{\kern-2pt} \int{\kern-3pt} d^K\mathbf{x}\; p(\mathbf{x}){\kern2.5pt} {\kern-4pt} \left[{\kern-2pt} \sum_{\mu=1}^M{\kern-2pt} \frac{1}{P(\mu|\mathbf{x})} {\kern-2pt}\nabla{\kern-2pt} P(\mu|\mathbf{x}) \nabla P(\mu|\mathbf{x})^{\top} {\kern-2pt} \right]{\kern-4pt} :{\kern-1pt} F_{\text{code}}^{-1}(\mathbf{x})\\ \end{array}$$

(56)

where $F_{\text{code}}(\mathbf{x})$ is the K×K Fisher information matrix of the neuronal population,

$$ \big[F_{\text{code}}(\mathbf{x})\big]_{kl} \;{\kern1.5pt} = {\kern1.5pt}\;- \;\int d^N\mathbf{r} \;P(\mathbf{r}|\mathbf{x}) {\kern3pt} \frac{\partial^2 \ln P(\mathbf{r}|\mathbf{x}) }{\partial x_k \partial x_l} \label{app_fisher_code_matrix} $$

(57)

Noticing that

$$ \sum_{\mu=1}^M{\kern-2pt} \frac{1}{P(\mu|\mathbf{x})} \frac{\partial P(\mu|\mathbf{x})}{\partial x_k} \frac{\partial P(\mu|\mathbf{x})}{\partial x_l} {\kern-2pt}= -{\kern-2pt}\sum_{\mu=1}^M{\kern-2pt} P(\mu|\mathbf{x}) \frac{\partial^2 \ln P(\mu|\mathbf{x}) }{\partial x_k \partial x_l} $$

(58)

we can introduce the K×K Fisher information matrix of the categories, $F_{\text{cat}}(\mathbf{x})$, characterizing the sensitivity of μ with respect to small variations of x:

$$ \big[F_{\text{cat}}(\mathbf{x})\big]_{kl}{\kern1.5pt} \; = -\sum_{\mu=1}^M P(\mu|\mathbf{x}) \frac{\partial^2 \ln P(\mu|\mathbf{x})}{\partial x_k \partial x_l} $$

(59)

This eventually leads to Eq. (23):

$$ I(\mu,\mathbf{x}) - I(\mu,\mathbf{r}) = \frac{1}{2} \int d^K\mathbf{x} \, p(\mathbf{x}) \, F_{\text{cat}}^{}(\mathbf{x}):F_{\text{code}}^{-1}(\mathbf{x}) $$

(60)

Since $F_{\text{code}}$ is of order N, one has in particular that I(μ,x) − I(μ,r) is of order 1/N.

Appendix B: Derivation of Eq. (28)

Each cell has an activity r _i equal to 1 if x is in [θ _i , θ _i + 1], and 0 otherwise. The width of the ith tuning curves is thus a _i  = θ _i + 1 − θ _i , and we define the preferred stimuli as the center of the receptive fields, x _i  ≡ (θ _i  + θ _i + 1)/2.

We want to compute

$$ \Delta \equiv I(\mu,x)-I(\mu,\mathbf{r}) = {\cal H}(\mu | \mathbf{r}) - {\cal H}(\mu | x) $$

(61)

where

$${\cal H}(\mu | x) = - \int dx {\kern1.5pt} \; p(x) \; \displaystyle\sum_{\mu=1}^M \; P(\mu|x) \ln P{(\mu|x)}; $$

$${\cal H}(\mu | \mathbf{r}) = - \int d^N\mathbf{r}{\kern1.5pt} \; P(\mathbf{r}) \; \displaystyle\sum_{\mu=1}^M \; Q(\mu | \mathbf{r}) \ln Q(\mu | \mathbf{r}) $$

in terms of the { θ _i , i = 1,...,N + 1}, and to see what is the optimal choice of the { θ _i , i = 1,...,N + 1}, or equivalently of the {x _i , i = 1,...,N}, for having this difference Δ as small as possible for a large but finite value of N.

If we set:

$$ \widetilde{P}_i \equiv \int_{\theta_{i}}^{\theta_{i+1}} dx {\kern1.5pt} \: p(x) $$

(62)

$$\widetilde{P}_i(\mu) \equiv \int_{\theta_{i}}^{\theta_{i+1}} dx {\kern1.5pt} \: p(x |\mu) $$

(63)

$$\widetilde{Q}_{i,\mu} \equiv P(\mu | r_i=1) = \frac{\widetilde{P}_i(\mu) \, q_{\mu}}{\widetilde{ P}_i} $$

(64)

$$Q_{\mu}(x) \equiv P(\mu | x) $$

(65)

then we can write Δ from Eq. (61) as :

$$ \Delta = \sum_{i} \, \Delta_i $$

(66)

with

$$ \Delta_i = \int_{\theta_i}^{\theta_{i+1}} dx {\kern3pt} p(x) {\kern3pt} \Big[ \mathcal{H}(\{ \widetilde{Q}_{i,\mu}\}) - \mathcal{H}(\{ Q_{\mu}(x)\}) \Big] $$

(67)

where $\mathcal{H}(\{Q_{\mu}\}_{\mu=1}^M )$ is the mixing entropy:

$$ \mathcal{H}\left(\{Q_{\mu}\}_{\mu=1}^M\right) \; = \; - \; \displaystyle\sum_{\mu=1}^M \, Q_{\mu} \ln Q_{\mu} $$

(68)

The quantity $\mathcal{H}(\{ Q_{\mu}(x)\})$ is zero on a homogeneous domain, hence there is no contribution from intervals included in such domain. Suppose on the contrary that on the full range under consideration $\mathcal{H}(\{ Q_{\mu}(x)\})$ is rapidly varying. We expect then the optimal θ _i ’s distribution to be dense, that is a _i  = θ _i + 1 − θ _i small.

Recalling that x _i  =  (θ _i + 1 + θ _i )/2 and assuming a _i  ≪ 1,

$$ \begin{array}{*{20}c} \widetilde{P}_i &=& \int_{-a_i/2}^{a_i/2} dz \left[p(x_i) + z p'(x_i) + \frac{z^2}{2} p''(x_i) \right] \\ &=& a_i \; p(x_i) + \frac{ a_i^3}{24} \; p''(x_i). \end{array}$$

Likewise, $\widetilde{P}_i (\mu) = a_i \; p(x_i|\mu) + a_i^3/24 \; p''(x_i|\mu)$. Thus,

$$\widetilde{Q}_{i,\mu} = Q_{\mu}(x_i) + \frac{a_i^2}{24} A_{i,\mu} $$

(69)

with

$$ A_{i,\mu} = Q_{\mu}''(x_i) + 2 \frac{p'(x_i)}{p(x_i)} \; Q_{\mu}'(x_i) $$

(70)

A Taylor expansion then gives:

$$ \begin{array}{*{20}c} &&{\kern-6pt} \mathcal{H}\big({\kern-.2pt}\{\widetilde{Q}_{i,\mu}\}{\kern-.2pt}\big){\kern-3pt} ={\kern-3pt} \mathcal{H}\big({\kern-.2pt} \{Q_{\mu}(x_i)\}{\kern-.2pt}\big) {\kern-3pt}+{\kern-3pt} \nabla \mathcal{H}\big({\kern-.2pt}\{Q_{\mu}(x_i)\}{\kern-.2pt}\big){\kern-2pt} \cdot{\kern-4pt} \left( {\kern-4pt}\cdots{\kern-2pt} \frac{a_i^2 A_{i,\mu}}{24}{\kern-1pt} \cdots{\kern-4pt} \right)^{{\kern-2pt}\top} \\ &&+ \frac{1}{2} \left( \; \cdots {\kern3pt} \frac{a_i^2 A_{i,\mu}}{24} \; \cdots \right) \mathbb{H}_{\mathcal{H}}\big( \{Q_{\mu}(x_i)\} \big) \left( \; \cdots {\kern3pt} \frac{a_i^2 A_{i,\mu}}{24} \; \cdots \right)^{\top} \end{array}$$

(71)

where $\nabla \mathcal{H}\big(\{Q_{\mu}(x_i)\}\big)$ and ℍare respectively the gradient and the hessian of $\mathcal{H}$ evaluated in {Q _μ (x _i )} (see (53) and (43). A second order expansion leads to:

$$ \mathcal{H}\big(\{\widetilde{Q}_{i,\mu}\}\big) = \mathcal{H}\big(\{Q_{\mu}(x_i)\}\big) - \frac{a_i^2}{24} \sum_{\mu=1}^M A_{i,\mu} (\ln Q_{\mu}(x_i) + 1) $$

(72)

Thus we have:

$$ \begin{array}{*{20}l} &&{\kern-8pt}\int_{\theta_i}^{\theta_{i+1}} dx {\kern1.5pt} p(x) {\kern3pt} \mathcal{H}\big(\{\widetilde{Q}_{i,\mu}\}\big)\\ &&{\kern.6pc} = a_i p(x_i) \mathcal{H}\big(\{Q_{\mu}(x_i)\}\big) + \frac{a_i^3}{24} \left(p''(x_i) \vphantom{\sum_{\mu=1}^M A_{i,\mu}} \mathcal{H}\big(\{Q_{\mu}(x_i)\}\big) \right.\\ &&{\kern1.5pc} \left.- p(x_i) \sum_{\mu=1}^M A_{i,\mu} (\ln Q_{\mu}(x_i) + 1) \right) \end{array}$$

(73)

Moreover,

$$ \begin{array}{*{20}l} &&{\kern-8pt}\int_{\theta_i}^{\theta_{i+1}} dx {\kern1.5pt} p(x) {\kern3pt} \mathcal{H}\big(\{Q_{\mu}(x)\}\big)\\ &&{\kern.6pc} = a_i p(x_i)\mathcal{H}\big(\{Q_{\mu}(x_i)\}\big) + \frac{a_i^3}{24} \frac{\partial^2 \; p(x) \, \mathcal{H}\big(\{Q_{\mu}(x)\}\big)}{\partial x^2} \Big|_{x_i} \\ \end{array}$$

(74)

with

$$ \begin{array}{*{20}l} &&{\kern-7.5pt} \frac{\partial^2 \; p(x) \mathcal{H}\big(\{Q_{\mu}(x)\}\big)}{\partial x^2} \Big|_{x_i}\\ &&{\kern.2pc} = - 2 p'(x_i) \sum_{\mu=1}^M Q_{\mu}'(x_i) (\ln Q_{\mu}(x_i) + 1) \\ &&{\kern1.15pc} - p(x_i) \left(\sum_{\mu=1}^M \left\{{\kern-1.5pt} Q_{\mu}''(x_i) (\ln Q_{\mu}(x_i) + 1) + \frac{Q_{\mu}'(x_i)^{2}}{Q_{\mu}(x_i)} {\kern-2pt} \right\}{\kern-3pt} \right) \\ &&{\kern1.15pc} - p''(x_i) \sum_{\mu=1}^M Q_{\mu}(x_i) \ln Q_{\mu}(x_i) \end{array}$$

(75)

Putting Eqs. (73), (74) and (75) together eventually leads to :

$$ \Delta_i = \frac{a_i^3}{24} \;p(x_i) \sum_{\mu=1}^{M} {\kern3pt}\frac{P'(\mu |x_i)^2}{P(\mu |x_i)} $$

(76)

hence

$$ \Delta = \sum_i \frac{a_i^3}{24} \;p(x_i) \; F_{\text{cat}}(x_i) $$

(77)