Adaptation using local information for maximizing the global cost

doi:10.1016/S0925-2312(02)00779-8

Neurocomputing

Volumes 52–54, June 2003, Pages 541-546

https://doi.org/10.1016/S0925-2312(02)00779-8 Get rights and content

Abstract

Recently the information transmission properties of noisy, parallel summing threshold arrays, have been investigated and interpreted in a neural coding context (see Stocks, Phys. Rev. Lett. 84 (2000) 2310; Phys. Rev. E 63 (2001) 1). The mutual information between certain stimuli and corresponding responses displays a maximum as a function of the noise level. This optimal noise level depends on the number N of neurons within the array, information that is not locally available for single neuron adaptation. We give an analytic expression for the optimal noise level, that only depends on locally available information. The result is based upon an approximation to the mutual information. In the large N limit both descriptions coincide.

Introduction

A noise induced maximum in the mutual information is a signature of stochastic resonance (see [3] for a review). Stochastic resonance has been extensively studied in the context of single neurons (see e.g. [4]). Recently the information transmission properties of noisy, parallel summing threshold arrays, have been investigated and interpreted in a neural coding context (see Stocks [5], [7], [6]). The mutual information between certain stimulus and response distributions is maximized as a function of the noise level. This optimal noise level depends on the number N of array elements. Adaptation of single neurons to the optimal noise level would require the knowledge of N. But this is information about the global architecture of the system and it is not—in an obvious way—available locally at the single cell. In this study we give an analytic expression for the optimal noise level, which depends only on information which is—in principal—locally available at any single neuron. This local optimality criterium is based upon an approximation of the mutual information. In the large N limit both descriptions yield the same optimal noise level. The quality of the approximation is investigated numerically.

This paper is organized as follows. First we introduce the abstract model which describes the input–output relationship of a summing array of parallel threshold elements. In Section 3 the mutual information between a distribution of inputs and a distributions of outputs is introduced. Here we give an approximation to the standard mutual information, based on the work of Brunel et al. [1] and the model given in Section 2. Using this approximation we deduce an analytical expression for the optimal noise level which depends only on information, which is locally available at any single neuron. Section 4 contains numerical results in which we demonstrate the quality of our approximation. Section 5, finally, concludes with a brief discussion.

Section snippets

Model

The model [5] consists of a parallel summing array of N threshold devices (Fig. 1). The input to each threshold device is the sum of the signal X and the noise η_i, which is compared to a constant threshold Θ, to yield the following input–output relation: $y_{i} =sign(x+η_{i} −Θ).$ The signal X is the same for all threshold devices and is drawn from a gaussian distribution with variance σ_x. Each neuron is corrupted with gaussian noise η_i with variance $σ_{η}, i∈{1,…,N}$ , which is assumed to be mutually

Approximation of the mutual information

The average mutual information is an information theoretic measure which quantifies the amount of information the output contains about the input [2]. The mutual information I^MI between P(X) and P(Z) is $I^{MI} =H(Z)−H(Z|X)$ $=− ∑ n=0 N P(n) log_{2} P(n)+ ∫ −∞ ∞ d xP_{X} (x) ∑ n=0 N P(n|x) log_{2} P(n|x).$ Stocks demonstrates [7], that maximum information transfer occurs at an optimal noise level (σ_opt^MI). To yield an analytical expression for the optimal noise level, we adapted an approximation introduced by Brunel and Nadal [1].

Results

In Fig. 2a the mutual information and I^F are plotted against σ=σ_n/σ_x for various N and Θ=0. The curves indicate that for increasing $N, I^{F}$ approximates the mutual information. Note that I^F can be negative, because the approximation is good only in the case F(x)⪢1. That means, that the estimator is sharply peaked around its mean value. Furthermore, Fig. 2a displays, that the optimal noise level of the Fisher information σ_opt^F is independent from the number of threshold devices, where as the

Discussion

In this model study we examined stochastic resonance in a summing parallel threshold array. We compared optimal noise levels of the mutual information I^MI between a distribution of inputs and a corresponding distribution of outputs, and I^F, what is an approximation to I^MI. The quality of this approximation increases with the number N of parallel nonlinearities. The optimal noise level σ_opt^MI for I^MI depends on N, thus a neuron that should adapt to the optimal noise level needs to have some

Acknowledgements

Supported by Wellcome Trust (061113/Z/00).

References (8)

N. Brunel et al.
Mutual information, Fisher information, and population coding
Neural Comput.
(1998)
T.M. Cover et al.
Elements of Information Theory, Wiley Series in Telecommunications
(1991)
L. Gammaitoni et al.
Stochastic resonance
Rev. Modern Phys.
(1998)
D.F. Russel et al.
Use of behavioral stochastic resonance by paddle fish for feeding
Nature
(1999)

There are more references available in the full text version of this article.

Cited by (17)

Optimising threshold levels for information transmission in binary threshold networks: Independent multiplicative noise on each threshold
2015, Physica A: Statistical Mechanics and its Applications
Citation Excerpt :
In Ref. [15], each threshold operates on the same Gaussian signal, but is subject to independent and identically distributed (i.i.d.) additive Gaussian noise, and the SSR effect is measured using mutual information. Since this seminal work, SSR has been extensively investigated for the case where all thresholds have the same value [17–27]. Closely related work has focused on the interesting case when threshold devices are replaced by identical neuron models [28–33].
The problem of optimising the threshold levels in multilevel threshold system subject to multiplicative Gaussian and uniform noise is considered. Similar to previous results for additive noise, we find a bifurcation phenomenon in the optimal threshold values, as the noise intensity changes. This occurs when the number of threshold units is greater than one. We also study the optimal thresholds for combined additive and multiplicative Gaussian noise, and find that all threshold levels need to be identical to optimise the system when the additive noise intensity is a constant. However, this identical value is not equal to the signal mean, unlike the case of additive noise. When the multiplicative noise intensity is instead held constant, the optimal threshold levels are not all identical for small additive noise intensity but are all equal to zero for large additive noise intensity. The model and our results are potentially relevant for sensor network design and understanding neurobiological sensory neurons such as in the peripheral auditory system.
Suprathreshold stochastic resonance in visual signal detection
2008, Behavioural Brain Research
The present study aimed to demonstrate stochastic resonance (SR) for a visual signal above the threshold by using a two-alternative forced-choice protocol. The percent correct response enhanced at non-zero intensity of noise, while decreased with more intense noise. SR for the suprathreshold signals was found for signals with relatively low intensity above the threshold, and disappeared for more intense signals. These findings show the suprathreshold SR in the visual perception.
Noise enhancement of signal transduction by parallel arrays of nonlinear neurons with threshold and saturation
2007, Neurocomputing
Citation Excerpt :
In neuronal arrays, Collins et al. [11], Chialvo et al. [9], and Hoch et al. [18,19] show stochastic resonance essentially with a subthreshold input signal, while Stocks [24], Stocks and Mannella [27], and Hoch et al. [17] show the novel form of suprathreshold stochastic resonance. In the neuronal arrays, suprathreshold stochastic resonance is shown in [24,17] with simple threshold binary neurons, meanwhile Collins et al. [11], and Stocks and Mannella [27] for this investigate an excitable FitzHugh–Nagumo model in its subthreshold and suprathreshold regimes. For isolated nonlinear systems, it has recently been shown that stochastic resonance can also operate in threshold-free nonlinearities with saturation, where the noise has the ability to reduce the distortion experienced by a signal because of the saturation [23], with an extension to arrays of threshold-free sensors with saturation given in [8].
A classic model neuron with threshold and saturation is used to form parallel uncoupled neuronal arrays in charge of the transduction of a periodic or aperiodic noisy input signal. The impact on the transduction efficacy of added noises is investigated. In isolated neurons, improvement by noise is possible only in the subthreshold and in the strongly saturating regimes of the neuronal response. In arrays, improvement by noise is always reinforced, and it becomes possible in all regimes of operation, i.e. in the threshold, in the saturation, and also in the intermediate curvilinear part of the neuronal response. All the configurations of improvement by noise apply equally to periodic and to aperiodic signals. These results extend the possible forms of stochastic resonance or improvement by noise accessible in neuronal systems for the processing of information.
Stochastic resonance and noise-enhanced fisher information
2005, Fluctuation and Noise Letters
Suprathreshold stochastic resonance in neural processing tuned by correlation
2011, Physical Review E - Statistical, Nonlinear, and Soft Matter Physics
Bio-inspired communication: Performance limits for information transmission and compression in stochastic pooling networks with binary quantizing nodes
2010, Journal of Computational and Theoretical Nanoscience

View all citing articles on Scopus

View full text

Neurocomputing

Adaptation using local information for maximizing the global cost

Abstract

Introduction

Section snippets

Model

Approximation of the mutual information

Results

Discussion

Acknowledgements

Mutual information, Fisher information, and population coding

Neural Comput.

Elements of Information Theory, Wiley Series in Telecommunications

Stochastic resonance

Rev. Modern Phys.

Use of behavioral stochastic resonance by paddle fish for feeding

Nature

Optimising threshold levels for information transmission in binary threshold networks: Independent multiplicative noise on each threshold

Suprathreshold stochastic resonance in visual signal detection

Noise enhancement of signal transduction by parallel arrays of nonlinear neurons with threshold and saturation

Stochastic resonance and noise-enhanced fisher information

Suprathreshold stochastic resonance in neural processing tuned by correlation

Bio-inspired communication: Performance limits for information transmission and compression in stochastic pooling networks with binary quantizing nodes