Alternative time representation in dopamine models

Abstract

Dopaminergic neuron activity has been modeled during learning and appetitive behavior, most commonly using the temporal-difference (TD) algorithm. However, a proper representation of elapsed time and of the exact task is usually required for the model to work. Most models use timing elements such as delay-line representations of time that are not biologically realistic for intervals in the range of seconds. The interval-timing literature provides several alternatives. One of them is that timing could emerge from general network dynamics, instead of coming from a dedicated circuit. Here, we present a general rate-based learning model based on long short-term memory (LSTM) networks that learns a time representation when needed. Using a naïve network learning its environment in conjunction with TD, we reproduce dopamine activity in appetitive trace conditioning with a constant CS-US interval, including probe trials with unexpected delays. The proposed model learns a representation of the environment dynamics in an adaptive biologically plausible framework, without recourse to delay lines or other special-purpose circuits. Instead, the model predicts that the task-dependent representation of time is learned by experience, is encoded in ramp-like changes in single-neuron activity distributed across small neural networks, and reflects a temporal integration mechanism resulting from the inherent dynamics of recurrent loops within the network. The model also reproduces the known finding that trace conditioning is more difficult than delay conditioning and that the learned representation of the task can be highly dependent on the types of trials experienced during training. Finally, it suggests that the phasic dopaminergic signal could facilitate learning in the cortex.

Appendices

Appendix A

1.1 Supplemental material 1: Java simulator

This program is a Java package containing the original simulation software used to perform the present simulations and can be downloaded from ModelDB (http://senselab.med.yale.edu/modeldb/) at http://senselab.med.yale.edu/ModelDB/ShowModel.asp?model=124329. No publication or commercial derivatives can be made without the author’s written consent. On the other hand, researchers who would like to have more options to try to make predictions and plan animal experiments are welcome to contact the author [FR]; model options, environmental or tasks options, as well as options to include other models can be discussed.

1.2 Supplemental material 2: Simulations data

Three MS Access .mdb files containing all recorded data for the present experiment can be downloaded from Université de Montréal institutional digital repository Papyrus (https://papyrus.bib.umontreal.ca/jspui/) at http://hdl.handle.net/1866/3073. We highly recommend contacting the author [FR] before making any inferences about the data.

The Master database contains the last 20 training blocks with the test blocks before, in between, and after them (there was one test block every 10 training blocks). These are then followed by an extra training block and a final control block. The PreTraining database contains 2 control blocks run on 30 untrained networks. The InitialTraining database contains 2 training blocks run on 30 untrained networks.

Appendix B

1.1 Theoretical p signal

A theoretical p signal can be computed to evaluate the internal model the networks have learned. We make few assumptions about the networks’ internal model of the task, compute a theoretical p signal for that model using TD, and compare it to the networks’ p signal.

We will assume that the networks are unable to keep track of time during the intertrial interval between the arrival of a US and the presentation of the next CS and that they have perfect knowledge of the task within a trial. We modeled this using a 7-state Markov model: the model begins in intertrial state; when the CS appears, it shift into a CS state followed by 5 other states representing 200 ms, 400 ms, 600 ms, 800 ms and 100 ms after CS onset; then the model automatically comes back in the intertrial state again. The p neurons learn an estimate of the sum of discounted future rewards, i.e.

$$ p \approx \sum {_{k = 0,\infty }{\gamma^k}{r_{t + k + 1}},} $$

γ = 0.98. To compute the theoretical p signal implied by the Markov model, we simply reran the simulations (without test blocks) using the same TD component as in the paper, but replacing its input by the 7-states model described above. We also decreased the TD learning rate on each training block (1% smaller) to ensure convergence.

This theoretical p signal converged to an expected value of 1.4 during intertrial intervals, increased from 2.0 to 2.1 from the CS onset to usual US arrival (Fig. 5, bottom row, middle panel, dashed line), and goes down to 1.2 on the US (1,000 ms) state. This means that in general, the networks could expect their sum of future rewards in a block, without more information, to be about 1.4. Once a trial has started, signalled by the arrival of CS, a more precise value that includes the imminent reward of value 1 can be computed, thus the sudden rise in the p estimate. The p value slowly increases as the expected reward gets closer in time due to discounting. Finally on US presentation, networks can expect less than in intertrial state since the US is never closely followed by a reward. Intertrial state estimate is less precise since all intertrial steps share the same estimate. In TD, the error is always pushed backward in time, since the only unpredictable even is the arrival of the CS, this is where most of the error signal (δ) appears, i.e. at CS presentation.

As shown on Fig. 5, the theoretical p signal and the averaged networks’ p signal are relatively close to each other.

Appendix C

1.1 Memory block responses

Some of the memory blocks in the 30 successful networks contained elements with unphysiological response patterns. In particular, the activity of some memory cells increased throughout the duration of entire training blocks, mainly because of sustained ramp increases during the intertrial intervals (Supplemental Figure 5). This increase probably would have continued indefinitely if the neurons’ activity was not reset between each training block.

A deeper analysis revealed that memory blocks with such memory cell behaviours did not contribute directly to the network output at time t, even though they had some phasic responses to the task’s signals. Of the 60 memory blocks (2 per network), 19 had one of the above patterns. No successful network had two memory blocks with such signals, indicating that it is probably impossible to learn the task with such signals only. Using a correlation measure between the memory block outputs and the LSTM outputs to evaluate their contribution to network function, 17 of the 19 memory blocks with unusual responses were removed from the LSTM representation analysis (memory blocks with absolute correlations <0.03 where removed from the analysis), leaving a total of 43 memory blocks (out of 60) analyzed. None of the 41 memory blocks considered normal were rejected by the correlation test. The finding that a number of networks learned the task even though one of their memory blocks had unphysiological response patterns and did not contribute significantly to network output indicates that one memory block is probably sufficient to learn this classical conditioning task.

In an extra experiment post-analysis, we ran a small number of networks using a single memory block and a single memory cell. Although the success rate of these networks was much lower (~6%), their final solution matched the one found in Section 3.5. This also suggests that blocks removed from the analysis were not contributing to the final solution and that the final solution probably stands within a single memory block most of the time.

Appendix D

1.1 Mesocortical performance

In order to verify that the mesocortical speed-up was not simply due to the addition of two constant learning rates, we trained 50 networks for each pair of hyper-parameters values in the range {0.0, 0.0032, 0.01, 0.0316, 0.1, 0.3162, 1.0, 3.1623}² (powers of square root of 10). We than reduce the space of hyper-parameters by looking only at the number of successful networks (networks still successful on the last training block, Supplemental Figure 6). With β = 0, there was at best 34% of successful networks (α _lstm  = 1). The averaged first successful epoch (using the 1,000 limit for the 13 that fails to have a successful block) was 480 blocks. Higher success rate with the mesocortical model (from 50% to 60%) were obtained within the range (α _lstm , β) ε {{0.0316, 0.1, 0.3162} × {0.3162, 1.0}}. In this range, the averaged first successful block of the fastest networks (α _lstm  = 0.3162 and β = 1.0) was 260 (only 6 networks failed to have a successful block and had a value of 1,000). Finally, we also looked whether networks could learn only using DA as learning rate (with no basic intrinsic learning rate, α _lstm  = 0.0). β = 0.3162 and β = 1.0 gave the best success rates (30% and 34% respectively), the fastest networks (β = 1.0) had an averaged first successful block of 616 meaning that in this task, DA alone, without a basic intrinsic learning rate, seemed sufficient to learn. We performed a one-way ANOVA on these 3 sets of networks (α _lstm  = 1.0 and β = 0.0, α _lstm  = 0.3162 and β = 1.0,α _lstm  = 0.0 and β = 1.0) and found a significant difference (P < 1.0E-6). A post-hoc Scheffe test showed that the mesocortical model using a basic learning was significantly faster than the two other groups.