Delving into Temporal-Spectral Connections in Spike-LFP Decoding by Transformer Networks

Abstract

Invasive brain-computer interfaces (iBCIs) have demonstrated great potential in neural function restoration by decoding intention from brain signals for external device control. Spike trains and local field potentials (LFPs) are two typical intracortical neural signals with good complementarity from time and frequency domains. However, existing studies mostly focused on a single type of signal, and the interaction between the two signals has not been well studied. This study proposes a temporal-spectral transformer network (TSNet) to model the temporal (with spikes), spectral (with LFPs), and mutual (with both signals) connections in spike-LFPs towards robust neural decoding. Experiments with clinical neural signals demonstrate that the attention-based connection model enables the dynamic temporal-spectral compensation in spike and LFP signals, which improves the robustness against temporal shifts and noises in neural decoding.

Appendices

Appendix

A Detail Settings Of Neural Decoders

We compare the performance of different neural decoders, including SVM, MLP, LSTM, CCA, DCCA, and ridge regression. The detail model settings are as follows:

• SVM: We use one-vs-rest (ovr) for multiple class classification and the regularization parameter C is chosen from $\left\{ 10^{-6}, 10^{-4}, \cdots , 10^{6}\right\}$. Spikes and LFPs are flattened to vectors as model input. For the fusion of spikes and LFPs, we concatenate the two signals as vectors and feed them to SVM.

• MLP: We use a one-layer MLP and the hidden size is selected from $\left\{ 50, 100, 200, 300, 400, 500\right\}$. For fusion, we train two individual models on spikes and LFPs, respectively, and concatenate the representation from them as the input of a one-layer MLP with the hidden size selected form $\left\{ 50, 100, 200\right\}$.

• LSTM: We use one layer LSTM whose hidden size is selected from $\left\{ 50, 100, 200, 300, 400, 500\right\}$ to extract the sequence’s representation and use the last hidden state as the final representation. The fusion strategy is the same as MLP.

• CCA: We select the linear projection dimension from $\{10, 20, 30, 40, 50, 100, 200\}$ and use an ECOC-SVM classifier as[22].

• DCCA: We use one layer neural network whose hidden size is selected from $\left\{ 50, 100, 200, 300, 400, 500, 800, 1000\right\}$ and the linear projection dimension and classification are the same as CCA.

• Ridge regression: The regularization strength $\alpha$ is chosen from $\left\{ 10^{-6}, 10^{-4}, \cdots , 10^{6}\right\}$.

Ten-fold cross-validation is used for parameter selection and performance evaluation. For the model training of MLP and LSTM, we set the batch size to 5 and use Adam as the optimizer with the learning rate of ${10^{-3}}$ and weight decay of ${10^{-4}}$. The loss function is cross-entropy loss. Early stop is applied to avoid overfitting.

B Estimating Movement Conduction Durations With Neuron Responses

We use neurons’ responses to estimate the true durations of movement intention conductions. We first select the neurons that respond strongly to specific motor tasks with the criterion of large response variation across the ’Go’ period. Fig. 4(a) shows two example electrodes that tune to “MouthOpen” and “RightToeTip”, respectively. Then we obtain the trial samples’ data for the selected channel and task and do min-max normalization to transform the responses into decimals between 0 and 1 which showed in Fig. 4(b) with the corresponding color. We mark the time that the firing rate is higher than 0.5 as the start of motor intention. Then we compare the firing rate at 800ms after the start. If the firing rate is below 0.5, we set the end time; if not, we search for the time when the firing rate was below 0.5 as the end. We plot the black dashed lines as the start and end marks in a trial in Fig. 4 (b). The connection weights are plotted along with the firing rate using the blue line.

Fig. 6.

Performance evaluation with Gaussian noises in spikes. (a) Dynamic connections adapt to noises. The first row illustrates spike signals with Gaussian masks. The second and third rows are the temporal (self-attention) and spectral-to-temporal (cross-attention) connection weights with different proportions of noises (solid lines) compared with weights without noises (dashed lines). (b) The average connection weights across trials. (c) Accuracy of different methods with Gaussian noises. (d) Accuracy descent ratio of different methods with Gaussian noises.

C Robustness To Gaussian Noises

Here, we analyze the result of masking a proportion of spike signals with Gaussian noise. In Fig. 6(a–b), we illustrate the adaptation of connection weights in temporal-spectral representations with different conditions of signal loss masking with Gaussian noise. Similar to results with signal loss using zero-masks, both the temporal (self-attention) and spectral-to-temporal (cross-attention) connection weights adjust dynamically to cope with changes in signals to improve the robustness and accuracy of neural decoding.

We evaluate the performance of the TSNet with different conditions of noises in spikes. Results are shown in Fig. 6(c) and Fig. 6(d). The results are consistent with the signal loss conditions. Firstly, the fusion of signals improves the robustness against noises. At the loss proportion of 40%, the accuracy descent using both signals outperform using spikes alone by 13.2%, 12.2%, 23.2%, and 16.2% in SVM, MLP, LSTM, and TSNet, respectively. Secondly, the TSNet achieves the best accuracy with high stability in spike-LFP fusion. As shown in Fig. 6 (c), TSNet achieves 77% of accuracy with the 40% loss proportion, outperforming LSTM and MLP by 6.8% and 4.0%, respectively. TSNet also achieves the lowest accuracy descent ratio (Fig. 6(d)), demonstrating our method’s robustness.