Identifying Good Donor Datasets for Transfer Learning Scenarios in Motor Imagery BCI

Abstract

We present a simple deep learning-based framework commonly used in computer vision and demonstrate its effectiveness for cross-dataset transfer learning in mental imagery decoding tasks that are common in the field of Brain-Computer Interfaces (BCI). We use this framework to characterize the compatibility for transfer learning between twelve motor-imagery datasets.

Challenges. Deep learning models typically require long training times and are data-hungry, which impedes their use for BCI systems that have to minimize the recording time for (training) examples and are subject to constraints induced by experiments involving human subjects. A solution to both issues is transfer learning, but it comes with its own challenge, i.e., substantial data distribution shifts between datasets, subjects and even between subsequent sessions of the same subject.

Approach. For every pair of pre-training (donor) and test (receiver) datasets, we first train a model on the donor dataset before training merely an additional new linear classification layer based on a few receiver trials. Performance of this transfer approach is then tested on other trials of the receiver dataset. We compile these results in a table characterizing the compatibility between the different datasets.

Significance. Our characterisation of compatibility between datasets can be used as a reference for future researchers to make informed donor choices. To strengthen this claim, we present an applied example of such usage. Finally, we lower the threshold to use transfer learning between motor imagery datasets: the overall framework is extremely simple and nevertheless obtains decent classification scores.

A Appendix

1.1 A.1 Description of the Datasets

Below, we provide the list of datasets we used in this study together with a link to a concise online description on the MOABB website. For the full description of each dataset, we recommend the reader to refer to the corresponding original publications cited in Table 1.  

AlexMI:

http://moabb.neurotechx.com/docs/generated/moabb.datasets.AlexMI.html#moabb.datasets.AlexMI

BNCI2014001:

http://moabb.neurotechx.com/docs/generated/moabb.datasets.BNCI2014_001.html#moabb.datasets.BNCI2014_001

BNCI2014004:

http://moabb.neurotechx.com/docs/generated/moabb.datasets.BNCI2014_004.html#moabb.datasets.BNCI2014_004

BNCI2015001:

http://moabb.neurotechx.com/docs/generated/moabb.datasets.BNCI2015_001.html#moabb.datasets.BNCI2015_001

BNCI2015004:

http://moabb.neurotechx.com/docs/generated/moabb.datasets.BNCI2015_004.html#moabb.datasets.BNCI2015_004

Cho2017:

http://moabb.neurotechx.com/docs/generated/moabb.datasets.Cho2017.html#moabb.datasets.Cho2017

Lee2019 MI:

http://moabb.neurotechx.com/docs/generated/moabb.datasets.Lee2019_MI.html#moabb.datasets.Lee2019_MI

Ofner2017:

http://moabb.neurotechx.com/docs/generated/moabb.datasets.Ofner2017.html#moabb.datasets.Ofner2017

PhysionetMI:

http://moabb.neurotechx.com/docs/generated/moabb.datasets.PhysionetMI.html#moabb.datasets.PhysionetMI

Schirrmeister2017:

http://moabb.neurotechx.com/docs/generated/moabb.datasets.Schirrmeister2017.html#moabb.datasets.Schirrmeister2017

Weibo2014:

http://moabb.neurotechx.com/docs/generated/moabb.datasets.Weibo2014.html#moabb.datasets.Weibo2014

Zhou2016:

http://moabb.neurotechx.com/docs/generated/moabb.datasets.Zhou2016.html#moabb.datasets.Zhou2016

 

1.2 A.2 Detailed compatibility tables

The AUC scores for the feet vs. right-hand classification task for 1, 2, 4, 8, 16, 32, and 64 calibration trials per class can respectively be found in Tables 2a, 3a, 4a, 5a, 6a, 7a, and 8a. The AUC scores for the left-hand vs. right-hand classification task can be found in Tables 2b, 3b, 4b, 5b, 6b, 7b, and 8b. Finally, the accuracy scores on all the classes from the test dataset can be found in Tables 2c, 3c, 4c, 5c, 6c, 7c, and 8c.

We present here the tables in which we collected our results. These results can be used to estimate the compatibility for transfer learning between the different motor imagery datasets we analysed. Due to editorial reasons, the compatibility tables had to be split according to the number of calibration trials per class and needed to be provided in black and white. A colour and un-split version of these tables is available online⁵.

Table 2. Compatibility tables when using 1 calibration example per class. Boldface values indicate the best score in each column.

Table 3. Compatibility tables when using 2 calibration examples per class. Boldface values indicate the best score in each column.

Table 4. Compatibility tables when using 4 calibration examples per class. Boldface values indicate the best score in each column.

Table 5. Compatibility tables when using 8 calibration examples per class. Boldface values indicate the best score in each column.

Table 6. Compatibility tables when using 16 calibration examples per class. Boldface values indicate the best score in each column.

Table 7. Compatibility tables when using 32 calibration examples per class. Boldface values indicate the best score in each column.

Table 8. Compatibility tables when using 64 calibration examples per class. Boldface values indicate the best score in each column.

1.3 A.3 Experimental Details of the Example Application

We attempted to reproduce the results that Xie and colleagues [19] presented in their Sect. 4.1. (entitled Classification accuracy of the DL models when fine-tuning on the whole training set), in particular their Table 2, of their article. Because the corresponding source code was not published, we had to re-create the experimental settings according to what was reported in the article. Unfortunately, we faced obstacles in this process. We will describe them in this section.

The first difficulty was to establish which fine-tuning scheme had been used in the experiment of Sect. 4.1. In Sect. 2.3, four different fine-tuning schemes were presented but it was not clear, from the text, which one was used to obtain the results in Table 2. Later in the article, the four fine-tuning schemes are compared (only with Schirrmeister2017 as donor) in Table 4 and it seems that the results of the fourth scheme are relatively similar to those presented in column “PT_by_HGD” of Table 2, despite not being perfectly equal. Therefore, we made the assumption that the whole Table 2 was obtained using fine-tuning scheme 4 and also used this scheme in our replication.

Then, it is mentioned that the same training strategy as Schirrmeister et al. [37] was used. However, the article follows up by explaining a training strategy different from the one presented by Schirrmeister and colleagues. Furthermore, the strategy of Xie and colleagues reads as “In the first stage, only training sets were used for model training. When the loss on the training set does not decrease for some epochs, we stop training and record the lowest loss on the training set. Then, training and validation sets were used to train the model and stop training when the loss on the validation set dropped to the same value as the lowest loss at the first training stage.”. This suggests that the early-stopping mechanism of their first stage may not take into account the validation loss. We assumed this was a simple inaccuracy in the textual description and chose to implement the training strategy initially described by Schirrmeister and colleagues. Additionally, the number of epochs without improvement to wait before the end of the first stage was not provided. We chose to utilize the same value for early stopping as previously mentioned in the text but in another context, i.e., 100 epochs.

Concerning the cross-validation procedure, it is mentioned that “Each configuration was evaluated twenty times using different random seeds”. However, it was not clear whether the different folds over a given test subject were all based on the same pre-trained model or if a new pre-training was done for each fold. We choose the first option as it is significantly faster to compute.

Table 9. Number of examples in the training, validation and testing sets of the fine-tuning phase in our reproduction of the work of Xie et al. (2023), Sect. 4.1. The eventual “$\,\cdot \,\pm \,\cdot \,$” indicate the average and standard deviation in case the number of examples is not constant across subjects within a dataset.

The construction of the train/validation/test data splits was also relatively obscure. Because of the absence of source code, the exact data split is unknown. For pre-training, it is mentioned in their Sect. 3.4.1. that the data of all the subjects was “divided into 90% training and 10% validation sets”. We assumed the examples from all subjects were first shuffled, leading to having all subjects present in both sets. Nevertheless, it would be highly unlikely that we reached the exact same split as the original authors. For fine-tuning, it is mentioned in Sect. 4.1. that the “train-test splition is the same as in the BCI Competition IV and the original paper [37]”. These train/test splits have been annotated in the MOABB library, so we used the library’s annotations to obtain the train/test splits. Then, it is stated that the training data is split into “80% training set and 20% validation set”. For each subject, we split the training data 20 times, each with a different random seed and in a stratified way, to create 20 folds. Again, it would be highly unlikely that we reached the exact same splits as the original authors. For sanity check, we reproduced the Table 1 of the original work compiling the amount of data in the training, validation and testing sets of each dataset. This leads to our Table 9. We can observe that the numbers in the two tables do not match but we suspect that the main difference can be explained by a confusion in the author’s table. Indeed, their line “Our dataset” is relatively similar, except regarding N_Subjects, to the Schirrmeister2017 line in our Table 9. The other disagreements are much smaller and could be explained by a difference in the counting methods. Regardless, we conducted our experiments despite the potential disagreement between the two tables.

Finally, some choices seemed rather unconventional to us. First, the band-pass frequency filtering is done after the epoching steps which can cause edge artefacts. Second, the signal is passed at 250 Hz to the neural networks. While Deep ConvNet and Shallow ConvNet were indeed developed for signals at this frequency [37], EEGNet was developed for signals at 128 Hz [23]. It would typically be recommended to adapt the hyperparameters of EEGNet to use it on such signals. Third, an electrode-wise exponential moving standardisation (EMS) step is also applied after epoching using an initialisation on the first 1000 samples. At 250 Hz, this means that the EMS is initialized on an interval of 4 s and only “moves” over the last 0.5 seconds of each epoch. Nevertheless, we implemented these preprocessing steps the way they were described in the original article.

The rest of the experimental settings strictly follow what was described in the original article by Xie and colleagues [19]. The source code of our attempt to reproduce these results, along with the list of libraries we used with their versions, can be found on github⁶.