Multi-source manifold feature transfer learning with domain selection for brain-computer interfaces

Abstract

Transfer learning uses the knowledge in source domains to improve the learning performance in the target domain, which is useful in electroencephalogram (EEG) based brain-computer interfaces (BCIs) with small training datasets. However, the existing transfer learning methods for EEG based BCI mainly consider the knowledge transfer from single-to-single (STS) domain or simply merge different source domains into a bigger one. In this paper, we propose a multi-source manifold feature transfer learning (MMFT) framework to transfer multi-source knowledge for EEG signals classification. MMFT minimizes marginal probability distribution on the Riemannian manifold using Riemannian alignment and Grassmann manifold feature learning, then transfers the manifold features with a conditional probability distribution adaptation in the structural risk minimization (SRM) function. Based on MMFT, w-MMFT is proposed to tackle the class imbalance issue for SRM, and the label similarity analysis (LSA) is proposed to select source domains for MMFT, forming a new LSA-MMFT framework. Experimental results on six datasets demonstrate that the proposed MMFT has achieved superior performance in classification accuracy and computational efficiency compared to state-of-the-art methods. The LSA-MMFT can get more stable performance than two other domain selection methods.

Introduction

A brain-computer interface (BCI) offers a direct communication pathway for users to interact with the environment, via their brain signals containing abundant information of the users’ cognitive state or intentions [1]. BCI has great research importance and application value in many fields, such as auxiliary control, entertainment and rehabilitation training [2]. Electroencephalogram (EEG) is the most widely used technology for BCI systems, because of its non-invasive nature and high temporal resolution. Different neurophysiological paradigms of EEG have been used to operate BCIs, such as motor imagery (MI), event-related potentials (ERPs) and steady-state visual evoked potentials (SSVEP) [3]. In MI tasks, the users imagine movements of their body parts (e.g., hands, feet, and tongue), without actually performing the movement and even without tensing the muscles, which causes the modulations of brain rhythms in the involved cortical areas. In ERP tasks, the user is stimulated by a majority of non-target stimuli but a few target stimuli, wherein a specific ERP pattern appears in the EEG response after the user perceives a target stimulus. However, EEG signals are notoriously difficult to analyze due to the low signal-to-noise ratio and significant subject-to-subject variations. Therefore, it is of great interest to be able to extract and recognize EEG features with machine learning methods with shallow and deep structures [4], [5], [6], [7], [8].

Despite several recent advances, most of the BCI systems are still faced with a major challenge of long calibration time. Different users have different neural responses to the same stimulus, and even the same user differs the response to the same stimulus, especially at different time/locations. Besides, calibrating the BCI system require sufficient labeled data to train the subject-specific BCI model, and thus the calibration phase is usually time-consuming. Transfer learning is helpful for BCIs to conquer the abovementioned challenges [9]. The knowledge in one or more source domains is fully utilized to help the classification in a target domain, which improves the calibration performance. And cross-subject classification has become one of the most popular directions where the generalization ability (of the classification model) can be enhanced by improving the cross-subject performance. Recently, there is a trend to utilize the covariance matrices of EEG trials, which are symmetric positive definite (SPD) and can be viewed as points on an SPD manifold, the covariance matrix can boost the role of Riemannian geometry in BCIs. The Riemannian alignment (RA) framework was proposed to align EEG covariance matrices from different source domains, which can quickly reduce the distribution difference between the source and target domains by changing the reference position on an SPD manifold [10]. However, the Riemannian alignment approaches are computationally expensive, and not compatible with the machine learning approaches defined in Euclidean space.

Transfer learning methods for BCI usually aim to solve single-to-single (STS) transfer problems. In practice, it is noted that a good source domain can help to obtain a high classification accuracy, even with a simple transfer learning algorithm. Therefore, the quality of the source domain is important. However, multiple source domains are factually available for EEG signal classification, such as labelled data from other subjects or other sessions used before. In multiple source domains cases, good source domains are more likely to be found. Due to the data expansion, in multi-source transfers, good source domains can reduce the negative transfer (NT) caused by bad source domains. Therefore, multi-source transfer learning usually achieves more stable and higher classification accuracy than STS transfer learning. Multi-source unsupervised domain adaptation is proved to be valuable in performance improvement [11], [12], [13].

Despite the progress mentioned above, there are some unresolved difficulties and challenges in the field. Firstly, traditional transfer learning methods usually seek to adapt marginal and conditional probability distributions [14], [15]. As an efficient way for EEG transfer, the Riemannian alignment approaches usually aligns distribution centers and cannot completely minimize the marginal probability distribution difference. If the aligned features can be transferred by traditional transfer learning methods further, marginal probability distribution still need to be adapted in the subspace. Moreover, the existing transfer learning methods for BCI are lack of information exploration between multi-source domains. It is necessary to propose a multi-source transfer learning framework with simple structure which can integrate traditional transfer learning method and Riemann alignment approach. Secondly, class imbalance often exists in many BCI scenarios. Most transfer learning methods ignore this issue by treating the classes as balanced across domains, or they only handle the bias in one domain [14], [16], and this may hinder the effectiveness of transfer learning. Therefore, how to handle the class imbalance situation in transfer learning for BCI is necessary. Thirdly, to overcome NT, many domain selection methods were proposed but most of them are based on mathematical similarity calculation [17], [18], which aims to explore the relationship between one source domain and the target domain, but lacks the exploration of the relationship between multiple source domains.

Different from the existing results, this paper presents a multi-source manifold feature transfer learning (MMFT) framework with simple structure and strong stability. More specifically, the distribution means of source and target domains are aligned on the SPD manifold, and then the tangent space features are extracted using tangent space mapping in the proposed MMFT. Next, the tangent space features are reconstructed to Grassmann manifold features via the Geodesic flow kernel (GFK) approach [18]. Finally, the classifier of MMFT is trained by adapting condition probability distribution in the structural risk minimization (SRM) function. Compared with existing multi-source transfer learning algorithms, the new method has several noteworthy advantages:

• 1) MMFT minimizes the marginal probability distribution on the Riemannian manifold, and then transfers the manifold features with conditional probability distribution adaptation. To the best of our knowledge, it’s the first attempt to align the two probability distributions separately in the field of EEG signal classification, where the calculation of their weights is avoided. Additionally, the MMFT has a simple structure with only three hyperparameters to be determined, and classification accuracy is not sensitive to the changes of these hyperparameters.

• 2) To better align conditional probability distributions for each pair of source and target domains, the MMFT utilizes a voting mechanism to transfer knowledge from source domains to the target domain individually, instead of transferring all source domains together.

• 3) To tackle the class imbalanced scenarios, a weighted MMFT algorithm is proposed which boosts the performance of MMFT for imbalanced training datasets.

• 4) To reduce the negative transfer in multi-source transfer learning, a label similarity analysis for MMFT (LSA-MMFT), is proposed for domain selection, which improves the classification accuracy and the time efficiency of MMFT.

The remainder of this paper is organized as follows. Section 2 introduces related work on Riemannian geometry, subspace adaptation and domain selection. Section 3 describes the details of the proposed MMFT, w-MMFT and LSA-MMFT. In Section 4, the experiments verify the performance of MMFT compared with several state-of-the art transfer learning approaches, and the performance of LSA-MMFT with two other state-of-the art domain selection approaches. Finally, the conclusion is given in Section 5.