A novel heterogeneous transfer learning method based on data stitching for the sequential coding brain computer interface

Highlights

• A sequential coding experimental paradigm is studied by time-frequency analysis.

• A novel heterogeneous transfer learning method based on data stitching is proposed.

• Our proposed method is applicable to cross-label and cross-subject situations.

• The classification performance of our proposed method is better than various transfer learning methods.

Abstract

Objective

For the brain computer interface (BCI), it is necessary to collect enough electroencephalography (EEG) signals to train the classification model. When the operation dimension of BCI is large, it will bring great burden to data acquisition. Fortunately, this problem can be solved by our proposed transfer learning method.

Method

For the sequential coding experimental paradigm, the multi-band data stitching with label alignment and tangent space mapping (MDSLATSM) algorithm is proposed as a novel heterogeneous transfer learning method. After filtering by multi-band filtering, the artificial signals can be obtained by data stitching from the source domain, which build a bridge between the source domain and target domain. To make the distribution of two domains closer, their covariance matrices are aligned by label alignment. After mapping to the tangent space, the features are extracted from the Riemannian manifold. Finally, the classification results are obtained with feature selection and classification.

Results

Our data set includes the EEG signals from 16 subjects. For the heterogeneous transfer learning of cross-label, the average classification accuracy is 78.28%. MDSLATSM is also tested for cross-subject, and the average classification accuracy is 64.01%, which is better than existing methods.

Significance

Combining multi-band filtering, data stitching, label alignment and tangent space mapping, a novel heterogeneous transfer learning method can be achieved with superior performance, which promotes the practical application of the BCI systems.

Introduction

Brain-computer interface (BCI) is a novel human-computer interaction system. Without relying on surrounding nerves and muscles, the brain activity can be transformed into the controlling signals of external devices by the BCI systems [[1], [2], [3]]. As the development of neuroscience and information technology, the BCIs have more and more application value. With the help of BCIs, the mechanical exoskeletons and the electric wheelchairs are directly controlled by the paralyzed patients’ brains [4]. Therefore, the patients can get more high-quality rehabilitation training. As a result, their recovery progress and quality of life can be both greatly improved. BCIs are also used as enhancement tools for healthy people [1], and their applications include robot control, word spelling and intelligent home control. In addition, the BCI systems have a wide range of applications in some special environment, such as education and military [3].

With the non-invasive, low risk, and ease of use, electroencephalography (EEG) is widely applied to record the brain activity [5]. Because the EEG signals have temporal, spatial, and frequency characteristics, various experimental paradigms have been developed for the BCI systems [[5], [6], [7]], such as steady-state visual evoked potentials (SSVEP), P300 evoked potentials, motor imagery (MI) and mental tasks. SSVEP is a passive experimental paradigm [8], and it takes full advantage of the frequency characteristics of the EEG signals. With different frequency variations of flickering stimuli, SSVEP is evoked in the visual function area of the brain. Another passive experimental paradigm is P300 [9], and it makes use of the temporal characteristics of EEG. MI and mental tasks are active experimental paradigms, and they do not require external stimulation. Compared with the passive BCIs, these kind of BCIs can be more autonomously manipulated [10]. When the motor imagery tasks are performed, the phenomenon of event-related desynchronization/synchronization (ERD/ERS) of the EEG signals will happen. Speech imagery is a high-level thinking activity, and it involves many cerebral cortices of the brain. Therefore, the signals of speech imagery have obvious spatial features. During the no-motion state, the imaginary vowels/a/and/u/have been used to induce the speech imagery [11]. In addition to the vowels, short words and long words have been also studied as the contents of speech imagery [12].

The EEG signals are time-varying signals, but their spatial resolution and signal-to-noise ratio is low. As a result, the number of instruction sets of active BCIs is very limited. Involved in motor imagery, the imaginary contents mainly include left hand, right hand, foot, and tongue [13]. Making use of the high temporal resolution of EEG signals, their low spatial resolution can be compensated to some extent. After analyzing decomposed hand and foot movements during simulated drumming, four types of limb motor imagery are proposed with sequential compound [14]. In order to realize four types of imaginary control, sequential imaginative tasks of left-hand and right-hand are proposed to control the wheelchair [15].

The experimental paradigms above all have a single type of imaginary content. In recent years, the experimental paradigms have been gradually extended from single paradigms to hybrid paradigms [16]. The number of instruction sets of BCIs can be increased by the hybrid paradigms, and their usefulness is also improved. Based on the above advantages, the hybrid paradigms have become a popular research object in the field of BCI systems. Consisting of P300 and SSVEP, a hybrid paradigm is proposed for the BCI systems [17]. In order to get more aspects of EEG features, another hybrid paradigm based on motor imagery and SSVEP is also proposed [18]. Higher classification accuracy and more operational instructions of the BCIs can be achieved by the hybrid paradigms. However, most of the existing hybrid BCIs combine active and passive experimental paradigms, and they still rely on external stimulation devices. Spontaneous EEG signals can be both induced by motor imagery and speech imagery. For the EEG signals, there are significant differences between them. Therefore, a hybrid experimental paradigm has been proposed with motor imagery and speech imagery in our previous study [19]. In daily life, language and movement can be switched very naturally. For example, weightlifters usually shout before lifting the barbells. With shouting, their morale and attention can be improved. This phenomenon reminds us that motor imagery and speech imagery can collaborate with each other in chronological order. Besides, the EEG signals with more abundant information can be obtained by the sequential coding of motor imagery and speech imagery. Therefore, without relying on external stimuli, proposing a novel hybrid experimental paradigm to increase the operational instructions is the first motivation of this paper.

In order to convert the EEG signals into the control instructions of BCIs, signal processing, feature extraction and machine learning algorithms are required. As the time series signals, the EEG signals also have useful features in the spatial domain and frequency domain. To extract the spatial features, common spatial pattern (CSP) is used [20]. The performance of feature extraction is strongly related to the frequency bands of the subjects’ EEG signals. Therefore, relying on specific frequency bands, filter bank common spatial pattern (FBCSP) and sub-band common spatial pattern (SBCSP) are proposed to extract more separable spatial features, respectively [21,22]. Multi-band filtering is also used by the multi-band tangent space mapping with sub-band selection (MTSMS), and higher classification accuracy is obtained [23]. Multi-band filtering can subdivide the frequency features of EEG signals, and it is helpful to improve the effect of feature extraction. After feature extraction, the next key step is feature classification. The commonly used classifiers are support vector machine (SVM), linear discriminant analysis (LDA) and so on [24].

For our novel hybrid experimental paradigm with more types of imagery, the process of collecting experimental data is too long. After participating in the long experiment, subjects may feel tired. The classification accuracy of EEG signals may be also reduced. Several effective methods have been proposed to reduce the data collection time, and one of them is transfer learning [[25], [26], [27], [28], [29], [30]]. Transfer learning has been gradually applied to the BCI systems in recent years. After learning the features in the source domain, the learned features can be applied by transfer learning to the target domain. At last, the amount of data collected in the target domain can be reduced. According to the features of source and target domains, transfer learning can be divided into homogeneous transfer and heterogeneous transfer. For our novel hybrid experiment paradigm of this paper, heterogeneous transfer learning is very suitable. The existing transfer learning techniques mainly consider the case of different feature spaces [[31], [32], [33], [34]]. For the heterogeneous feature space, active weighted adaptation regularization (AwAR) is proposed [31]. The signals of source domain and target domain come from different EEG caps. In addition, as a kind of transfer learning, some domain adaptation (DA) algorithms are also suitable for heterogeneous label space [[35], [36], [37]]. Busto et al. propose the concept of open set DA for the first time [35], and it can be applicable to more realistic situations. The source domain and target domain have some common known categories and unknown categories, respectively. With the adversarial training, Saito et al. study another case of open set DA [36]. The target domain has all categories of the source domain, but there are unknown categories in the target domain that are not in the source domain. You et al. propose the concept of universal DA [37]. If the label category of the target domain belongs to a known category of the source domain, the samples of the target domain can be classified. Otherwise, the samples of the target domain are marked as unknown categories. When the label spaces of the source and target domains are different, existing transfer learning methods are not easy to implement. Therefore, with a novel heterogeneous transfer learning method, the second motivation of this paper is to reduce the amount of data collected. High classification accuracy is one of the key factors to determine the practicability of BCI systems, and the third motivation is that our proposed transfer learning model can achieve high classification accuracy.

As a novel hybrid experimental paradigm, the sequential coding experimental paradigm is proposed to increase the operational instructions. It includes single imagination tasks and sequence combination imagination tasks. To reduce the time of data collection, based on data stitching, the multi-band data stitching with label alignment and tangent space mapping (MDSLATSM) algorithm is proposed as a novel heterogeneous transfer learning method. With data stitching, the EEG signals of the source domain are spliced into the artificial signals, and the degree of similarity in the EEG signals between the source domain and target domain can be enhanced. To obtain high classification accuracy, MDSLATSM performs multi-band filtering, data stitching, label alignment (LA) [38], tangent space mapping, feature selection and feature classification in turn. In the preprocessing steps of MDSLATSM, the EEG signals are band-pass filtered into multiple sub-bands at first, and then data stitching is implemented. The spliced signals of each sub-band are aligned to the space of target domain by LA. Therefore, the distribution difference between the source domain and target domain is reduced further. With the framework of Riemannian geometry, the tangent space features of EEG signals are extracted. As the high-dimensional features, more representational features of each sub-band should be selected. These features are dimensionally reduced by the minimum Redundancy Maximum Relevance (mRMR) and principle component analysis (PCA) in turn [39,40]. Finally, the features are classified by LDA [41].

The main contributions of our paper summarized are as follows:

• Combining motor imagery and speech imagery in chronological order, a novel sequential coding experimental paradigm is proposed. The number of instruction sets of BCIs can be increased by the experimental paradigm.

• Based on data stitching, MDSLATSM is proposed as a novel heterogeneous transfer learning method. According to the temporal characteristics of our experimental paradigm, the artificial signals can be obtained by data stitching from the source domain, which build a bridge between the source domain and target domain.

• The classification performance of MDSLATSM is better than various existing transfer learning methods. A satisfactory result is obtained by the combination of multi-band filtering, data stitching, label alignment and tangent space mapping.

The following section introduces the experimental paradigm and setup. Section 3 introduces the related theories and system framework of MDSLATSM. Section 4 describes the comparison and analysis of the results of MDSLATSM with other transfer learning algorithms. Section 5 discusses the proposed method, limitations and future work. Conclusions are drawn in Section 6.