Ensemble learning method based on temporal, spatial features with multi-scale filter banks for motor imagery EEG classification

https://doi.org/10.1016/j.bspc.2022.103634Get rights and content

Highlights

Abstract

Decoding motion intention from electroencephalogram (EEG) is a key part of the motor imagery-based brain-computer interface (MI-BCI). To help the disabled with neuromuscular disabilities to restore motor function through BCI, it is necessary to build an efficient and stable classification algorithm to decode the motor intention contained in the EEG signal. However, EEG signals are non-stationary and vary greatly between individuals. In this work, we propose an ensemble learning method based on temporal, spatial features and multi-scale filter banks, called TSMFBEL, which aims to design an ensemble classifier model with strong generalization capabilities for MI EEG classification. To obtain diverse ensemble classifiers, the original EEG data are divided into subsets with sample diversity by bootstrap sampling method, and then decomposed into time–frequency subsets with time–frequency distribution diversity by multi-scale filter banks method. For each time–frequency subset, features with domain diversity are extracted from temporal domain, spatial domain and temporal-spatial domain, and heterogeneous classifiers with diversity are trained based on each set of features. To obtain the optimal decision, we describe the ensemble strategy as a minimum classification error optimization problem, and propose an ensemble classifier weight optimization method based on the L2-norm, and finally integrate the decision of the ensemble classifier by weighted fusion. The proposed method was evaluated on two public datasets (BCI Competition IV Dataset IIa and BCI Competition IV Dataset IIb), and the results are compared with the classification method of the state-of-the-art methods. Experimental results show that the proposed TSMFBEL algorithm can effectively construct a diversified ensemble classifier, and the average classification accuracy on the two datasets is 88.80% and 86.53% respectively, which are the highest among the state-of-the-art methods, and the standard deviation of the results is also the lowest. Excellent classification performance shows that the proposed algorithm has great potential in the decoding of MI EEG signals.

Introduction

Brain-computer interface (BCI) is a promising way of communication, which can establish communication channels between brain and external environment without resorting to peripheral nerve and muscle pathways [1]. BCI system is commonly used in the medical field and can be used to help patients suffering from severe neuromuscular injuries, such as spinal cord injury, amyotrophic lateral sclerosis, brain stroke and cerebral palsy [2]. Among the non-invasive methods, electroencephalogram (EEG) has the advantages of safety, simplicity and low cost, and is considered to be one of the most practical methods in BCI [3]. When the patient imagines moving any part of the body, it will trigger the neurons in a specific area of the brain to produce regular electrical activity, so that the amplitude of the neuroelectric signal on the sensorimotor cortex on the opposite side will decrease, and the amplitude of the neuroelectric signal on the sensorimotor cortex on the ipsilateral side will increase [4], [5]. This movement-related rhythmic activity change is called event-related desynchronization/synchronization (ERD/ERS) phenomenon [6]. The ERD/ERS phenomenon generated by motor imagery (MI) is very similar to actual motion, so it can be converted into computer instructions for controlling external devices by decoding the motion intention information in the EEG [7], [8]. Some excellent MI-based BCI applications have been developed to assist patients in recovering some daily motor functions and improving the quality of life, such as prosthesis [9], exoskeletons [10], [11], [45], and wheelchairs [12].

However, the raw recorded EEG signal usually contains a lot of noise, resulting in a low signal-to-noise ratio and non-stationarity of the signal [13]. In addition, the spatial location, start time, and stability of the ERD/ERS phenomenon in the brain are very different among different subjects [4], [14], which makes it a great challenge to accurately decode the motor intention contained in the EEG.

To accurately decode the motion intention in the MI EEG, many effective decoding algorithms have been proposed [14], [16], [15], [17], [18], [19]. Common spatial pattern (CSP) algorithm is one of the most commonly used algorithms in MI EEG [14]. CSP algorithm extracts features with maximum difference from binary classification signals through a set of optimal spatial filters. However, the classification performance of the CSP algorithm largely depends on the choice of frequency range [15]. CSP algorithm can achieve good classification results in the frequency range of 8–30 Hz, but existing studies have proved that refining the frequency band of CSP for different subjects can improve the classification performance of MI EEG [16]. Novi et al. [17]. proposed a sub-band CSP (SCSP), which extracts reliable CSP features from multiple small frequency bands, and then uses a score fusion strategy for EEG classification. Ang et al. [18]. proposed the filter bank common spatial pattern (FBCSP), which divides a wide frequency band into multiple small frequency bands, and then autonomously selects the discriminative CSP features from the small frequency bands. However, the above method does not further explore the time information of EEG. Zhang et al. [19]. proposed a temporally constrained sparse group spatial pattern (TSGSP), which can optimize the filter frequency band and time window of the CSP algorithm at the same time, and further improve the classification accuracy of MI EEG. Peterson et al. [20]. proposed a penalized time–frequency band CSP (PTFBCSP), which extracts CSP features in each time–frequency sub-band, and then selects the best subject-specific time–frequency band.

Existing studies show that different feature extraction methods have their own advantages, and the combination of multiple types of features can usually achieve higher classification accuracy than using a single type of feature [21], [44]. Lee et al. [22]. recorded MI EEG data from 10 subjects to analyze the performance of spatial, spectral and temporal features on MI-BCI, and showed that temporal and spatial features were superior to spectral features. Al-Qazzaz et al. [23]. proposed to extract features from temporal domain, entropy and frequency domain to characterize the EEG of the MI-BCI, and verified that the fusion of multiple features has higher classification performance than the single feature method.

Most of the above research work is implemented by using a single classifier. Some studies show that by using the idea of ensemble learning multi-classifier decision, the result of training multi-classifier ensemble can obtain better classification performance and robustness than that of single classifier [24], [25]. Zhang et al. [26]. input the extracted time-domain, frequency-domain, time–frequency domain, and spatial domain features into the ensemble LDA classifier, verifying that multiple domain features can complement each other and improve decoding performance. Deng et al. [27]. proposed a random subspace ensemble learning method based on multi-domain features, which used the extracted multi-domain features respectively to train ensemble classifiers, and verified that the classification accuracy of multi-domain ensemble method was higher than that of single-domain ensemble method. The key of ensemble learning to obtain excellent classification performance is to train diverse classifiers [28]. Zuo et al. [29]. proposed a cluster decomposing based ensemble learning framework (CDECL), which obtained different sub-data sets through cluster decomposition, and then trained the ensemble learning heterogeneous classifier model on each sub-data set. In the existing research work, the ensemble classification model is only constructed by multi-domain feature methods, or constructed by subdividing the frequency band of EEG signals. However, the advantages of temporal-spatial information and multi-domain feature information are not complementary to obtain the ensemble learning model with better generalization performance.

In this paper, we first evaluate the algorithms in the preprocessing stage and feature extraction stage of MI EEG analysis, and the results show that the algorithm combinations in different stages have great differences in the classification performance of the subjects. Then, we propose an ensemble learning method based on temporal, spatial features and multi-scale filter banks, called TSMFBEL, to improve the classification performance of MI EEG. For the raw EEG signal, we use the bootstrap sampling method to sample data subsets with differences in sample distribution, and design four time–frequency decomposition methods to preprocess each data subset to obtain time–frequency subsets with time–frequency differences. Then extract discriminative features from multiple domains to train a diverse ensemble classifier. Finally, a classifier weight optimization method based on the minimum error rate of the L2-norm is proposed, and the decision of the ensemble classifier is merged by weighted summation. We evaluated the classification performance of the proposed TSMFBEL algorithm on two public data sets, and compared the results with the state-of-the-art papers. The results show that our proposed TSMFBEL method has better classification performance. The main innovations and contributions of our work are summarized as follows:

  • 1.

    Multi-scale temporal-spatial preprocessing method is applied to time–frequency decomposition of EEG, which increases the temporal-spatial diversity of EEG subset. Feature extraction methods in spatial, temporal and temporal-spatial domains are used to extract discriminative features and improve the domain diversity of features.

  • 2.

    The ensemble learning framework proposed by us combines the time–frequency preprocessing method and feature extraction method, and the trained ensemble classifier model has the diversity of sample distribution, time–frequency diversity, and domain diversity.

  • 3.

    An ensemble classifier weight optimization algorithm based on L2-norm is designed, which can effectively alleviate the over-fitting problem in the weight learning process and improve the robustness of multi-classifier decision fusion.

  • 4.

    The proposed TSMFBEL algorithm achieved 88.80% and 86.53% average classification accuracy on BCI Competition IV Dataset IIa and Dataset IIb respectively, which is the highest among the state-of-the-art methods.

The organization of this paper is as follows: Section 2 introduces two public data sets and the proposed TSMFBEL algorithm in detail. Section 3 describes the experimental setup and presents the experimental results. Section 4 discusses the results of this study. Section 5 draws the conclusion and future work.

Section snippets

Dataset Description

In this study, we selected two public datasets to evaluate the performance of our proposed method.

BCI Competition IV Dataset IIa: This dataset records 22 channels of MI EEG signals of 9 subjects at a sampling rate of 250 Hz (subject ID: ”A01”, ”A02”, , ”A09”). During the experiment, each subject was required to perform four types of MI tasks based on visual cues: left-hand, right-hand, foot and tongue. The experiment is divided into two sessions, and each session contains 288 experiments. That

Experimental Setup

In the preprocessing step, for dataset IIa, the single-band method performs band-pass filtering at a frequency of 8–35 Hz, and the multi-band method divides the frequency range of 4–40 Hz into a series of sub-bands with a size of 4 Hz and an overlapping size of 2 Hz, (i.e., 4–8 Hz, 6–10 Hz, , 36–40 Hz). The single-time window method selects the EEG data of 0-3s during the execution of the MI task, and the multi-time window method divides the 0-3s time into a series of time windows with a

Discussion

In this study, we analyzed the algorithms of each step in the EEG signal processing process, and found that the subjects have great differences in the selection of time–frequency bands and feature extraction in spatial–temporal domain. Subject B02 can obtain more discriminative feature information in the spatial–temporal features of a single frequency band and single time window, while subjects B06 and B08 have more discriminative feature information in the spatial features of multiple

Conclusion

In this study, we propose an ensemble learning method based on temporal, spatial features and multi-scale filter banks, called TSMFBEL. To construct a diverse ensemble learning classifier, we constructed an EEG time–frequency subset with sample diversity and time–frequency distribution diversity from a limited data set, and then discriminative features are extracted from multiple domains to train diverse ensemble classifiers. Finally, the final result is obtained by weighted fusion of the

CRediT authorship contribution statement

Liangsheng Zheng: Writing - original draft, Conceptualization, Methodology, Software, Formal analysis. Wei Feng: Writing - review & editing, Supervision. Yue Ma: Writing - review & editing, Methodology, Validation. Pengchen Lian: Investigation, Visualization. Yang Xiao: Resources, Data curation. Zhengkun Yi: Validation. Xinyu Wu: Project administration, Funding acquisition.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This research was funded by International Science & Technology Cooperation Program of China (2018YFE0125600), National Natural Science Foundation of China (Grant No. 62103401, Grants No. 62125307, Grant No. U2013209), China Postdoctoral Science Foundation (No. 2021M693314) and Guangdong Basic and Applied Basic Research Foundation (2020A1515111151).

References (45)

  • U. Ghafoor et al.

    Selectivity and longevity of peripheralnerve and machine interfaces: a review

    Frontiers in Neurorobotics

    (2017)
  • I. Lazarou et al.

    EEG-based brain–computer interfaces for communication and rehabilitation of people with motor impairment: a novel approach of the 21st century

    Frontiers in Human Neuroscience

    (2018)
  • F. Marini et al.

    A comparative evaluation of signal quality between a research-grade and a wireless dry-electrode mobile EEG system

    Journal of Neural Engineering

    (2019)
  • S. Sakhavi et al.

    Learning temporal information for brain-computer interface using convolutional neural networks

    IEEE Transactions on Neural Networks and Learning Systems

    (2018)
  • M. Jochumsen et al.

    Quantification of Movement-Related EEG Correlates Associated with Motor Training: A Study on Movement-Related Cortical Potentials and Sensorimotor Rhythms

    Frontiers in Human Neuroscience

    (2017)
  • D. Liu et al.

    Decoding of self-paced lower-limb movement intention: a case study on the influence factors

    Frontiers in Human Neuroscience

    (2017)
  • S. Crea et al.

    Feasibility and safety of shared EEG/EOG and vision-guided autonomous whole-arm exoskeleton control to perform activities of daily living

    Scientific Reports

    (2018)
  • R. Mane et al.

    Prognostic and monitory EEG-biomarkers for BCI upper-limb stroke rehabilitation

    IEEE Transactions on Neural Systems and Rehabilitation Engineering

    (2019)
  • W. Cao et al.

    Effect of hip assistance modes on metabolic cost of walking with a soft exoskeleton

    IEEE Transactions on Automation Science and Engineering

    (2020)
  • W. Cao et al.

    A Lower Limb Exoskeleton with Rigid and Soft Structure for Loaded Walking Assistance

    IEEE Robotics and Automation Letters

    (2021)
  • Y. Li et al.

    A hybrid BCI system combining P300 and SSVEP and its application to wheelchair control

    IEEE Transactions on Biomedical Engineering

    (2013)
  • D. Suma et al.

    Spatial-temporal aspects of continuous EEG-based neurorobotic control

    Journal of Neural Engineering

    (2020)
  • Cited by (12)

    • Adaptive transfer learning-based multiscale feature fused deep convolutional neural network for EEG MI multiclassification in brain–computer interface

      2022, Engineering Applications of Artificial Intelligence
      Citation Excerpt :

      More recently, there are various CNN based methods (Tang et al., 2020; Nour et al., 2021; Dai et al., 2020; Zhang et al., 2019a; Liu et al., 2021) have been developed which demonstrate excellent performance in EEG MI classification. More recent works indicate the growing popularity of various machine learning (ML) techniques, in particular DL, in EEG-based MI classification (Zheng et al., 2022; Yu et al., 2022; Zhao et al., 2022; Li et al., 2022). However, DL-based algorithms contain large numbers of trainable model parameters which require a significant amount of training data and lead to an increase in computation time to train the classifier (Sakhavi and Guan, 2017; Kwon et al., 2019).

    View all citing articles on Scopus
    1

    These authors contributed equally to this work.

    View full text