Ensemble learning method based on temporal, spatial features with multi-scale filter banks for motor imagery EEG classification
Introduction
Brain-computer interface (BCI) is a promising way of communication, which can establish communication channels between brain and external environment without resorting to peripheral nerve and muscle pathways [1]. BCI system is commonly used in the medical field and can be used to help patients suffering from severe neuromuscular injuries, such as spinal cord injury, amyotrophic lateral sclerosis, brain stroke and cerebral palsy [2]. Among the non-invasive methods, electroencephalogram (EEG) has the advantages of safety, simplicity and low cost, and is considered to be one of the most practical methods in BCI [3]. When the patient imagines moving any part of the body, it will trigger the neurons in a specific area of the brain to produce regular electrical activity, so that the amplitude of the neuroelectric signal on the sensorimotor cortex on the opposite side will decrease, and the amplitude of the neuroelectric signal on the sensorimotor cortex on the ipsilateral side will increase [4], [5]. This movement-related rhythmic activity change is called event-related desynchronization/synchronization (ERD/ERS) phenomenon [6]. The ERD/ERS phenomenon generated by motor imagery (MI) is very similar to actual motion, so it can be converted into computer instructions for controlling external devices by decoding the motion intention information in the EEG [7], [8]. Some excellent MI-based BCI applications have been developed to assist patients in recovering some daily motor functions and improving the quality of life, such as prosthesis [9], exoskeletons [10], [11], [45], and wheelchairs [12].
However, the raw recorded EEG signal usually contains a lot of noise, resulting in a low signal-to-noise ratio and non-stationarity of the signal [13]. In addition, the spatial location, start time, and stability of the ERD/ERS phenomenon in the brain are very different among different subjects [4], [14], which makes it a great challenge to accurately decode the motor intention contained in the EEG.
To accurately decode the motion intention in the MI EEG, many effective decoding algorithms have been proposed [14], [16], [15], [17], [18], [19]. Common spatial pattern (CSP) algorithm is one of the most commonly used algorithms in MI EEG [14]. CSP algorithm extracts features with maximum difference from binary classification signals through a set of optimal spatial filters. However, the classification performance of the CSP algorithm largely depends on the choice of frequency range [15]. CSP algorithm can achieve good classification results in the frequency range of 8–30 Hz, but existing studies have proved that refining the frequency band of CSP for different subjects can improve the classification performance of MI EEG [16]. Novi et al. [17]. proposed a sub-band CSP (SCSP), which extracts reliable CSP features from multiple small frequency bands, and then uses a score fusion strategy for EEG classification. Ang et al. [18]. proposed the filter bank common spatial pattern (FBCSP), which divides a wide frequency band into multiple small frequency bands, and then autonomously selects the discriminative CSP features from the small frequency bands. However, the above method does not further explore the time information of EEG. Zhang et al. [19]. proposed a temporally constrained sparse group spatial pattern (TSGSP), which can optimize the filter frequency band and time window of the CSP algorithm at the same time, and further improve the classification accuracy of MI EEG. Peterson et al. [20]. proposed a penalized time–frequency band CSP (PTFBCSP), which extracts CSP features in each time–frequency sub-band, and then selects the best subject-specific time–frequency band.
Existing studies show that different feature extraction methods have their own advantages, and the combination of multiple types of features can usually achieve higher classification accuracy than using a single type of feature [21], [44]. Lee et al. [22]. recorded MI EEG data from 10 subjects to analyze the performance of spatial, spectral and temporal features on MI-BCI, and showed that temporal and spatial features were superior to spectral features. Al-Qazzaz et al. [23]. proposed to extract features from temporal domain, entropy and frequency domain to characterize the EEG of the MI-BCI, and verified that the fusion of multiple features has higher classification performance than the single feature method.
Most of the above research work is implemented by using a single classifier. Some studies show that by using the idea of ensemble learning multi-classifier decision, the result of training multi-classifier ensemble can obtain better classification performance and robustness than that of single classifier [24], [25]. Zhang et al. [26]. input the extracted time-domain, frequency-domain, time–frequency domain, and spatial domain features into the ensemble LDA classifier, verifying that multiple domain features can complement each other and improve decoding performance. Deng et al. [27]. proposed a random subspace ensemble learning method based on multi-domain features, which used the extracted multi-domain features respectively to train ensemble classifiers, and verified that the classification accuracy of multi-domain ensemble method was higher than that of single-domain ensemble method. The key of ensemble learning to obtain excellent classification performance is to train diverse classifiers [28]. Zuo et al. [29]. proposed a cluster decomposing based ensemble learning framework (CDECL), which obtained different sub-data sets through cluster decomposition, and then trained the ensemble learning heterogeneous classifier model on each sub-data set. In the existing research work, the ensemble classification model is only constructed by multi-domain feature methods, or constructed by subdividing the frequency band of EEG signals. However, the advantages of temporal-spatial information and multi-domain feature information are not complementary to obtain the ensemble learning model with better generalization performance.
In this paper, we first evaluate the algorithms in the preprocessing stage and feature extraction stage of MI EEG analysis, and the results show that the algorithm combinations in different stages have great differences in the classification performance of the subjects. Then, we propose an ensemble learning method based on temporal, spatial features and multi-scale filter banks, called TSMFBEL, to improve the classification performance of MI EEG. For the raw EEG signal, we use the bootstrap sampling method to sample data subsets with differences in sample distribution, and design four time–frequency decomposition methods to preprocess each data subset to obtain time–frequency subsets with time–frequency differences. Then extract discriminative features from multiple domains to train a diverse ensemble classifier. Finally, a classifier weight optimization method based on the minimum error rate of the L2-norm is proposed, and the decision of the ensemble classifier is merged by weighted summation. We evaluated the classification performance of the proposed TSMFBEL algorithm on two public data sets, and compared the results with the state-of-the-art papers. The results show that our proposed TSMFBEL method has better classification performance. The main innovations and contributions of our work are summarized as follows:
- 1.
Multi-scale temporal-spatial preprocessing method is applied to time–frequency decomposition of EEG, which increases the temporal-spatial diversity of EEG subset. Feature extraction methods in spatial, temporal and temporal-spatial domains are used to extract discriminative features and improve the domain diversity of features.
- 2.
The ensemble learning framework proposed by us combines the time–frequency preprocessing method and feature extraction method, and the trained ensemble classifier model has the diversity of sample distribution, time–frequency diversity, and domain diversity.
- 3.
An ensemble classifier weight optimization algorithm based on L2-norm is designed, which can effectively alleviate the over-fitting problem in the weight learning process and improve the robustness of multi-classifier decision fusion.
- 4.
The proposed TSMFBEL algorithm achieved 88.80% and 86.53% average classification accuracy on BCI Competition IV Dataset IIa and Dataset IIb respectively, which is the highest among the state-of-the-art methods.
The organization of this paper is as follows: Section 2 introduces two public data sets and the proposed TSMFBEL algorithm in detail. Section 3 describes the experimental setup and presents the experimental results. Section 4 discusses the results of this study. Section 5 draws the conclusion and future work.
Section snippets
Dataset Description
In this study, we selected two public datasets to evaluate the performance of our proposed method.
BCI Competition IV Dataset IIa: This dataset records 22 channels of MI EEG signals of 9 subjects at a sampling rate of 250 Hz (subject ID: ”A01”, ”A02”, , ”A09”). During the experiment, each subject was required to perform four types of MI tasks based on visual cues: left-hand, right-hand, foot and tongue. The experiment is divided into two sessions, and each session contains 288 experiments. That
Experimental Setup
In the preprocessing step, for dataset IIa, the single-band method performs band-pass filtering at a frequency of 8–35 Hz, and the multi-band method divides the frequency range of 4–40 Hz into a series of sub-bands with a size of 4 Hz and an overlapping size of 2 Hz, (i.e., 4–8 Hz, 6–10 Hz, , 36–40 Hz). The single-time window method selects the EEG data of 0-3s during the execution of the MI task, and the multi-time window method divides the 0-3s time into a series of time windows with a
Discussion
In this study, we analyzed the algorithms of each step in the EEG signal processing process, and found that the subjects have great differences in the selection of time–frequency bands and feature extraction in spatial–temporal domain. Subject B02 can obtain more discriminative feature information in the spatial–temporal features of a single frequency band and single time window, while subjects B06 and B08 have more discriminative feature information in the spatial features of multiple
Conclusion
In this study, we propose an ensemble learning method based on temporal, spatial features and multi-scale filter banks, called TSMFBEL. To construct a diverse ensemble learning classifier, we constructed an EEG time–frequency subset with sample diversity and time–frequency distribution diversity from a limited data set, and then discriminative features are extracted from multiple domains to train diverse ensemble classifiers. Finally, the final result is obtained by weighted fusion of the
CRediT authorship contribution statement
Liangsheng Zheng: Writing - original draft, Conceptualization, Methodology, Software, Formal analysis. Wei Feng: Writing - review & editing, Supervision. Yue Ma: Writing - review & editing, Methodology, Validation. Pengchen Lian: Investigation, Visualization. Yang Xiao: Resources, Data curation. Zhengkun Yi: Validation. Xinyu Wu: Project administration, Funding acquisition.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This research was funded by International Science & Technology Cooperation Program of China (2018YFE0125600), National Natural Science Foundation of China (Grant No. 62103401, Grants No. 62125307, Grant No. U2013209), China Postdoctoral Science Foundation (No. 2021M693314) and Guangdong Basic and Applied Basic Research Foundation (2020A1515111151).
References (45)
- et al.
Event-related EEG/MEG synchronization and desynchronization: basic principles
Clinical Neurophysiology
(1999) - et al.
Multi-feature fusion method based on WOSF and MSE for four-class MI EEG identification
Biomedical Signal Processing and Control
(2021) - et al.
Comparative analysis of features extracted from EEG spatial, spectral and temporal domains for binary and multiclass motor imagery classification
Information Sciences
(2019) EEG feature fusion for motor imagery: A new robust framework towards stroke patients rehabilitation
Computers in Biology and Medicine
(2021)- et al.
Boosting-LDA algriothm with multi-domain feature fusion for motor imagery EEG decoding
Biomedical Signal Processing and Control
(2021) - et al.
Subject-specific time-frequency selection for multi-class motor imagery-based BCIs using few Laplacian EEG channels
Biomedical Signal Processing and Control
(2017) - et al.
Gene selection algorithm by combining reliefF and mRMR
BMC Genomics
(2008) - et al.
A self-adaptive frequency selection common spatial pattern and least squares twin support vector machine for motor imagery electroencephalography recognition
Biomedical Signal Processing and Control
(2018) - et al.
A SVM based classification method for homogeneous data
Applied Soft Computing
(2015) - et al.
Optimization of preprocessing stage in EEG based BCI systems in terms of accuracy and timing cost
Biomedical Signal Processing and Control
(2021)
Selectivity and longevity of peripheralnerve and machine interfaces: a review
Frontiers in Neurorobotics
EEG-based brain–computer interfaces for communication and rehabilitation of people with motor impairment: a novel approach of the 21st century
Frontiers in Human Neuroscience
A comparative evaluation of signal quality between a research-grade and a wireless dry-electrode mobile EEG system
Journal of Neural Engineering
Learning temporal information for brain-computer interface using convolutional neural networks
IEEE Transactions on Neural Networks and Learning Systems
Quantification of Movement-Related EEG Correlates Associated with Motor Training: A Study on Movement-Related Cortical Potentials and Sensorimotor Rhythms
Frontiers in Human Neuroscience
Decoding of self-paced lower-limb movement intention: a case study on the influence factors
Frontiers in Human Neuroscience
Feasibility and safety of shared EEG/EOG and vision-guided autonomous whole-arm exoskeleton control to perform activities of daily living
Scientific Reports
Prognostic and monitory EEG-biomarkers for BCI upper-limb stroke rehabilitation
IEEE Transactions on Neural Systems and Rehabilitation Engineering
Effect of hip assistance modes on metabolic cost of walking with a soft exoskeleton
IEEE Transactions on Automation Science and Engineering
A Lower Limb Exoskeleton with Rigid and Soft Structure for Loaded Walking Assistance
IEEE Robotics and Automation Letters
A hybrid BCI system combining P300 and SSVEP and its application to wheelchair control
IEEE Transactions on Biomedical Engineering
Spatial-temporal aspects of continuous EEG-based neurorobotic control
Journal of Neural Engineering
Cited by (12)
Temporal–spatial transformer based motor imagery classification for BCI using independent component analysis
2024, Biomedical Signal Processing and ControlAutomated identification of the preclinical stage of coal workers' pneumoconiosis from digital chest radiography using three-stage cascaded deep learning model
2023, Biomedical Signal Processing and ControlAdaptive transfer learning-based multiscale feature fused deep convolutional neural network for EEG MI multiclassification in brain–computer interface
2022, Engineering Applications of Artificial IntelligenceCitation Excerpt :More recently, there are various CNN based methods (Tang et al., 2020; Nour et al., 2021; Dai et al., 2020; Zhang et al., 2019a; Liu et al., 2021) have been developed which demonstrate excellent performance in EEG MI classification. More recent works indicate the growing popularity of various machine learning (ML) techniques, in particular DL, in EEG-based MI classification (Zheng et al., 2022; Yu et al., 2022; Zhao et al., 2022; Li et al., 2022). However, DL-based algorithms contain large numbers of trainable model parameters which require a significant amount of training data and lead to an increase in computation time to train the classifier (Sakhavi and Guan, 2017; Kwon et al., 2019).
Lightweight Privacy-Preserving Feature Extraction for EEG Signals under Edge Computing
2024, IEEE Internet of Things Journal
- 1
These authors contributed equally to this work.