Skip to main content
SpringerLink
Log in

Automatic feature extraction and fusion recognition of motor imagery EEG using multilevel multiscale CNN

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

A motor imagery EEG (MI-EEG) signal is often selected as the driving signal in an active brain computer interface (BCI) system, and it has been a popular field to recognize MI-EEG images via convolutional neural network (CNN), which poses a potential problem for maintaining the integrity of the time-frequency-space information in MI-EEG images and exploring the feature fusion mechanism in the CNN. However, information is excessively compressed in the present MI-EEG image, and the sequential CNN is unfavorable for the comprehensive utilization of local features. In this paper, a multidimensional MI-EEG imaging method is proposed, which is based on time-frequency analysis and the Clough-Tocher (CT) interpolation algorithm. The time-frequency matrix of each electrode is generated via continuous wavelet transform (WT), and the relevant section of frequency is extracted and divided into nine submatrices, the longitudinal sums and lengths of which are calculated along the directions of frequency and time successively to produce a 3 × 3 feature matrix for each electrode. Then, feature matrix of each electrode is interpolated to coincide with their corresponding coordinates, thereby yielding a WT-based multidimensional image, called WTMI. Meanwhile, a multilevel and multiscale feature fusion convolutional neural network (MLMSFFCNN) is designed for WTMI, which has dense information, low signal-to-noise ratio, and strong spatial distribution. Extensive experiments are conducted on the BCI Competition IV 2a and 2b datasets, and accuracies of 92.95% and 97.03% are yielded based on 10-fold cross-validation, respectively, which exceed those of the state-of-the-art imaging methods. The kappa values and p values demonstrate that our method has lower class skew and error costs. The experimental results demonstrate that WTMI can fully represent the time-frequency-space features of MI-EEG and that MLMSFFCNN is beneficial for improving the collection of multiscale features and the fusion recognition of general and abstract features for WTMI.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Qiao W, Bi X (2020) Ternary-task convolutional bidirectional neural turing machine for assessment of EEG-based cognitive workload[J]. Biomedl Signal Process Control 57:101745

    Article  Google Scholar 

  2. Usman SM, Khalid S, Akhtar R, Bortolotto Z, Bashir Z, Qiu H (2019) Using scalp EEG and intracranial EEG signals for predicting epileptic seizures: review of available methodologies[J]. Seizure 71:258–269

    Article  Google Scholar 

  3. Sandheep P, Vineeth S, Poulose M, Subha DP (2019) “Performance analysis of deep learning CNN in classification of depression EEG signals,” TENCON 2019 - 2019 IEEE Region 10 Conference (TENCON), Kochi, India, 2019: 1339–1344.

  4. Lotte F, Bougrain L, Cichocki A, Clerc M, Congedo M, Rakotomamonjy A, Yger F (2018) A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update[J]. J Neural Eng 15(3):031005

    Article  CAS  Google Scholar 

  5. Pfurtscheller G, Neuper C (2001) Motor imagery and direct brain-computer communication[J]. Proc IEEE 89(7):1123–1134

    Article  Google Scholar 

  6. Royer AS, Doud AJ, Rose ML, He B (2010) EEG control of a virtual helicopter in 3-dimensional space using intelligent control strategies[J]. IEEE Trans Neural Syst Rehabil Eng 18(6):581–589

    Article  Google Scholar 

  7. Cahn BR, Polich (2016) Meditation states and traits: EEG, ERP, and neuroimaging studies[J]. Psychol Bull 132(2):180–211

    Article  Google Scholar 

  8. Gouy-Pailler C, Congedo M, Brunner C, Jutten C, Pfurtscheller G (2010) Nonstationary brain source separation for multiclass motor imagery[J]. IEEE Trans Biomed Eng 57(2):469–478

    Article  Google Scholar 

  9. Brunner C, Naeem M, Leeb R, Graimann B, Pfurtscheller G (2007) Spatial filtering and selection of optimized components in four class motor imagery EEG data using independent components analysis[J]. Pattern Recogn Lett 28(8):957–964

    Article  Google Scholar 

  10. Vidaurre C, Kawanabe M, von Bünau P, Blankertz B, Müller KR (2011) Toward unsupervised adaptation of LDA for brain–computer interfaces[J]. IEEE Trans Biomed Eng 58(3):587–597

    Article  CAS  Google Scholar 

  11. Dong E, Li C, Li L, du S, Belkacem AN, Chen C (2017) Classification of multi-class motor imagery with a novel hierarchical SVM algorithm for brain–computer interfaces[J]. Med Biol Eng Comput 55(10):1809–1818

    Article  Google Scholar 

  12. Sitaram R, Zhang H, Guan C, Thulasidas M, Hoshi Y, Ishikawa A, Shimizu K, Birbaumer N (2007) Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain–computer interface[J]. NeuroImage 34(4):1416–1427

    Article  Google Scholar 

  13. Hu J, Xiao D, Mu Z (2009) Application of energy entropy in motor imagery EEG classification[J]. International Journal of Digital Content Technology and its Applications 3(2):83–90

    Google Scholar 

  14. Alom MZ, Taha TM, Yakopcic C et al (2018) The history began from alexnet: a comprehensive survey on deep learning approaches[J]. arXiv preprint arXiv 1803:01164

    Google Scholar 

  15. Dose H, Møller JS, Iversen HK, Puthusserypady S (2018) An end-to-end deep learning approach to MI-EEG signal classification for BCIs[J]. Expert Syst Appl 114:532–542

    Article  Google Scholar 

  16. Tang Xuebin, Zhao Jinchuang, Fu Wenlin et al (2019) A novel classification algorithm for MI-EEG based on deep learning[C]. 2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC):606–611

  17. Wei CS, Koike-Akino T, Wang Y (2019) Spatial component-wise convolutional network (SCCNet) for motor-imagery EEG classification[C]. 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER) 2019:328–331

  18. Yang H, Sakhavi S, Ang KK et al (2015) On the use of convolutional neural networks and augmented CSP features for multi-class motor imagery of EEG signals classification[C]. 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 2015:2620–2623

    Google Scholar 

  19. Korhan N, Dokur Z, Olmez T (2019) Motor imagery based EEG classification by using common spatial patterns and convolutional neural networks[C]. 2019 Scientific Meeting on Electrical-Electronics & Biomedical Engineering and Computer Science (EBBT) 2019:1–4

    Google Scholar 

  20. Sakhavi S, Guan C, Yan S (2015) Parallel convolutional-linear neural network for motor imagery classification[C]. 2015 23rd European Signal Processing Conference (EUSIPCO) 2015:2736–2740

    Article  Google Scholar 

  21. Sakhavi S, Guan C, Yan S (2018) Learning temporal information for brain-computer interface using convolutional neural networks[J]. IEEE Trans Neural Netw Learn Syst 29(11):5619–5629

    Article  Google Scholar 

  22. Zhu X, Li P, Li C et al (2018) Separated channel convolutional neural network to realize the training free motor imagery BCI systems[J]. Biomedl Signal Process Control 49:396–403

    Article  Google Scholar 

  23. Uktveris T, Jusas V (2017) Application of convolutional neural networks to four-class motor imagery classification problem[J]. Journal of Information Technology and Control 46(2):260–273 May

    Google Scholar 

  24. Tabar YR, Halici U (2017) A novel deep learning approach for classification of EEG motor imagery signals[J]. J Neural Eng 14(1):016003

    Article  Google Scholar 

  25. Dai M, Zheng D, Na R, Wang S, Zhang S (2019) EEG classification of motor imagery using a novel deep learning framework[J]. Sensors 19(3):551

    Article  Google Scholar 

  26. Tian G, Liu Y (2018) Study on classification of left-right hands motor imagery EEG signals based on CNN[C]. 2018 IEEE 17th International Conference on Cognitive Informatics & Cognitive Computing (ICCI* CC) 2018:324–329

    Article  Google Scholar 

  27. Wang Zijian, Cao Lei, Zhang Zuo, Gong Xiaoliang, Sun Yaoru, Wang Haoran (2018) Short time Fourier transformation and deep neural networks for motor imagery brain computer interface recognition[J]. Concurrency and Computation: Practice and Experience 30(23):e4413

  28. Tayeb Z, Fedjaev J, Ghaboosi N et al (2019) Validating deep neural networks for online decoding of motor imagery movements from EEG signals[J]. Sensors 19(1):210

    Article  Google Scholar 

  29. Alazrai R, Abuhijleh M, Alwanni H, Daoud MI (2019) A deep learning framework for decoding motor imagery tasks of the same hand using EEG signals[J]. IEEE Access 7:109612–109627

    Article  Google Scholar 

  30. Xu B, Zhang L, Song A et al (2018) Wavelet transform time-frequency image and convolutional network-based motor imagery EEG classification[J]. IEEE Access 7:6084–6093

    Article  Google Scholar 

  31. Yu Z, Zhang X et al (2017) Portable brain-computer interface based on novel convolutional neural network[J]. Comput Biol Med 107:248–256

    Google Scholar 

  32. Alfeld P (1984) A trivariate clough—tocher scheme for tetrahedral data[J]. Comput Aided Geom Des 1(2):169–181

    Article  Google Scholar 

  33. Chaudhary S, Taran S, Bajaj V (2019) Convolutional neural network based approach towards motor imagery tasks EEG signals classification[J]. IEEE Sensors J 19(12):4494–4500

    Article  Google Scholar 

  34. Simonyan Karen, Zisserman Andrew(2014) “Very deep convolutional networks for large-scale image recognition,” CoRR, vol. abs/ 1409.1556

  35. Deng Jia, Dong Wei, Socher Richard, Li Li-Jia, Li Kai, Fei-Fei Li (2009) “ImageNet: a large-scale hierarchical image database”, Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops

  36. Xu G, Shen X, Chen S, Zong Y, Zhang C, Yue H, Liu M, Chen F, Che W (2019) A deep transfer convolutional neural network framework for EEG signal classification[J]. IEEE Access 7:112767–112776

    Article  Google Scholar 

  37. Ling SH, Makgawinata H, Monsivais FH et al (2019) Classification of EEG motor imagery tasks using convolution neural networks[C]. 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 2019:758–761

    Article  Google Scholar 

  38. Zhao Wei, Ye Jianbo, Yang Min, et al. (2018) Investigating capsule networks with dynamic routing for text classification[J]. arXiv preprint arXiv:1804.00538

  39. Ha KW, Jeong JW (2019) Motor imagery EEG classification using capsule networks[J]. Sensors 19(13):2854

    Article  Google Scholar 

  40. Li M-A, Han J-F, Duan L-J (2019) A novel MI-EEG imaging with the location information of electrodes[J]. IEEE Access 8:3197–3211

    Article  Google Scholar 

  41. Amin SU, Alsulaiman M, Muhammad G, Bencherif MA, Hossain MS (2019) Multilevel weighted feature fusion using convolutional neural networks for EEG motor imagery classification[J]. IEEE Access 7:18940–18950

    Article  Google Scholar 

  42. Amin SU, Alsulaiman M, Muhammad G, Mekhtiche MA, Shamim Hossain M (2019) Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion[J]. Futur Gener Comput Syst 101:542–554

    Article  Google Scholar 

  43. Zhao X, Zhang H, Zhu G, You F, Kuang S, Sun L (2019) A multi-branch 3D convolutional neural network for EEG-based motor imagery classification[J]. IEEE Trans Neural Syst Rehabil Eng 27(10):2164–2177

    Article  Google Scholar 

  44. Leeb R, Lee F, Keinrath C, Scherer R, Bischof H, Pfurtscheller G (2007) Brain–computer communication: motivation, aim, and impact of exploring a virtual apartment. IEEE Trans Neural Syst Rehabil Eng 15(4):473–482

    Article  Google Scholar 

  45. Li CH, Lee CK (1993) Minimum cross entropy thresholding[J]. Pattern Recogn 26(4):617–625

    Article  Google Scholar 

  46. Lee GR, Gommers R, Wasilewski F, Wohlfahrt K, O’Leary A (2019) PyWavelets: a Python package for wavelet analysis. J Open Source Softw 4(36):1237

    Article  Google Scholar 

Download references

Acknowledgement

We would like to thank the provider of dataset and all of the people who have given us helpful suggestions and advice. The authors are obliged to the anonymous referee for carefully looking over the details and for useful comments that improved this paper.

Funding

The National Natural Science Foundation of China (No. 11832003, No. 61672070) and the Natural Science Foundation of Beijing (No. 7132021, No. 4182009) financially supported this work.

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Beijing University of Technology, Beijing, 100124, China

    Ming-ai Li, Jian-fu Han & Jin-fu Yang

  2. Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing, 100124, China

    Ming-ai Li & Jin-fu Yang

  3. Engineering Research Center of Digital Community, Ministry of Education, Beijing, 100124, China

    Ming-ai Li

Authors
  1. Ming-ai Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jian-fu Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jin-fu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ming-ai Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Ma., Han, Jf. & Yang, Jf. Automatic feature extraction and fusion recognition of motor imagery EEG using multilevel multiscale CNN . Med Biol Eng Comput 59, 2037–2050 (2021). https://doi.org/10.1007/s11517-021-02396-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-021-02396-w

Keywords

Search

Navigation