Skip to main content

Advertisement

Springer Nature Link
Log in

A novel classification framework using multiple bandwidth method with optimized CNN for brain–computer interfaces with EEG-fNIRS signals

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The most effective way to communicate between the brain and electronic devices in the outside world is the brain–computer interface (BCI) systems. BCI systems use signals of being through neural activity in the brain to fulfill this function. Traditional BCI systems use electroencephalography (E.E.G.) signals due to their characteristics, such as temporal resolution, cost, and noninvasive nature. However, the inherent complex features make the analysis process very difficult. In addition, its sensitivity to internal and external noise affects performance negatively. Near-infrared spectroscopy (NIRS), which describes brain hemodynamics, is a noninvasive method and robust against the problems that E.E.G. suffers. We present an effective study examining the effects of E.E.G. and NIRS signals for BCI and investigating the contribution of their combination to performance. Also, a novel classification framework using multiple bandwidth method with optimized convolution neural network (CNN) is proposed. The proposed method classifies the recorded E.E.G. and NIRS signals according to the imagination of opening and closing the subjects' right and left hands. A CNN architecture including fully connected layer optimization using E.E.G. and NIRS signals is trained in an end-to-end manner. Instead of using a single bandwidth as in the literature, multiple bandwidths are used in the training process. In this way, information loss in band filtering tasks is prevented. Performance indicators obtained from experiments performed using the proposed framework are superior to current state-of-the-art methods in the literature in the most significant performance metrics: accuracy and stability. The proposed approach has a higher classification performance than current state-of-the-art methods, with an accuracy performance of 99.85%. On the other hand, in order to test the performance of the proposed CNN method, a detailed ablation study section on single-band experiments and including analysis of each component is presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

References

  1. Khare SK, Bajaj V (2020) A facile and flexible motor imagery classification using electroencephalogram signals, Computer Methods and Programs in Biomedicine, 197

  2. Kai Keng A, Zhang Yang C, Haihong Z, Cuntai G (2008) Filter Bank Common Spatial Pattern (FBCSP) in Brain-Computer Interface, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), pp. 2390–2397.

  3. Zhang R, Li Y, Yan Y, Zhang H, Wu S, Yu T, Gu Z (2016) Control of a wheelchair in an indoor environment based on a brain-computer interface and automated navigation. IEEE Trans Neural Syst Rehabil Eng 24:128–139

    Article  Google Scholar 

  4. Wang H, Dong X, Chen Z, Shi BE (2015) Hybrid gaze/E.E.G. brain computer interface for robot arm control on a pick and place task, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 1476–1479

  5. Paszkiel S (2020) Using BCI Technology for Controlling a Mobile Vehicle, Analysis and Classification of E.E.G. Signals for Brain–Computer Interfaces, pp. 71–77.

  6. LaFleur K, Cassady K, Doud A, Shades K, Rogin E, He B (2013) Quadcopter control in three-dimensional space using a non-invasive motor imagery-based brain–computer interface. J Neural Eng. https://doi.org/10.1088/1741-2560/10/4/046003

    Article  Google Scholar 

  7. Schirrmeister RT, Springenberg JT, Fiederer LDJ, Glasstetter M, Eggensperger K, Tangermann M, Hutter F, Burgard W, Ball T (2017) Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp 38:5391–5420

    Article  Google Scholar 

  8. Sawangjai P, Hompoonsup S, Leelaarporn P, Kongwudhikunakorn S, Wilaiprasitporn T (2020) Consumer Grade E.E.G. Measuring Sensors as Research Tools: A Review, IEEE Sensors Journal, 20: 3996–4024.

  9. Ergun E, Aydemir O (2020) A hybrid BCI using singular value decomposition values of the fast walsh hadamard transform coefficients. IEEE Trans Cogn Dev Syst. https://doi.org/10.1109/TCDS.2020.3028785

    Article  Google Scholar 

  10. Ghonchi H, Fateh M, Abolghasemi V, Ferdowsi S, Rezvani M (2020) Deep recurrent–convolutional neural network for classification of simultaneous E.E.G. fNIRS signals. TET Signal Process 14:142–153

    Google Scholar 

  11. Kwon OY, Lee M-H, Guan C, Lee S-W (2020) Subject-independent brain-computer interfaces based on deep convolutional neural networks. IEEE Trans Neural Netw Learn Syst 31:3839–3852

    Article  Google Scholar 

  12. Yongwook C, Jaeseung J, Sungho J (2012) Toward brain-actuated humanoid robots: asynchronous direct control using an EEG-based BCI. IEEE Trans Rob 28:1131–1144

    Article  Google Scholar 

  13. Li Y, Li X, Ratcliffe M, Liu L, Qi Y, Liu Q (2011) A real-time EEG-based BCI system for attention recognition in ubiquitous environment, Proceedings of 2011 international workshop on Ubiquitous affective awareness and intelligent interaction - UAAII '11

  14. Alazrai R, Alwanni H, Daoud MI (2019) EEG-based BCI system for decoding finger movements within the same hand. Neurosci Lett 698:113–120

    Article  Google Scholar 

  15. Mondini V, Mangia AL, Cappello A (2016) EEG-based BCI system using adaptive features extraction and classification procedures. Comput Intell Neurosci 2016:1–14

    Article  Google Scholar 

  16. Arvaneh M, Guan C, Ang KK, Quek C (2011) Optimizing the channel selection and classification accuracy in EEG-based BCI. IEEE Trans Biomed Eng 58:1865–1873

    Article  Google Scholar 

  17. Gaur P, Pachori RB, Wang H, Prasad G (2018) A multi-class EEG-based BCI classification using multivariate empirical mode decomposition based filtering and Riemannian geometry. Expert Syst Appl 95:201–211

    Article  Google Scholar 

  18. Paszkiel S (2020) Using neural networks for classification of the changes in the E.E.G. signal based on facial expressions. In: Paszkiel S (ed) Analysis and classification of E.E.G. signals for brain-computer interfaces. Springer International Publishing, pp 41–69

    Chapter  Google Scholar 

  19. Jiahui P, Yuanqing L, Jun W (2016) An EEG-based brain-computer interface for emotion recognition. Int Joint Conf Neural Netw (IJCNN) 2016:2063–2067

    Google Scholar 

  20. Tan C, Sun F, Zhang W (2018) Deep Transfer Learning for EEG-Based Brain Computer Interface, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 916–920.

  21. Fahimi F, Zhang Z, Goh WB, Lee T-S, Ang KK, Guan C (2019) Inter-subject transfer learning with an end-to-end deep convolutional neural network for EEG-based BCI. J Neural Eng . https://doi.org/10.1088/1741-2552/aaf3f6

    Article  Google Scholar 

  22. Borgheai SB, McLinden J, Zisk AH, Hosni SI, Deligani RJ, Abtahi M, Mankodiya K, Shahriari Y (2020) Enhancing communication for people in late-stage A.L.S using an fNIRS-based BCI system. IEEE Trans Neural Syst Rehabil Eng 28:1198–1207

    Article  Google Scholar 

  23. Bauernfeind G, Steyrl D, Brunner C, Muller-Putz GR (2014) Single trial classification of fNIRS-based brain-computer interface mental arithmetic data: A comparison between different classifiers, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2004–2007.

  24. Zhang S, Zheng Y, Wang D, Wang L, Ma J, Zhang J, Xu W, Li D, Zhang D (2017) Application of a common spatial pattern-based algorithm for an fNIRS-based motor imagery brain-computer interface. Neurosci Lett 655:35–40

    Article  Google Scholar 

  25. Erdoĝan SB, Özsarfati E, Dilek B, Kadak KS, Hanoĝlu L, Akın A (2019) Classification of motor imagery and execution signals with population-level feature sets: implications for probe design in fNIRS based BCI. J Neural Eng 16:026029

    Article  Google Scholar 

  26. Noori FM, Naseer N, Qureshi NK, Nazeer H, Khan RA (2017) Optimal feature selection from fNIRS signals using genetic algorithms for BCI. Neurosci Lett 647:61–66

    Article  Google Scholar 

  27. Choi JW , Kim K, Lee J, Behboodi B, Trakoolwilaiwan T (2017) Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain–computer interface: three-class classification of rest, right-, and left-hand motor execution, Neurophotonics, 5

  28. Janani A, Sasikala M, Chhabra H, Shajil N, Venkatasubramanian G (2020) Investigation of deep convolutional neural network for classification of motor imagery fNIRS signals for BCI applications. Biomed Signal Process Control. https://doi.org/10.1016/j.bspc.2020.102133

    Article  Google Scholar 

  29. Fazli S, Mehnert J, Steinbrink J, Curio G, Villringer A, Müller K-R, Blankertz B (2012) Enhanced performance by a hybrid NIRS–EEG brain computer interface. Neuroimage 59:519–529

    Article  Google Scholar 

  30. Liu Y, Ayaz H, Shewokis PA (2011) Multisubject “Learning” for mental workload classification using concurrent E.E.G., fNIRS, and physiological measures. Front Hum Neurosci. https://doi.org/10.3389/fnhum.2017.00389

    Article  Google Scholar 

  31. Firooz S, Setarehdan SK (2019) IQ estimation by means of EEG-fNIRS recordings during a logical-mathematical intelligence test. Comput Biol Med 110:218–226

    Article  Google Scholar 

  32. Kim HJ, Wang IN, Kim YT, Kim H, Kim DJ (2020) Comparative analysis of NIRS-EEG motor imagery data using features from spatial, spectral and temporal domain, 2020 8th International Winter Conference on Brain-Computer Interface (BCI), pp. 1–4

  33. Chiarelli AM, Croce P, Merla A, Zappasodi F (2018) Deep learning for hybrid EEG-fNIRS brain–computer interface: application to motor imagery classification. J Neural Eng 15:036028

    Article  Google Scholar 

  34. Rahman MA, Uddin MS, Ahmad M (2019) Modeling and classification of voluntary and imagery movements for brain–computer interface from fNIR and E.E.G. signals through convolutional neural network. Health Inf Sci Syst. https://doi.org/10.1007/s13755-019-0081-5

    Article  Google Scholar 

  35. Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance BJ (2018) EEGNet a compact convolutional neural network for EEG-based brain–computer interfaces. J Neural Eng 15:056013

    Article  Google Scholar 

  36. Öztürk Ş (2020) Stacked auto-encoder based tagging with deep features for content-based medical image retrieval. Expert Systt Appl 161:113693

    Article  Google Scholar 

  37. Abdel-Basset M, El-Shahat D, El-henawy I, Victor Hugo C, de Albuquerque S, Mirjalili, (2020) A new fusion of grey wolf optimizer algorithm with a two-phase mutation for feature selection. Expert Syst Appl 139:112824

    Article  Google Scholar 

  38. Al-Tashi Q, Abdulkadir SJ, Rais HM, Mirjalili S, Alhussian H, Ragab MG, Alqushaibi A (2020) Binary multi-objective grey wolf optimizer for feature selection in classification. IEEE Access 8:106247–106263

    Article  Google Scholar 

  39. Mirjalili S, Mirjalili SM, Lewis A (2014) Grey wolf optimizer. Adv Eng Softw 69:46–61

    Article  Google Scholar 

  40. Shin J, von Luhmann A, Blankertz B, Kim D-W, Jeong J, Hwang H-J, Muller K-R (2017) Open access dataset for EEG+NIRS single-trial classification. IEEE Trans Neural Syst Rehabil Eng 25:1735–1745

    Article  Google Scholar 

  41. Müller K-R, Curio G, Sturm I, Ramsey L, Maeder C, Haufe S, Sannelli C, Fazli S, Vidaurre C, Tangermann M, Blankertz B (2010) The berlin brain-computer interface: non-medical uses of BCI technology. Front Neurosci 4:198

    Google Scholar 

  42. Khan MJ, Hong K-S (2017) Hybrid EEG–fNIRS-based eight-command decoding for BCI application to quadcopter control. Front Neurorobotics 11:6

    Article  Google Scholar 

  43. Yang J, Yao S, Wang J (2018) Deep fusion feature learning network for MI-EEG classification. IEEE Access 6:79050–79059

    Article  Google Scholar 

  44. Zeng H, Yang C, Dai G, Qin F, Zhang J, Kong W (2018) ) E.E.G. classification of driver mental states by deep learning. Cognit Neurodyn 12:597–606

    Article  Google Scholar 

  45. Zhang D, Yao L, Zhang X, Wang S, Chen W, Boots R, Benatallah B (2018) Cascade and parallel convolutional recurrent neural networks on EEG-based intention recognition for brain computer interface, AAAI

  46. Ho TKK, Gwak J, Park CM, Khare A, Song JI (2019) Deep leaning-based approach for mental workload discrimination from multi-channel fNIRS. Recent Trends Commun Comput Electron. https://doi.org/10.1007/978-981-13-2685-1_41

    Article  Google Scholar 

  47. Stamoulis C, Gruber LJ, Schomer DL, Chang BS (2012) High-frequency neuronal network modulations encoded in scalp E.E.G. precede the onset of focal seizures. Epilepsy Behav 23:471–480

    Article  Google Scholar 

  48. Marinazzo D, Melia U, Vallverdú M, Borrat X, Valencia JF, Jospin M, Jensen EW, Gambus P, Caminal P (2015) Prediction of nociceptive responses during sedation by linear and non-linear measures of E.E.G. signals in high frequencies. Plos One. https://doi.org/10.1371/journal.pone.0123464

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Faculty of Engineering, King Abdulaziz University, Jeddah, 21589, Saudi Arabia

    Majid Nour

  2. Department of Electrical and Electronics Engineering, Technology Faculty, Amasya University, Amasya, 05100, Turkey

    Şaban Öztürk

  3. Department of Electrical and Electronics Engineering, Faculty of Engineering, Bolu Abant Izzet Baysal University, Bolu, Turkey

    Kemal Polat

Authors
  1. Majid Nour
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Şaban Öztürk
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Kemal Polat
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Kemal Polat.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest.

Human and animal rights

The paper does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nour, M., Öztürk, Ş. & Polat, K. A novel classification framework using multiple bandwidth method with optimized CNN for brain–computer interfaces with EEG-fNIRS signals. Neural Comput & Applic 33, 15815–15829 (2021). https://doi.org/10.1007/s00521-021-06202-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06202-4

Keywords