Abstract
In brain–computer interfaces, imagined speech is one of the most promising paradigms due to its intuitiveness and direct communication. However, it is challenging to decode an imagined speech EEG, because of its complicated underlying cognitive processes, resulting in complex spectro-spatio-temporal patterns. In this work, we propose a novel convolutional neural network structure for representing such complex patterns and identifying an intended imagined speech. The proposed network exploits two feature extraction flows for learning richer class-discriminative information. Specifically, our proposed network is composed of a spatial filtering path and a temporal structure learning path running in parallel, then integrates their output features for decision-making. We demonstrated the validity of our proposed method on a publicly available dataset by achieving state-of-the-art performance. Furthermore, we analyzed our network to show that our method learns neurophysiologically plausible patterns.
This work was supported by Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea government under Grant 2017-0-00451 (Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning) and Grant 2019-0-00079 (Department of Artificial Intelligence, Korea University).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Available at: https://osf.io/pq7vb/.
References
Ang, K.K., Chin, Z.Y., Zhang, H., Guan, C.: Filter bank common spatial pattern (FBCSP) in brain-computer interface. In: 2008 IEEE International Joint Conference on Neural Networks, pp. 2390–2397. IEEE (2008)
Bakhshali, M.A., Khademi, M., Ebrahimi-Moghadam, A., Moghimi, S.: EEG signal classification of imagined speech based on Riemannian distance of correntropy spectral density. Biomed. Signal Process. Control 59, 101899 (2020)
Brigham, K., Kumar, B.V.: Imagined speech classification with EEG signals for silent communication: a preliminary investigation into synthetic telepathy. In: 2010 4th International Conference on Bioinformatics and Biomedical Engineering, pp. 1–4. IEEE (2010)
Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)
Cooney, C., Korik, A., Folli, R., Coyle, D.: Evaluation of hyperparameter optimization in machine and deep learning methods for decoding imagined speech EEG. Sensors 20(16), 4629 (2020)
DaSalla, C.S., Kambara, H., Sato, M., Koike, Y.: Single-trial classification of vowel speech imagery using common spatial patterns. Neural Netw. 22(9), 1334–1339 (2009)
Deng, S., Srinivasan, R., Lappas, T., D’Zmura, M.: EEG classification of imagined syllable rhythm using Hilbert spectrum methods. J. Neural Eng. 7(4), 046006 (2010)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256. JMLR Workshop and Conference Proceedings (2010)
Haufe, S., et al.: On the interpretation of weight vectors of linear models in multivariate neuroimaging. Neuroimage 87, 96–110 (2014)
Ko, W., Jeon, E., Jeong, S., Phyo, J., Suk, H.I.: A survey on deep learning-based short/zero-calibration approaches for EEG-based brain-computer interfaces. Front. Hum. Neurosci. 15, 643386 (2021)
Ko, W., Jeon, E., Jeong, S., Suk, H.I.: Multi-scale neural network for EEG representation learning in BCI. IEEE Comput. Intell. Mag. 16(2), 31–45 (2021)
Ko, W., Oh, K., Jeon, E., Suk, H.I.: VigNet: a deep convolutional neural network for EEG-based driver vigilance estimation. In: 2020 8th International Winter Conference on Brain-Computer Interface, BCI, pp. 1–3. IEEE (2020)
Ko, W., Yoon, J., Kang, E., Jun, E., Choi, J.S., Suk, H.I.: Deep recurrent spatio-temporal neural network for motor imagery based BCI. In: 2018 6th International Conference on Brain-Computer Interface, BCI, pp. 1–3. IEEE (2018)
Lawhern, V.J., Solon, A.J., Waytowich, N.R., Gordon, S.M., Hung, C.P., Lance, B.J.: EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J. Neural Eng. 15(5), 056013 (2018)
Lee, S.H., Lee, M., Lee, S.W.: Neural decoding of imagined speech and visual imagery as intuitive paradigms for BCI communication. IEEE Trans. Neural Syst. Rehabil. Eng. 28(12), 2647–2659 (2020)
Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint http://arxiv.org/abs/1312.4400 (2013)
Liu, D.C., Nocedal, J.: On the limited memory BFGS method for large scale optimization. Math. Program. 45(1), 503–528 (1989)
Lotte, F., Roy, R.N.: Brain-Computer Interface Contributions to Neuroergonomics. In: Neuroergonomics, pp. 43–48. Elsevier (2019)
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)
Matsumoto, M., Hori, J.: Classification of silent speech using support vector machine and relevance vector machine. Appl. Soft Comput. 20, 95–102 (2014)
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., Müller, K.R.: Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recogn. 65, 211–222 (2017)
Nguyen, C.H., Karavas, G.K., Artemiadis, P.: Inferring imagined speech using EEG signals: a new approach using Riemannian manifold features. J. Neural Eng. 15(1), 016002 (2017)
Sakhavi, S., Guan, C., Yan, S.: Parallel convolutional-linear neural network for motor imagery classification. In: 2015 23rd European Signal Processing Conference, EUSIPCO, pp. 2736–2740. IEEE (2015)
Schirrmeister, R.T., et al.: Deep learning with convolutional neural networks for EEG decoding and visualization. Hum. Brain Mapp. 38(11), 5391–5420 (2017)
Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems, vol. 25 (2012)
Suk, H.I., Lee, S.W.: A novel Bayesian framework for discriminative feature extraction in brain-computer interfaces. IEEE Trans. Pattern Anal. Mach. Intell. 35(2), 286–299 (2012)
Wang, Y., Jung, T.P., et al.: Visual stimulus design for high-rate SSVEP BCI. Electron. Lett. 46(15), 1057–1058 (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Ko, W., Jeon, E., Suk, HI. (2022). Spectro-Spatio-Temporal EEG Representation Learning for Imagined Speech Recognition. In: Wallraven, C., Liu, Q., Nagahara, H. (eds) Pattern Recognition. ACPR 2021. Lecture Notes in Computer Science, vol 13189. Springer, Cham. https://doi.org/10.1007/978-3-031-02444-3_25
Download citation
DOI: https://doi.org/10.1007/978-3-031-02444-3_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-02443-6
Online ISBN: 978-3-031-02444-3
eBook Packages: Computer ScienceComputer Science (R0)