Skip to main content

Advertisement

SpringerLink
Log in

Bi-hemisphere asymmetric attention network: recognizing emotion from EEG signals based on the transformer

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

EEG-based emotion recognition is not only an important branch in the field of affective computing, but is also an indispensable task for harmonious human–computer interaction. Recently, many deep learning emotion recognition algorithms have achieved good results, but most of them have been based on convolutional and recurrent neural networks, resulting in complex model design, poor modeling of long-distance dependency, and the inability to parallelize computations. Here, we proposed a novel bi-hemispheric asymmetric attention network (Bi-AAN) combining a transformer structure with the asymmetric property of the brain’s emotional response. In this way, we modeled the difference of bi-hemispheric attention, and mined the long-term dependency between EEG sequences, which exacts more discriminative emotional representations. First, the differential entropy (DE) features of each frequency band were calculated using the DE-embedding block, and the spatial information between the electrode positions was extracted using positional encoding. Then, a bi-headed attention mechanism was employed to capture the intra-attention of frequency bands in each hemisphere and the attentional differences between the bi-hemispheric frequency bands. After carring out experiments both in DEAP and DREAMER datasets, we found that the proposed Bi-AAN achieved superior recognition performance as compared to state-of-the-art EEG emotion recognition methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Koelstra S (2012) DEAP: A Database for Emotion Analysis ;Using Physiological Signals. IEEE Trans Affect Comput 3(1):18–31

    Article  Google Scholar 

  2. Huang Y, Wu C, Wong AM, Lin B (2014) Novel active comb-shaped dry electrode for EEG measurement in hairy site. Biomed Engin IEEE Transact 62(1):256–263

    Article  Google Scholar 

  3. Xiaohua H et al (2017) Discriminative Spatiotemporal Local Binary Pattern with Revisited Integral Projection for Spontaneous Facial Micro-Expression Recognition. Affective Comput IEEE Transac 10(1):32–47

    MathSciNet  Google Scholar 

  4. C. Li, W. Tao, J. Cheng, Y. Liu, and X. Chen, "Robust Multichannel EEG Compressed Sensing in the Presence of Mixed Noise," IEEE Sensors Journal, vol. PP, no. 99, 2019

  5. Alarcao SM, Fonseca MJ (2019) Emotions recognition using EEG signals: a survey. IEEE Trans Affect Comput 10(3):374–393. https://doi.org/10.1109/taffc.2017.2714671

    Article  Google Scholar 

  6. Alhagry S, Aly A, Reda A (2017) Emotion Recognition based on EEG using LSTM Recurrent Neural Network. Int J Adv Comput Sci Appl 8(10)

  7. Subasi A, Tuncer T, Dogan S, Tanko D, Sakoglu U (2021) EEG-based emotion recognition using tunable Q wavelet transform and rotation forest ensemble classifier. Biomed Sig Proces Contrl 68:102648. https://doi.org/10.1016/j.bspc.2021.102648

    Article  Google Scholar 

  8. Tuncer T, Dogan S, Subasi A (2021) A new fractal pattern feature generation function based emotion recognition method using EEG. Chaos, Solitons Fractals 144:110671. https://doi.org/10.1016/j.chaos.2021.110671

    Article  MathSciNet  Google Scholar 

  9. Tuncer T, Dogan S, Baygin M, Rajendra Acharya U (2022) Tetromino pattern based accurate EEG emotion classification model. Artif Intell Med 123:102210. https://doi.org/10.1016/j.artmed.2021.102210

    Article  Google Scholar 

  10. Dogan A et al (2021) PrimePatNet87: prime pattern and tunable q-factor wavelet transform techniques for automated accurate EEG emotion recognition. Comput Biol Med 138:104867. https://doi.org/10.1016/j.compbiomed.2021.104867

    Article  Google Scholar 

  11. M. A. Asghar, M. J. Khan, M. Rizwan, R. M. Mehmood, S-H. Kim, "An Innovative Multi-Model Neural Network Approach for Feature Selection in Emotion Recognition Using Deep Feature Clustering," Sensors, vol. 20, no. 13, 2020, Art no 3765, https://doi.org/10.3390/s20133765

  12. R. Alhalaseh and S. Alasasfeh, "Machine-Learning-Based Emotion Recognition System Using EEG Signals," Computers, vol. 9, no. 4, 2020, Art no. 95, https://doi.org/10.3390/computers9040095

  13. Moon S-E, Chen C-J, Hsieh C-J, Wang J-L, Lee J-S (2020) Emotional EEG classification using connectivity features and convolutional neural networks. Neural Netw 132:96–107. https://doi.org/10.1016/j.neunet.2020.08.009

    Article  Google Scholar 

  14. Pandey P, Seeja KR (2021) Subject independent emotion recognition system for people with facial deformity: an EEG based approach. J Ambient Intell Humaniz Comput 12(2):2311–2320. https://doi.org/10.1007/s12652-020-02338-8

    Article  Google Scholar 

  15. Senguer D, Siuly S (2020) Efficient approach for EEG-based emotion recognition. Electron Lett 56(25). https://doi.org/10.1049/el.2020.2685

  16. Y. Cimtay and E. Ekmekcioglu, "Investigating the Use of Pretrained Convolutional Neural Network on Cross-Subject and Cross-Dataset EEG Emotion Recognition," Sensors, vol. 20, no. 7, 2020, Art no. 2034, https://doi.org/10.3390/s20072034

  17. Demir F, Sobahi N, Siuly S, Sengur A (2021) Exploring deep learning features for automatic classification of human emotion using EEG rhythms. IEEE Sensors J:1–1. https://doi.org/10.1109/JSEN.2021.3070373

  18. Tao W et al (2020) EEG-based emotion recognition via channel-wise attention and self attention. IEEE Trans Affect Comput:1–1. https://doi.org/10.1109/taffc.2020.3025777

  19. Youjun L, Jiajin H, Haiyan Z, Ning Z (2017) Human emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks. Appl Sci 7(10):1060

    Article  Google Scholar 

  20. Yang Y, Wu Q, Qiu M, Wang Y, Chen X (2018) "Emotion recognition from Multi-Channel EEG through parallel convolutional recurrent neural network," presented at the 2018 International Joint Conference on Neural Networks (IJCNN)

  21. Kim Y, Choi A EEG-Based Emotion Classification Using Long Short-Term Memory Network with Attention Mechanism. Sensors (Basel, Switzerland) 20(23):6727

  22. Y. Yin, X. Zheng, B. Hu, Y. Zhang, and X. Cui, "EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM," Appl Soft Comput, vol. 100, 2021, Art no 106954, https://doi.org/10.1016/j.asoc.2020.106954

  23. Li J, Wu X, Zhang Y, Yang H, Wu X (2022) DRS-net: a spatial–temporal affective computing model based on multichannel EEG data. Biomed Sig Proces Contrl 76:103660. https://doi.org/10.1016/j.bspc.2022.103660

    Article  Google Scholar 

  24. Liu S, et. al. (n.d.) "3DCANN: a Spatio-temporal convolution attention neural network for EEG emotion recognition," IEEE Journal of Biomedical and Health Informatics, https://doi.org/10.1109/JBHI.2021.3083525

  25. X. Tan, K. L. Gao, B. Liu, Y. M. Fu, and L. Kang, "Deep global-local transformer network combined with extended morphological profiles for hyperspectral image classification," J Appl Remote Sens, vol. 15, no. 3, 2021, Art no. 038509, https://doi.org/10.1117/1.Jrs.15.038509

  26. Y. Dai, Y. Gao, and F. Liu, "TransMed: Transformers Advance Multi-Modal Medical Image Classification," Diagnostics, vol. 11, no. 8, 2021, Art no. 1384, https://doi.org/10.3390/diagnostics11081384

  27. Zhao B, Gong M, Li X (2022) Hierarchical multimodal transformer to summarize videos. Neurocomputing 468:360–369. https://doi.org/10.1016/j.neucom.2021.10.039

    Article  Google Scholar 

  28. A. Vaswani et al. 2017 "attention is all you need," arXiv

  29. J. Schmidhuber (2001) Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. Gradient flow in recurrent nets: The Difficulty of Learning LongTerm Dependencies

  30. T. Lin, Y. Wang, X. Liu, and X. Qiu (2021) "A Survey of Transformers"

  31. D. Kostas, S. Aroca-Ouellette, and F. Rudzicz (2021) "BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data"

  32. Chen H, Jiang D, Sahli H (2020) Transformer encoder with multi-modal multi-head attention for continuous affect recognition. IEEE Transac Multimed 99:1–1

    Google Scholar 

  33. Craik A, He Y, Contreras-Vidal JL (2019) Deep learning for electroencephalogram (EEG) classification tasks: a review. J Neural Eng 16(3):031001. https://doi.org/10.1088/1741-2552/ab0ab5

    Article  Google Scholar 

  34. Zatorre RJ, Jonesgotman M, Evans AC, Meyer E (1992) Functional localization and lateralization of human olfactory cortex. Nature 360(6402):339–340. https://doi.org/10.1038/360339a0

    Article  Google Scholar 

  35. Dimond SJ, Farrington L, Johnson P (1976) Differing emotional response from right and left hemispheres. Nature 261(5562):690–692. https://doi.org/10.1038/261690a0

    Article  Google Scholar 

  36. Davidson RJ (1992) Emotion and affective style - hemispheric substrates. Psychol Sci 3(1):39–43. https://doi.org/10.1111/j.1467-9280.1992.tb00254.x

    Article  MathSciNet  Google Scholar 

  37. R. J. Davidson, C. D. Saron, J. A. Senulis, P. Ekman, and W. V. Friesen, "Approach withdrawal and cerebral asymmetry - emotional expression and brain physiology I," J Pers Soc Psychol, vol. 58, no. 2, pp. 330–341, Feb 1990, https://doi.org/10.1037/0022-3514.58.2.330

  38. Li Y, Zheng W, Zong Y, Cui Z, Zhang T, Zhou X (2018) A bi-hemisphere domain adversarial neural network model for EEG emotion recognition. IEEE Trans Affect Comput:1–1

  39. Huang D, Chen S, Liu C, Zheng L, Jiang D (2021) Differences first in asymmetric brain: a bi-hemisphere discrepancy convolutional neural network for EEG emotion recognition. Neurocomputing 448

  40. Katsigiannis S, Ramzan N (2017) DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE J Biomed Health Inform:1–1

  41. Yang Y, Wu Q, Fu Y, Chen X (2018, ch. Chapter) Continuous Convolutional Neural Network with 3D Input for EEG-Based Emotion Recognition. Neural Inform Proc, (Lecture Notes in Comp Science) 39:433–443

    Google Scholar 

  42. Gao Z, Wang X, Yang Y, Li Y, Ma K, Chen G (2020) A channel-fused dense convolutional network for EEG-based emotion recognition. IEEE Transac Cogn Develop Syst:1–1. https://doi.org/10.1109/tcds.2020.2976112

  43. Y. Zhu and Q. Zhong, "Differential entropy feature signal extraction based on activation mode and its recognition in convolutional gated recurrent unit network," Front Phys, vol. 8, 2021, Art no. 629620, https://doi.org/10.3389/fphy.2020.629620

  44. Chao H, Dong L (2021) Emotion recognition using three-dimensional feature and convolutional neural network from multichannel EEG signals. IEEE Sensors J 21(2):2024–2034. https://doi.org/10.1109/jsen.2020.3020828

    Article  Google Scholar 

  45. Kalchbrenner N, Espeholt L, Simonyan K, Oord A, Graves A, Kavukcuoglu K (2016) "Neural Machine Translation in Linear Time"

  46. Kaiser U, Sutskever I (2015) "Neural GPUs learn algorithms", Computer Science

  47. Luong MT, Pham H, Manning CD (2015) "Effective approaches to attention-based neural machine translation," Computer ence

  48. Devlin J, Chang MW, Lee K, Toutanova K (2018) "BERT: pre-training of deep bidirectional transformers for language understanding"

  49. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Houlsby N (2020) "An image is worth 16x16 words: Transformers for Image Recognition at Scale"

  50. Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) End-to-End Object Detection with Transformers

  51. Gulati A et al. (2020) "Conformer: Convolution-augmented Transformer for Speech Recognition"

  52. Chen X, Wu Y, Wang Z, Liu S, Li J (2020) "Developing real-time streaming transformer transducer for speech recognition on large-scale dataset"

  53. Rives A, Meier J, Sercu T, Goyal S, Fergus R (2021) Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc Natl Acad Sci 118(15):e2016239118

    Article  Google Scholar 

  54. Schwaller P, Laino T, Gaudin T, Bolgar P, Bekas C, Lee AA (2018) "Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction"

  55. Fox E (2008) Emotion Science: Cognitive and Neuroscientific Approaches to Understanding Human Emotions. Emotion science: cognitive and neuroscientific approaches to understanding human emotions

  56. Song T, Zheng W, Song P, Cui Z (2020) EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput 11(3):532–541. https://doi.org/10.1109/taffc.2018.2817622

    Article  Google Scholar 

  57. Liu S, Wang X, Zhao L, Zhao J, Xin Q, Wang S (2020) "Subject-independent Emotion Recognition of EEG Signals Based on Dynamic Empirical Convolutional Neural Network", IEEE/ACM Trans Comput Biol Bioinform, vol. PP, https://doi.org/10.1109/TCBB.2020.3018137

  58. He K, Zhang X, Ren S, Sun J (2016) "Deep residual learning for image recognition," IEEE

  59. Ba JL, Kiros JR, Hinton GE (2016) "Layer normalization"

    Google Scholar 

  60. Shawky E, El-Khoribi R, Shoman MAI, Wahby MA (2018) EEG-based emotion recognition using 3D convolutional neural networks. Int J Adv Comput Sci Appl 9(8):329

    Google Scholar 

  61. Wang Y, Huang Z, Mccane B, Neo P (2018) EmotioNet: a 3-D convolutional neural network for EEG-based emotion. Recognition:1–7

  62. Xiang L, Song D, Peng Z, Yu G, and Hu B (2017) "Emotion recognition from multi-channel EEG data through convolutional recurrent neural network," in IEEE International Conference on Bioinformatics & Biomedicine

  63. Mert A, Akan A (2016) Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal Applic 21(1):81–89

    Article  MathSciNet  Google Scholar 

  64. Tripathi S, Acharya S, Sharma RD, Mittal S, Bhattacharya S (2017) "Using deep and convolutional neural networks for accurate emotion classification on DEAP dataset”

  65. Xing X, Li Z, Xu T, Shu L, Xu X (2019) "SAE+LSTM: A New Framework for Emotion Recognition From Multi-Channel EEG", Frontiers in Neurorobotics, vol. 13

  66. Zhang J, Ming C, Hu S, Yu C, Kozma R (2016) "PNN for EEG-based emotion recognition," in 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC)

  67. Zhang G, Yu M, Liu YJ, Zhao G, Zhang D, Zheng W (2021) SparseDGCNN: recognizing emotion from multichannel EEG signals. IEEE Trans Affect Comput:1–1. https://doi.org/10.1109/TAFFC.2021.3051332

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (No. 61872301 and No. 61472330).

Author information

Authors and Affiliations

  1. Institute of Affective Computing and Information Processing, Southwest University, Chongqing, China

    Xinyue Zhong, Yutong Luo, Xiaomei Zeng & Guangyuan Liu

  2. School of Electronic and Information Engineering, Southwest University, Chongqing, China

    Xinyue Zhong, Yun Gu, Yutong Luo, Xiaomei Zeng & Guangyuan Liu

Authors
  1. Xinyue Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yun Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yutong Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaomei Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guangyuan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guangyuan Liu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhong, X., Gu, Y., Luo, Y. et al. Bi-hemisphere asymmetric attention network: recognizing emotion from EEG signals based on the transformer. Appl Intell 53, 15278–15294 (2023). https://doi.org/10.1007/s10489-022-04228-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04228-2

Keywords

Search

Navigation