Abstract
Affective brain—computer interfaces have become an increasingly important topic to achieve emotional intelligence in human—machine collaboration. However, due to the complexity of electroencephalogram (EEG) signals and the individual differences in emotional response, it is still a great challenge to design a reliable and effective model. Considering the influence of personality traits on emotional response, it would be helpful to integrate personality information and EEG signals for emotion recognition. This study proposes a personality-guided attention neural network that can use personality information to learn effective EEG representations for emotion recognition. Specifically, we first use a convolutional neural network to extract rich temporal and regional representations of EEG signals, and a special convolution kernel is designed to learn inter- and intra-regional correlations simultaneously. Second, inspired by the fact that electrodes within distinct brain scalp regions play different roles in emotion recognition, a personality-guided regional-attention mechanism is proposed to further explore the contributions of electrodes within a region and between regions. Finally, attention-based long short-term memory is designed to explore the temporal dynamics of EEG signals. Experiments on the AMIGOS dataset, which is a dataset for multimodal research for affect, personality traits, and mood on individuals and groups, show that the proposed method can significantly improve the performance of subject-independent emotion recognition and outperform state-of-the-art methods.
摘要
情感脑机接口 (brain–computer interfaces, BCIs) 已成为在人机协作中实现情感智能的一个重要途径. 然而, 由于脑电图 (electroencephalogram, EEG) 信号的复杂性和情绪反应的个体差异性, 设计一个可靠和有效的模型仍然是一个巨大挑战. 考虑到不同人格特征的个体在情绪感知和反应过程中的差异, 整合人格信息和脑电信号对情绪识别是有帮助的. 鉴于此, 提出一种人格引导的注意力神经网络, 其可以利用人格信息学习更为有效的EEG表征以用于情感识别. 具体来说, 我们首先利用卷积神经网络提取脑电信号的时域和空域表征, 进而设计一种特殊的卷积核同时学习大脑头皮不同区域间和区域内的EEG导联相关关系. 其次, 考虑到不同大脑头皮区域在情绪识别中可能发挥不同的作用, 提出一种人格引导的区域注意力机制, 以进一步探索区域内和区域间EEG导联的贡献. 最后, 设计一种基于注意力的长短期记忆网络 (long short-term memory, LSTM) 建模EEG信号的时域动态特征. 在AMIGOS数据集 (一个用于个人和群体的情感、 人格特征和情绪多模态研究的数据集) 的实验结果表明, 本研究所提方法可以显著提升被试独立策略下情感识别的性能, 并优于现有情感识别方法.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alarcão SM, Fonseca MJ, 2019. Emotions recognition using EEG signals: a survey. IEEE Trans Affect Comput, 10(3):374–393. https://doi.org/10.1109/TAFFC.2017.2714671
Alhagry S, Fahmy AA, El-Khoribi RA, 2017. Emotion recognition based on EEG using LSTM recurrent neural network. Int J Adv Comput Sci Appl, 8(10):355–358. https://doi.org/10.14569/IJACSA.2017.081046
Alhussein M, 2016. Automatic facial emotion recognition using Weber local descriptor for e-Healthcare system. Clust Comput, 19(1):99–108. https://doi.org/10.1007/s10586-016-0535-3
Ayata D, Yaslan Y, Kamasak ME, 2018. Emotion based music recommendation system using wearable physiological sensors. IEEE Trans Consum Electron, 64(2):196–203. https://doi.org/10.1109/TCE.2018.2844736
Barkana BD, Ozkan Y, Badara JA, 2022. Analysis of working memory from EEG signals under different emotional states. Biomed Signal Process Contr, 71:103249. https://doi.org/10.1016/j.bspc.2021.103249
Bhardwaj A, Gupta A, Jain P, et al., 2015. Classification of human emotions from EEG signals using SVM and LDA classifiers. 2nd Int Conf on Signal Processing and Integrated Networks, p.180–185. https://doi.org/10.1109/SPIN.2015.7095376
Chaplin TM, 2015. Gender and emotion expression: a developmental contextual perspective. Emot Rev, 7(1):14–21. https://doi.org/10.1177/1754073914544408
Chen H, Song Y, Li XL, 2019. A deep learning framework for identifying children with ADHD using an EEG-based brain network. Neurocomputing, 356:83–96. https://doi.org/10.1016/j.neucom.2019.04.058
Chevalier P, Martin JC, Isableu B, et al., 2015. Impact of personality on the recognition of emotion expressed via human, virtual, and robotic embodiments. 24th IEEE Int Symp on Robot and Human Interactive Communication, p.229–234. https://doi.org/10.1109/ROMAN.2015.7333686
Cui H, Liu AP, Zhang X, et al., 2020. EEG-based emotion recognition using an end-to-end regional-asymmetric convolutional neural network. Knowl Based Syst, 205: 106243. https://doi.org/10.1016/j.knosys.2020.106243
Ding Y, Robinson N, Zeng QH, et al., 2020. TSception: a deep learning framework for emotion detection using EEG. Int Joint Conf on Neural Networks, p.1–7. https://doi.org/10.1109/IJCNN48605.2020.9206750
Ding Y, Hu X, Xia ZY, et al., 2021. Inter-brain EEG feature extraction and analysis for continuous implicit emotion tagging during video watching. IEEE Trans Affect Comput, 12(1):92–102. https://doi.org/10.1109/TAFFC.2018.2849758
Ding Y, Robinson N, Zhang S, et al., 2022. TSception: capturing temporal dynamics and spatial asymmetry from EEG for emotion recognition. IEEE Trans Affect Comput, early access. https://doi.org/10.1109/TAFFC.2022.3169001
Duan RN, Zhu JY, Lu BL, 2013. Differential entropy feature for EEG-based emotion classification. 6th Int IEEE/EMBS Conf on Neural Engineering, p.81–84. https://doi.org/10.1109/NER.2013.6695876
Etkin A, Egner T, Kalisch R, 2011. Emotional processing in anterior cingulate and medial prefrontal cortex. Trends Cogn Sci, 15(2):85–93. https://doi.org/10.1016/j.tics.2010.11.004
Farahat A, Reichert C, Sweeney-Reed CM, et al., 2019. Convolutional neural networks for decoding of covert attention focus and saliency maps for EEG feature visualization. J Neur Eng, 16(6):066010. https://doi.org/10.1088/1741-2552/ab3bb4
Fiterau M, Bhooshan S, Fries J, et al., 2017. ShortFuse: biomedical time series representations in the presence of structured information. Proc 2nd Machine Learning for Healthcare Conf, p.59–74.
Fossum TA, Barrett LF, 2000. Distinguishing evaluation from description in the personality-emotion relationship. Pers Soc Psychol Bull, 26(6):669–678. https://doi.org/10.1177/0146167200268003
Furnes D, Berg H, Mitchell RM, et al., 2019. Exploring the effects of personality traits on the perception of emotions from prosody. Front Psychol, 10:184. https://doi.org/10.3389/fpsyg.2019.00184
He KM, Zhang XY, Ren SQ, et al., 2016. Deep residual learning for image recognition. IEEE Conf on Computer Vision and Pattern Recognition, p.770–778. https://doi.org/10.1109/CVPR.2016.90
Kehoe EG, Toomey JM, Balsters JH, et al., 2012. Personality modulates the effects of emotional arousal and valence on brain activation. Soc Cogn Affect Neurosci, 7(7):858–870. https://doi.org/10.1093/scan/nsr059
Klados MA, Konstantinidi P, Dacosta-Aguayo R, et al., 2020. Automatic recognition of personality profiles using EEG functional connectivity during emotional processing. Brain Sci, 10(5):278. https://doi.org/10.3390/brainsci10050278
Koelstra S, Muhl C, Soleymani M, et al., 2012. DEAP: a database for emotion analysis using physiological signals. IEEE Trans Affect Comput, 3(1):18–31. https://doi.org/10.1109/T-AFFC.2011.15
Kragel PA, LaBar KS, 2016. Decoding the nature of emotion in the brain. Trends Cogn Sci, 20(6):444–455. https://doi.org/10.1016/j.tics.2016.03.011
Lan ZR, Sourina O, Wang LP, et al., 2016. Real-time EEG-based emotion monitoring using stable features. Vis Comput, 32(3):347–358. https://doi.org/10.1007/s00371-015-1183-y
Larsen RJ, Ketelaar T, 1991. Personality and susceptibility to positive and negative emotional states. J Pers Soc Psychol, 61(1):132–140. https://doi.org/10.1037/0022-3514.61.1.132
Lawhern VJ, Solon AJ, Waytowich NR, et al., 2018. EEG-Net: a compact convolutional neural network for EEG-based brain—computer interfaces. J Neur Eng, 15(5): 056013. https://doi.org/10.1088/1741-2552/aace8c
Li WY, Hu X, Long XF, et al., 2020. EEG responses to emotional videos can quantitatively predict Big-Five personality traits. Neurocomputing, 415:368–381. https://doi.org/10.1016/j.neucom.2020.07.123
Li X, Song DW, Zhang P, et al., 2016. Emotion recognition from multi-channel EEG data through convolutional recurrent neural network. IEEE Int Conf on Bioinformatics and Biomedicine, p.352–359. https://doi.org/10.1109/BIBM.2016.7822545
Li Y, Zheng WM, Zong Y, et al., 2021. A bi-hemisphere domain adversarial neural network model for EEG emotion recognition. IEEE Trans Affect Comput, 12(2):494–504. https://doi.org/10.1109/TAFFC.2018.2885474
Li Y, Zheng WM, Wang L, et al., 2022. From regional to global brain: a novel hierarchical spatial-temporal neural network model for EEG emotion recognition. IEEE Trans Affect Comput, 13(2):568–578. https://doi.org/10.1109/TAFFC.2019.2922912
Lindquist KA, Barrett LF, 2012. A functional architecture of the human brain: emerging insights from the science of emotion. Trends Cogn Sci, 16(11):533–540. https://doi.org/10.1016/j.tics.2012.09.005
Lotfi E, Akbarzadeh-T MR, 2014. Practical emotional neural networks. Neur Netw, 59:61–72. https://doi.org/10.1016/j.neunet.2014.06.012
Ma WF, Gong YF, Zhou GX, et al., 2021. A channel-mixing convolutional neural network for motor imagery EEG decoding and feature visualization. Biomed Signal Process Contr, 70:103021. https://doi.org/10.1016/j.bspc.2021.103021
Maaoui C, Pruski A, 2010. Emotion recognition through physiological signals for human-machine communication. In: Kordic V (Ed.), Cutting Edge Robotics 2010. IntechOpen, Rijeka, Croatia. https://doi.org/10.5772/10312
Martínez-Tejada LA, Maruyama Y, Yoshimura N, et al., 2020. Analysis of personality and EEG features in emotion recognition using machine learning techniques to classify arousal and valence labels. Mach Learn Knowl Extr, 2(2):99–124. https://doi.org/10.3390/make2020007
Miranda-Correa JA, Abadi MK, Sebe N, et al., 2021. AMIGOS: a dataset for affect, personality and mood research on individuals and groups. IEEE Trans Affect Comput, 12(2):479–493. https://doi.org/10.1109/TAFFC.2018.2884461
Mohammadi Z, Frounchi J, Amiri M, 2017. Wavelet-based emotion recognition system using EEG signal. Neur Comput Appl, 28(8):1985–1990. https://doi.org/10.1007/s00521-015-2149-8
Mühl C, Allison B, Nijholt A, et al., 2014. A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges. Brain Comput Interf, 1(2):66–84. https://doi.org/10.1080/2326263X.2014.912881
Niemic CP, 2002. Studies of emotion: a theoretical and empirical review of psychophysiological studies of emotion. J Undergrad Res, 1:15–18.
Orgeta V, 2009. Specificity of age differences in emotion regulation. Aging Ment Health, 13(6):818–826. https://doi.org/10.1080/13607860902989661
Özerdem MS, Polat H, 2017. Emotion recognition based on EEG features in movie clips with channel selection. Brain Inform, 4(4):241–252. https://doi.org/10.1007/s40708-017-0069-3
Perugini M, Di Blas L, 2002. Analyzing personality related adjectives from an eticemic perspective: the Big Five marker scales (BFMS) and the Italian AB5C taxonomy. In: de Raad B, Perugini M (Eds.), Big Five Assessment. Hogrefe & Huber Publishers, Seattle, USA, p.281–304.
Picard RW, Vyzas E, Healey J, 2001. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Patt Anal Mach Intell, 23(10):1175–1191. https://doi.org/10.1109/34.954607
Rao HC, Wang SQ, Hu XP, et al., 2021. A self-supervised gait encoding approach with locality-awareness for 3D skeleton based person re-identification. IEEE Trans Patt Anal Mach Intell, early access. https://doi.org/10.1109/TPAMI.2021.3092833
Revelle WR, Scherer KR, 2009. Personality and emotion. In: Sander D, Scherer KR (Eds.), Oxford Companion to Emotion and the Affective Sciences. Oxford University Press, Oxford, UK, p.304–305.
Rukavina S, Gruss S, Hoffmann H, et al., 2016. Affective computing and the impact of gender and age. PLoS ONE, 11(3):e0150584. https://doi.org/10.1371/journal.pone.0150584
Salovey P, Mayer JD, 1990. Emotional intelligence. Imaginat Cognit Pers, 9(3):185–211. https://doi.org/10.2190/DUGG-P24E-52WK-6CDG
Schirrmeister RT, Springenberg JT, Fiederer LDJ, et al., 2017. Deep learning with convolutional neural networks for EEG decoding and visualization. Hum Brain Mapp, 38(11):5391–5420. https://doi.org/10.1002/hbm.23730
Schmidtke JI, Heller W, 2004. Personality, affect and EEG: predicting patterns of regional brain activity related to extraversion and neuroticism. Pers Individ Differ, 36(3):717–732. https://doi.org/10.1016/S0191-8869(03)00129-6
Shen J, Zhang XW, Wang G, et al., 2022. An improved empirical mode decomposition of electroencephalogram signals for depression detection. IEEE Trans Affect Comput, 13(1):262–271. https://doi.org/10.1109/TAFFC.2019.2934412
Shen T, Zhou TY, Long GD, et al., 2018. DiSAN: directional self-attention network for RNN/CNN-free language understanding. Proc 32nd AAAI Conf on Artificial Intelligence, p.5446–5455. https://doi.org/10.1609/aaai.v32i1.11941
Shu YY, Wang SF, 2017. Emotion recognition through integrating EEG and peripheral signals. IEEE Int Conf on Acoustics, Speech and Signal Processing, p.2871–2875. https://doi.org/10.1109/ICASSP.2017.7952681
Steinert S, Friedrich O, 2020. Wired emotions: ethical issues of affective brain-computer interfaces. Sci Eng Ethics, 26(1):351–367. https://doi.org/10.1007/s11948-019-00087-2
Topic A, Russo M, Stella M, et al., 2022. Emotion recognition using a reduced set of EEG channels based on holographic feature maps. Sensors, 22(9):3248. https://doi.org/10.3390/s22093248
van Leeuwen KG, Sun H, Tabaeizadeh M, et al., 2019. Detecting abnormal electroencephalograms using deep convolutional networks. Clin Neurophysiol, 130(1):77–84. https://doi.org/10.1016/j.clinph.2018.10.012
Vaswani A, Shazeer N, Parmar N, et al., 2017. Attention is all you need. Proc 31st Int Conf on Neural Information Processing Systems, p.6000–6010.
Vuoskoski JK, Eerola T, 2011. The role of mood and personality in the perception of emotions represented by music. Cortex, 47(9):1099–1106. https://doi.org/10.1016/j.cortex.2011.04.011
Waytowich N, Lawhern VJ, Garcia JO, et al., 2018. Compact convolutional neural networks for classification of asynchronous steady-state visual evoked potentials. J Neur Eng, 15(6):066031. https://doi.org/10.1088/1741-2552/aae5d8
Wen ZY, Xu RF, Du JC, 2017. A novel convolutional neural networks for emotion recognition based on EEG signal. Int Conf on Security, Pattern Analysis, and Cybernetics, p.672–677. https://doi.org/10.1109/SPAC.2017.8304360
Wolpaw JR, Birbaumer N, McFarland DJ, et al., 2002. Brain—computer interfaces for communication and control. Clin Neurophysiol, 113(6):767–791. https://doi.org/10.1016/S1388-2457(02)00057-3
Xu SH, Rao HC, Peng H, et al., 2021. Attention-based multilevel co-occurrence graph convolutional LSTM for 3-D action recognition. IEEE Internet Things J, 8(21):15990–16001. https://doi.org/10.1109/JIOT.2020.3042986
Yang YL, Wu QF, Qiu M, et al., 2018. Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network. Int Joint Conf on Neural Networks, p.1–7. https://doi.org/10.1109/IJCNN.2018.8489331
Zhang DL, Yao LN, Zhang X, et al., 2018. Cascade and parallel convolutional recurrent neural networks on EEG-based intention recognition for brain computer interface. Proc 32nd AAAI Conf on Artificial Intelligence, p.1703–1710. https://doi.org/10.1609/aaai.v32i1.11496
Zhang DL, Yao LN, Chen KX, et al., 2019. A convolutional recurrent attention model for subject-independent EEG signal analysis. IEEE Signal Process Lett, 26(5):715–719. https://doi.org/10.1109/LSP.2019.2906824
Zhang GH, Yu MJ, Liu YJ, et al., 2021. SparseDGCNN: recognizing emotion from multichannel EEG signals. IEEE Trans Affect Comput, early access. https://doi.org/10.1109/TAFFC.2021.3051332
Zhang T, Cui Z, Xu CY, et al., 2020. Variational pathway reasoning for EEG emotion recognition. Proc 34th AAAI Conf on Artificial Intelligence, 2709–2716. https://doi.org/10.1609/aaai.v34i03.5657
Zhang XW, Li JL, Hou KC, et al., 2020. EEG-based depression detection using convolutional neural network with demographic attention mechanism. 42nd Annual Int Conf of the IEEE Engineering in Medicine & Biology Society, p.128–133. https://doi.org/10.1109/EMBC44109.2020.9175956
Zhang XW, Liu JY, Shen J, et al., 2021. Emotion recognition from multimodal physiological signals using a regularized deep fusion of kernel machine. IEEE Trans Cybern, 51(9):4386–4399. https://doi.org/10.1109/TCYB.2020.2987575
Zhang XW, Pan J, Shen J, et al., 2022. Fusing of electroencephalogram and eye movement with group sparse canonical correlation analysis for anxiety detection. IEEE Trans Affect Comput, 13(2):958–971. https://doi.org/10.1109/TAFFC.2020.2981440
Zhang YH, Prasad S, Kilicarslan A, et al., 2017. Multiple kernel based region importance learning for neural classification of gait states from EEG signals. Front Neurosci, 11:170. https://doi.org/10.3389/fnins.2017.00170
Zhao GZ, Zhang YL, Ge Y, et al., 2018a. Asymmetric hemisphere activation in tenderness: evidence from EEG signals. Sci Rep, 8(1):8029. https://doi.org/10.1038/s41598-018-26133-w
Zhao GZ, Ge Y, Shen BY, et al., 2018b. Emotion analysis for personality inference from EEG signals. IEEE Trans Affect Comput, 9(3):362–371. https://doi.org/10.1109/TAFFC.2017.2786207
Zhao GZ, Zhang YL, Ge Y, 2018c. Frontal EEG asymmetry and middle line power difference in discrete emotions. Front Behav Neurosci, 12:225. https://doi.org/10.3389/fnbeh.2018.00225
Zhao SC, Ding GG, Han JG, et al., 2018. Personality-aware personalized emotion recognition from physiological signals. Proc 27th Int Joint Conf on Artificial Intelligence, p.1660–1667. https://doi.org/10.24963/ijcai.2018/230
Zheng WL, Lu BL, 2015. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev, 7(3):162–175. https://doi.org/10.1109/TAMD.2015.2431497
Author information
Authors and Affiliations
Contributions
Shaojie LI and Wei LI designed the research and processed the data. Zejian XING and Wenjie YUAN drafted the paper. Xiangyu WEI helped organize the paper. Xiaowei ZHANG and Bin HU supervised the research, and revised and finalized the paper.
Corresponding authors
Ethics declarations
Shaojie LI, Wei LI, Zejian XING, Wenjie YUAN, Xiangyu WEI, Xiaowei ZHANG, and Bin HU declare that they have no conflict of interest.
Additional information
Project supported by the National Key R&D Program of China (No. 2019YFA0706200), the National Natural Science Foundation of China (Nos. 62072219 and 61632014), and the National Basic Research Program (973) of China (No. 2014CB744600)
Rights and permissions
About this article
Cite this article
Li, S., Li, W., Xing, Z. et al. A personality-guided affective brain—computer interface for implementation of emotional intelligence in machines. Front Inform Technol Electron Eng 23, 1158–1173 (2022). https://doi.org/10.1631/FITEE.2100489
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2100489