skip to main content
10.1145/3474085.3475697acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

A Multi-Domain Adaptive Graph Convolutional Network for EEG-based Emotion Recognition

Published: 17 October 2021 Publication History

Abstract

Among all solutions of emotion recognition tasks, electroencephalogram (EEG) is a very effective tool and has received broad attention from researchers. In addition, information across multimedia in EEG often provides a more complete picture of emotions. However, few of the existing studies concurrently incorporate EEG information from temporal domain, frequency domain and functional brain connectivity. In this paper, we propose a Multi-Domain Adaptive Graph Convolutional Network (MD-AGCN), fusing the knowledge of both the frequency domain and the temporal domain to fully utilize the complementary information of EEG signals. MD-AGCN also considers the topology of EEG channels by combining the inter-channel correlations with the intra-channel information, from which the functional brain connectivity can be learned in an adaptive manner. Extensive experimental results demonstrate that our model exceeds state-of-the-art methods in most experimental settings. At the same time, the results show that MD-AGCN could extract complementary domain information and exploit channel relationships for EEG-based emotion recognition effectively.

References

[1]
Fatemeh Bahari and Amin Janghorbani. 2013. EEG-based emotion recognition using recurrence plot analysis and k nearest neighbor classifier. In 2013 20th Iranian Conference on Biomedical Engineering (ICBME). IEEE, 228--233.
[2]
Andrey V Bocharov, Gennady G Knyazev, and Alexander N Savostyanov. 2017. Depression and implicit emotion processing: An EEG study. Neurophysiologie Clinique/Clinical Neurophysiology, Vol. 47, 3 (2017), 225--230.
[3]
Ruo-Nan Duan, Jia-Yi Zhu, and Bao-Liang Lu. 2013. Differential entropy feature for EEG-based emotion classification. In 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 81--84.
[4]
Lester I Goldfischer. 1965. Autocorrelation function and power spectral density of laser-produced speckle patterns. Josa, Vol. 55, 3 (1965), 247--253.
[5]
Matti Hämäläinen, Riitta Hari, Risto J Ilmoniemi, Jukka Knuutila, and Olli V Lounasmaa. 1993. Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain. Reviews of Modern Physics, Vol. 65, 2 (1993), 413.
[6]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and Pattern Recognition. 770--778.
[7]
Tiffany C Ho, Colm G Connolly, Eva Henje Blom, Kaja Z LeWinn, Irina A Strigo, Martin P Paulus, Guido Frank, Jeffrey E Max, Jing Wu, Melanie Chan, et al. 2015. Emotion-dependent functional connectivity of the default mode network in adolescent depression. Biological Psychiatry, Vol. 78, 9 (2015), 635--646.
[8]
Robert Jenke, Angelika Peer, and Martin Buss. 2014. Feature extraction and selection for emotion recognition from EEG. IEEE Transactions on Affective Computing, Vol. 5, 3 (2014), 327--339.
[9]
Ziyu Jia, Youfang Lin, Xiyang Cai, Haobin Chen, Haijun Gou, and Jing Wang. 2020. SST-EmotionNet: Spatial-spectral-temporal based attention 3d dense network for EEG emotion recognition. In Proceedings of the 28th ACM International Conference on Multimedia. 2909--2917.
[10]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature, Vol. 521, 7553 (2015), 436--444.
[11]
You-Yun Lee and Shulan Hsieh. 2014. Classifying different emotional states by means of EEG-based functional connectivity patterns. PloS one, Vol. 9, 4 (2014), e95415.
[12]
Tian-Hao Li, Wei Liu, Wei-Long Zheng, and Bao-Liang Lu. 2019 a. Classification of five emotions from EEG and eye movement signals: Discrimination ability and stability over time. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 607--610.
[13]
Yang Li, Lei Wang, Wenming Zheng, Yuan Zong, Lei Qi, Zhen Cui, Tong Zhang, and Tengfei Song. 2020. A novel bi-hemispheric discrepancy model for eeg emotion recognition. IEEE Transactions on Cognitive and Developmental Systems, Vol. 13, 2 (2020), 354--367.
[14]
Yang Li, Wenming Zheng, Zhen Cui, Tong Zhang, and Yuan Zong. 2018. A Novel Neural Network Model based on Cerebral Hemispheric Asymmetry for EEG Emotion Recognition. In IJCAI. 1561--1567.
[15]
Yang Li, Wenming Zheng, Lei Wang, Yuan Zong, and Zhen Cui. 2019 b. From regional to global brain: A novel hierarchical spatial-temporal neural network model for EEG emotion recognition. IEEE Transactions on Affective Computing (2019).
[16]
Wei Liu, Jie-Lin Qiu, Wei-Long Zheng, and Bao-Liang Lu. 2021. Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition. IEEE Transactions on Cognitive and Developmental Systems (2021).
[17]
Yifei Lu, Wei-Long Zheng, Binbin Li, and Bao-Liang Lu. 2015. Combining eye movements and EEG to enhance emotion recognition. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
[18]
P MatLab. 2018. 9.7. 0.1190202 (R2019b). MathWorks Inc Natick MA USA (2018).
[19]
Iris B Mauss and Michael D Robinson. 2009. Measures of emotion: A review. Cognition and Emotion, Vol. 23, 2 (2009), 209--237.
[20]
Michael Murias, Sara J Webb, Jessica Greenson, and Geraldine Dawson. 2007. Resting state cortical connectivity reflected in EEG coherence in individuals with autism. Biological Psychiatry, Vol. 62, 3 (2007), 270--273.
[21]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).
[22]
Louis A Schmidt and Laurel J Trainor. 2001. Frontal brain electrical activity (EEG) distinguishes valence and intensity of musical emotions. Cognition & Emotion, Vol. 15, 4 (2001), 487--500.
[23]
Maryam M Shanechi. 2019. Brain--machine interfaces from motor to mood. Nature Neuroscience, Vol. 22, 10 (2019), 1554--1564.
[24]
Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. 2019. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12026--12035.
[25]
Tengfei Song, Wenming Zheng, Peng Song, and Zhen Cui. 2018. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Transactions on Affective Computing, Vol. 11, 3 (2018), 532--541.
[26]
Xiao-Wei Wang, Dan Nie, and Bao-Liang Lu. 2014. Emotional state classification from EEG data using machine learning approach. Neurocomputing, Vol. 129 (2014), 94--106.
[27]
Alexis E Whitton, Stephanie Deccy, Manon L Ironside, Poornima Kumar, Miranda Beltzer, and Diego A Pizzagalli. 2018. Electroencephalography source functional connectivity reveals abnormal high-frequency communication among large-scale functional networks in depression. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, Vol. 3, 1 (2018), 50--58.
[28]
Xun Wu, Wei-Long Zheng, and Bao-Liang Lu. 2019. Identifying functional brain connectivity patterns for EEG-based emotion recognition. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 235--238.
[29]
Yongqiang Yin, Xiangwei Zheng, Bin Hu, Yuang Zhang, and Xinchun Cui. 2021. EEG emotion recognition using fusion model of graph convolutional neural networks and LSTM. Applied Soft Computing, Vol. 100 (2021), 106954.
[30]
Zhongliang Yin, Jun Li, Yun Zhang, Aifeng Ren, Karen M Von Meneen, and Liyu Huang. 2017. Functional brain network analysis of schizophrenic patients with positive and negative syndrome based on mutual information of EEG time series. Biomedical Signal Processing and Control, Vol. 31 (2017), 331--338.
[31]
Guanhua Zhang, Minjing Yu, Yong-Jin Liu, Guozhen Zhao, Dan Zhang, and Wenming Zheng. 2021. SparseDGCNN: Recognizing Emotion from Multichannel EEG Signals. IEEE Transactions on Affective Computing (2021).
[32]
Tong Zhang, Wenming Zheng, Zhen Cui, Yuan Zong, and Yang Li. 2018. Spatial--temporal recurrent neural network for emotion recognition. IEEE Transactions on Cybernetics, Vol. 49, 3 (2018), 839--847.
[33]
Li-Ming Zhao, Rui Li, Wei-Long Zheng, and Bao-Liang Lu. 2019. Classification of five emotions from EEG and eye movement signals: complementary representation properties. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 611--614.
[34]
Wei-Long Zheng, Bo-Nan Dong, and Bao-Liang Lu. 2014. Multimodal emotion recognition using EEG and eye tracking data. In 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 5040--5043.
[35]
Wei-Long Zheng, Wei Liu, Yifei Lu, Bao-Liang Lu, and Andrzej Cichocki. 2018. Emotionmeter: A multimodal framework for recognizing human emotions. IEEE Transactions on Cybernetics, Vol. 49, 3 (2018), 1110--1122.
[36]
Wei-Long Zheng and Bao-Liang Lu. 2015. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development, Vol. 7, 3 (2015), 162--175.
[37]
Wei-Long Zheng, Jia-Yi Zhu, and Bao-Liang Lu. 2017. Identifying stable patterns over time for emotion recognition from EEG. IEEE Transactions on Affective Computing, Vol. 10, 3 (2017), 417--429.
[38]
Peixiang Zhong, Di Wang, and Chunyan Miao. 2020. EEG-based emotion recognition using regularized graph neural networks. IEEE Transactions on Affective Computing (2020).

Cited By

View all
  • (2025)Fusing temporal-frequency information with Contrast Learning on Graph Convolution Network to decoding EEGBiomedical Signal Processing and Control10.1016/j.bspc.2024.106986100(106986)Online publication date: Feb-2025
  • (2025)Multi-source-Domain Adaptation for TMS-EEG Based Depression DetectionSocial Robotics10.1007/978-981-96-1151-5_26(250-261)Online publication date: 7-Feb-2025
  • (2024)A Comprehensive Survey on Emerging Techniques and Technologies in Spatio-Temporal EEG Data AnalysisChinese Journal of Information Fusion10.62762/CJIF.2024.8768301:3(183-211)Online publication date: 15-Dec-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MM '21: Proceedings of the 29th ACM International Conference on Multimedia
October 2021
5796 pages
ISBN:9781450386517
DOI:10.1145/3474085
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. adaptive graph convolutional network
  2. affective computing
  3. eeg-based emotion recognition
  4. functional brain connectivity

Qualifiers

  • Research-article

Conference

MM '21
Sponsor:
MM '21: ACM Multimedia Conference
October 20 - 24, 2021
Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)193
  • Downloads (Last 6 weeks)10
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Fusing temporal-frequency information with Contrast Learning on Graph Convolution Network to decoding EEGBiomedical Signal Processing and Control10.1016/j.bspc.2024.106986100(106986)Online publication date: Feb-2025
  • (2025)Multi-source-Domain Adaptation for TMS-EEG Based Depression DetectionSocial Robotics10.1007/978-981-96-1151-5_26(250-261)Online publication date: 7-Feb-2025
  • (2024)A Comprehensive Survey on Emerging Techniques and Technologies in Spatio-Temporal EEG Data AnalysisChinese Journal of Information Fusion10.62762/CJIF.2024.8768301:3(183-211)Online publication date: 15-Dec-2024
  • (2024)VSGTProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/341(3078-3086)Online publication date: 3-Aug-2024
  • (2024)EEG emotion recognition based on differential entropy feature matrix through 2D-CNN-LSTM networkEURASIP Journal on Advances in Signal Processing10.1186/s13634-024-01146-y2024:1Online publication date: 8-Apr-2024
  • (2024)Joint Contrastive Learning with Feature Alignment for Cross-Corpus EEG-based Emotion RecognitionProceedings of the 1st International Workshop on Brain-Computer Interfaces (BCI) for Multimedia Understanding10.1145/3688862.3689112(9-17)Online publication date: 28-Oct-2024
  • (2024)Self-Supervised EEG Representation Learning for Robust Emotion RecognitionACM Transactions on Sensor Networks10.1145/367497520:5(1-22)Online publication date: 5-Jul-2024
  • (2024)Multi-View Hierarchical Attention Graph Convolutional Network with Domain Adaptation for EEG Emotion RecognitionProceedings of the 2024 3rd International Conference on Cryptography, Network Security and Communication Technology10.1145/3673277.3673384(624-630)Online publication date: 19-Jan-2024
  • (2024)Research Progress of EEG-Based Emotion Recognition: A SurveyACM Computing Surveys10.1145/366600256:11(1-49)Online publication date: 8-Jul-2024
  • (2024)WSEL: EEG Feature Selection with Weighted Self-expression Learning for Incomplete Multi-dimensional Emotion RecognitionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681570(350-359)Online publication date: 28-Oct-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media