research-article

Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels

Authors:

Wei-Bang Jiang,

Wei-Long Zheng,

Bao-Liang LuAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 5975 - 5984

https://doi.org/10.1145/3581783.3613797

Published: 27 October 2023 Publication History

Abstract

Emotion recognition from physiological signals is a topic of widespread interest, and researchers continue to develop novel techniques for perceiving emotions. However, the emergence of deep learning has highlighted the need for high-quality emotional datasets to accurately decode human emotions. In this study, we present a novel multimodal emotion dataset that incorporates electroencephalography (EEG) and eye movement signals to systematically explore human emotions. Seven basic emotions (happy, sad, fear, disgust, surprise, anger, and neutral) are elicited by a large number of 80 videos and fully investigated with continuous labels that indicate the intensity of the corresponding emotions. Additionally, we propose a novel Multimodal Adaptive Emotion Transformer (MAET), that can flexibly process both unimodal and multimodal inputs. Adversarial training is utilized in MAET to mitigate subject discrepancy, which enhances domain generalization. Our extensive experiments, encompassing both subject-dependent and cross-subject conditions, demonstrate MAET's superior performance in handling various inputs. The filtering of data for high emotional evocation using continuous labels proved to be effective in the experiments. Furthermore, the complementary properties between EEG and eye movements are observed. Our code is available at https://github.com/935963004/MAET.

References

[1]

Mehmet Berkehan Akçay and Kaya O?uz. 2020. Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers. Speech Communication 116 (2020), 56--76.

Digital Library

[2]

Soraia M. Alarcão and Manuel J. Fonseca. 2019. Emotions Recognition Using EEG Signals: A Survey. IEEE Transactions on Affective Computing 10, 3 (2019), 374--393. https://doi.org/10.1109/TAFFC.2017.2714671

Digital Library

[3]

Arjun Arjun, Aniket Singh Rajpoot, and Mahesh Raveendranatha Panicker. 2021. Introducing attention mechanism for EEG signals: Emotion recognition with vision transformers. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 5723--5726.

[4]

Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, and Furu Wei. 2021. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. arXiv preprint arXiv:2111.02358 (2021).

[5]

Danny Oude Bos et al. 2006. EEG-based emotion recognition. The Influence of Visual and Auditory Stimuli 56, 3 (2006), 1--17.

[6]

Margaret M Bradley, Laura Miccoli, Miguel A Escrig, and Peter J Lang. 2008. The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology 45, 4 (2008), 602--607.

[7]

Felipe Zago Canal, Tobias Rossi Müller, Jhennifer Cristine Matias, Gustavo Gino Scotton, Antonio Reis de Sa Junior, Eliane Pozzebon, and Antonio Carlos Sobieranski. 2022. A survey on facial emotion recognition techniques: A state-of-the-art literature review. Information Sciences 582 (2022), 593--617. https: //doi.org/10.1016/j.ins.2021.10.005

Digital Library

[8]

Xinyu Cheng, Wei Wei, Changde Du, Shuang Qiu, Sanli Tian, Xiaojun Ma, and Huiguang He. 2022. VigilanceNet: Decouple Intra-and Inter-Modality Learning for Multimodal Vigilance Estimation in RSVP-Based BCI. In Proceedings of the 30th ACM International Conference on Multimedia. 209--217.

Digital Library

[9]

Thomas Cover and Peter Hart. 1967. Nearest neighbor pattern classification. IEEE Transactions on Information Theory 13, 1 (1967), 21--27.

Digital Library

[10]

Fernando Lopes da Silva. 2013. EEG and MEG: relevance to neuroscience. Neuron 80, 5 (2013), 1112--1128.

[11]

Ruo-Nan Duan, Jia-Yi Zhu, and Bao-Liang Lu. 2013. Differential entropy feature for EEG-based emotion classification. In 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 81--84.

[12]

Monika Dubey and Lokesh Singh. 2016. Automatic emotion recognition using facial expression: a review. International Research Journal of Engineering and Technology (IRJET) 3, 2 (2016), 488--492.

[13]

Paul Ekman and Wallace V Friesen. 1971. Constants across cultures in the face and emotion. Journal of Personality and Social Psychology 17, 2 (1971), 124.

[14]

Moataz El Ayadi, Mohamed S. Kamel, and Fakhri Karray. 2011. Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition 44, 3 (2011), 572--587. https://doi.org/10.1016/j.patcog.2010.09.020

Digital Library

[15]

Sybil BG Eysenck, Hans J Eysenck, and Paul Barrett. 1985. A revised version of the psychoticism scale. Personality and Individual Differences 6, 1 (1985), 21--29.

[16]

N. Fragopanagos and J.G. Taylor. 2005. Emotion recognition in human--computer interaction. Neural Networks 18, 4 (2005), 389--405. https://doi.org/10.1016/j. neunet.2005.03.006

Digital Library

[17]

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The Journal of Machine Learning Research 17, 1 (2016), 2096--2030.

Digital Library

[18]

Alexandre Gramfort, Martin Luessi, Eric Larson, Denis A. Engemann, Daniel Strohmeier, Christian Brodbeck, Roman Goj, Mainak Jas, Teon Brooks, Lauri Parkkonen, and Matti S. Hämäläinen. 2013. MEG and EEG Data Analysis with MNE-Python. Frontiers in Neuroscience 7, 267 (2013), 1--13. https://doi.org/10. 3389/fnins.2013.00267

[19]

Kairui Guo, Rifai Chai, Henry Candra, Ying Guo, Rong Song, Hung Nguyen, and Steven Su. 2019. A hybrid fuzzy cognitive map/support vector machine approach for EEG-based emotion classification using compressed sensing. International Journal of Fuzzy Systems 21 (2019), 263--273.

[20]

Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016).

[21]

Yu-Liang Hsu, Jeen-Shing Wang, Wei-Chun Chiang, and Chien-Han Hung. 2017. Automatic ECG-based emotion recognition in music listening. IEEE Transactions on Affective Computing 11, 1 (2017), 85--99.

[22]

Xin Hu, Fei Wang, and Dan Zhang. 2022. Similar brains blend emotion in similar ways: Neural representations of individual difference in emotion profiles. Neuroimage 247 (2022), 118819.

[23]

S Jerritta, M Murugappan, Khairunizam Wan, and Sazali Yaacob. 2014. Emotion recognition from facial EMG signals using higher order statistics and principal component analysis. Journal of the Chinese Institute of Engineers 37, 3 (2014), 385--394.

[24]

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. 2022. Visual prompt tuning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part XXXIII. Springer, 709--727.

Digital Library

[25]

Ziyu Jia, Youfang Lin, Jing Wang, Zhiyang Feng, Xiangheng Xie, and Caijie Chen. 2021. HetEmotionNet: two-stream heterogeneous graph recurrent neural network for multi-modal emotion recognition. In Proceedings of the 29th ACM International Conference on Multimedia. 1047--1056.

Digital Library

[26]

Wei-Bang Jiang, Li-Ming Zhao, Ping Guo, and Bao-Liang Lu. 2021. Discriminating surprise and anger from EEG and eye movements with a graph network. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 1353--1357.

[27]

Stamos Katsigiannis and Naeem Ramzan. 2017. DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices. IEEE Journal of Biomedical and Health Informatics 22, 1 (2017), 98--107.

[28]

Ruhul Amin Khalil, Edward Jones, Mohammad Inayatullah Babar, Tariqullah Jan, Mohammad Haseeb Zafar, and Thamer Alhussain. 2019. Speech Emotion Recognition Using Deep Learning Techniques: A Review. IEEE Access 7 (2019), 117327--117345. https://doi.org/10.1109/ACCESS.2019.2936124

[29]

Jonghwa Kim and Elisabeth André. 2008. Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 12 (2008), 2067--2083.

Digital Library

[30]

Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. 2011. DEAP: A database for emotion analysis using physiological signals. IEEE Transactions on Affective Computing 3, 1 (2011), 18--31.

Digital Library

[31]

Jinpeng Li, Zhaoxiang Zhang, and Huiguang He. 2018. Hierarchical convolutional neural networks for EEG-based emotion recognition. Cognitive Computation 10 (2018), 368--380.

[32]

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Comput. Surveys 55, 9 (2023), 1--35.

Digital Library

[33]

Wei Liu, Jie-Lin Qiu, Wei-Long Zheng, and Bao-Liang Lu. 2021. Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition. IEEE Transactions on Cognitive and Developmental Systems 14, 2 (2021), 715--729.

[34]

Wei Liu, Wei-Long Zheng, and Bao-Liang Lu. 2016. Emotion recognition using multimodal deep learning. In International Conference on Neural Information Processing. Springer, 521--529.

Digital Library

[35]

Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).

[36]

Iris B. Mauss and Michael D. Robinson. 2009. Measures of emotion: A review. Cognition and Emotion 23, 2 (2009), 209--237. https://doi.org/10.1080/ 02699930802204677

[37]

RAM KUMAR Mdupu, CHIRANJEEVI Kothapalli, VASANTHI Yarra, SONTI Harika, and Cmak ZEELAN Basha. 2020. Automatic Human Emotion Recognition System using Facial Expressions with Convolution Neural Network. In 2020 4th International Conference on Electronics, Communication and Aerospace Technology (ICECA). 1179--1183. https://doi.org/10.1109/ICECA49313.2020.9297483

[38]

Wafa Mellouk and Wahida Handouzi. 2020. Facial emotion recognition using deep learning: review and insights. Procedia Computer Science 175 (2020), 689--694.

[39]

Shraddha A. Mithbavkar and Milind S. Shah. 2021. Analysis of EMG Based Emotion Recognition for Multiple People and Emotions. In 2021 IEEE 3rd Eurasia Conference on Biomedical Engineering, Healthcare and Sustainability (ECBIOS). 1--4. https://doi.org/10.1109/ECBIOS51820.2021.9510858

[40]

Hye Jeong Park and Jae Hwa Lee. 2020. Looking into the personality traits to enhance empathy ability: A review of literature. In HCI International 2020-Posters: 22nd International Conference, HCII 2020, Copenhagen, Denmark, July 19-24, 2020, Proceedings, Part I 22. Springer, 173--180.

[41]

R.W. Picard, E. Vyzas, and J. Healey. 2001. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 10 (2001), 1175--1191. https://doi.org/10.1109/34.954607

Digital Library

[42]

Aniket Singh Rajpoot, Mahesh Raveendranatha Panicker, et al. 2022. Subject independent emotion recognition using EEG signals employing attention driven neural networks. Biomedical Signal Processing and Control 75 (2022), 103547.

[43]

James A Russell. 1980. A circumplex model of affect. Journal of personality and social psychology 39, 6 (1980), 1161.

[44]

Heini Saarimäki, Athanasios Gotsopoulos, Iiro P. Jääskeläinen, Jouko Lampinen, Patrik Vuilleumier, Riitta Hari, Mikko Sams, and Lauri Nummenmaa. 2015. Discrete Neural Signatures of Basic Emotions. Cerebral Cortex 26, 6 (04 2015), 2563--2573. https://doi.org/ 10.1093/cercor/bhv086 arXiv:https://academic.oup.com/cercor/article-pdf/26/6/2563/17309892/bhv086.pdf

[45]

Tomasz Sapi?ski, Dorota Kami?ska, Adam Pelikant, and Gholamreza Anbarjafari. 2019. Emotion recognition from skeletal movements. Entropy 21, 7 (2019), 646.

[46]

Teresa Schreckenbach, Falk Ochsendorf, Jasmina Sterz, Miriam Rüsseler, Wolf Otto Bechstein, Bernd Bender, and Myriam N Bechtoldt. 2018. Emotion recognition and extraversion of medical students interact to predict their empathic communication perceived by simulated patients. BMC medical education 18, 1 (2018), 1--10.

[47]

Zhijuan Shen, Jun Cheng, Xiping Hu, and Qian Dong. 2019. Emotion Recognition Based on Multi-View Body Gestures. In 2019 IEEE International Conference on Image Processing (ICIP). 3317--3321. https://doi.org/10.1109/ICIP.2019.8803460

[48]

Li-Chen Shi and Bao-Liang Lu. 2010. Off-line and on-line vigilance estimation based on linear dynamical system and manifold learning. In 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology. 6587--6590. https://doi.org/10.1109/IEMBS.2010.5627125

[49]

Mohammad Soleymani, Sadjad Asghari-Esfeden, Yun Fu, and Maja Pantic. 2016. Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection. IEEE Transactions on Affective Computing 7, 1 (2016), 17--28. https://doi.org/10.1109/TAFFC.2015.2436926

Digital Library

[50]

Mohammad Soleymani, Jeroen Lichtenauer, Thierry Pun, and Maja Pantic. 2011. A multimodal database for affect recognition and implicit tagging. IEEE Transactions on Affective Computing 3, 1 (2011), 42--55.

Digital Library

[51]

Mohammad Soleymani, Maja Pantic, and Thierry Pun. 2012. Multimodal Emotion Recognition in Response to Videos. IEEE Transactions on Affective Computing 3, 2 (2012), 211--223. https://doi.org/10.1109/T-AFFC.2011.37

Digital Library

[52]

Tengfei Song, Wenming Zheng, Peng Song, and Zhen Cui. 2020. EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks. IEEE Transactions on Affective Computing 11, 3 (2020), 532--541. https://doi.org/10. 1109/TAFFC.2018.2817622

[53]

Bo Sun, Liandong Li, Xuewen Wu, Tian Zuo, Ying Chen, Guoyan Zhou, Jun He, and Xiaoming Zhu. 2016. Combining feature-level and decision-level fusion in a hierarchical classifier for emotion recognition in the wild. Journal on Multimodal User Interfaces 10 (2016), 125--137.

[54]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ?ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017).

[55]

Juan Vazquez-Rodriguez, Grégoire Lefebvre, Julien Cumin, and James L Crowley. 2022. Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals. In 2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 1--8.

[56]

Yiting Wang, Wei-Bang Jiang, Rui Li, and Bao-Liang Lu. 2021. Emotion transformer fusion: Complementary representation properties of EEG and eye movements on recognizing anger and surprise. In 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 1575--1578.

[57]

Zhe Wang, Yongxiong Wang, Chuanfei Hu, Zhong Yin, and Yu Song. 2022. Transformers for EEG-based emotion recognition: A hierarchical spatial information learning model. IEEE Sensors Journal 22, 5 (2022), 4359--4368.

[58]

D. Watts, R. F. Pulice, J. Reilly, A. R. Brunoni, F. Kapczinski, and I. C. Passos. 2022. Predicting treatment response using EEG in major depressive disorder: A machine-learning meta-analysis. Transl Psychiatry 12, 1 (2022), 332. https: //doi.org/10.1038/s41398-022-02064-z

[59]

YiFan Zhang, Xue Wang, Jian Liang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. 2023. Free Lunch for Domain Adversarial Training: Environment Label Smoothing. In International Conference on Learning Representations.

[60]

Li-Ming Zhao, Rui Li, Wei-Long Zheng, and Bao-Liang Lu. 2019. Classification of five emotions from EEG and eye movement signals: complementary representation properties. In 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, 611--614.

[61]

Wei-Long Zheng, Wei Liu, Yifei Lu, Bao-Liang Lu, and Andrzej Cichocki. 2018. Emotionmeter: A multimodal framework for recognizing human emotions. IEEE Transactions on Cybernetics 49, 3 (2018), 1110--1122.

[62]

Wei-Long Zheng and Bao-Liang Lu. 2015. Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks. IEEE Transactions on Autonomous Mental Development 7, 3 (2015), 162--175.

Digital Library

[63]

Peixiang Zhong, Di Wang, and Chunyan Miao. 2022. EEG-Based Emotion Recognition Using Regularized Graph Neural Networks. IEEE Transactions on Affective Computing 13, 3 (2022), 1290--1301. https://doi.org/10.1109/TAFFC.2020.2994159

Cited By

Xu ZChen NLi GLi JZhu HZhu Z(2025)The mitigation of heterogeneity in temporal scale among different cortical regions for EEG emotion recognitionKnowledge-Based Systems10.1016/j.knosys.2024.112826309(112826)Online publication date: Jan-2025
https://doi.org/10.1016/j.knosys.2024.112826
Dong YJing CMahmud MNg MWang S(2024)Enhancing cross-subject emotion recognition precision through unimodal EEG: a novel emotion preceptor modelBrain Informatics10.1186/s40708-024-00245-811:1Online publication date: 18-Dec-2024
https://doi.org/10.1186/s40708-024-00245-8
Jiang WLan YLu BCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)REmoNet: Reducing Emotional Label Noise via Multi-regularized Self-supervisionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681406(2204-2213)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681406
Show More Cited By

Index Terms

Multimodal Adaptive Emotion Transformer with Flexible Modality Inputs on A Novel Dataset with Continuous Labels
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence
      1. Cognitive science
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods

Recommendations

Multimodal Emotion Recognition in Response to Videos

This paper presents a user-independent emotion recognition method with the goal of recovering affective tags for videos using electroencephalogram (EEG), pupillary response and gaze distance. We first selected 20 video clips with extrinsic emotional ...
Emotion and mood recognition in response to video

This paper presents a subject-dependent homogenous emotion recognition method using electroencephalogram (EEG) signals in response to video contents, and a correlation between emotions and moods of subjects in resting state. In the recent years, there ...
AMHUSE: a multimodal dataset for HUmour SEnsing
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal Interaction

We present AMHUSE (A Multimodal dataset for HUmour SEnsing) along with a novel web-based annotation tool named DANTE (Dimensional ANnotation Tool for Emotions). The dataset is the result of an experiment concerning amusement elicitation, involving 36 ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Shanghai Municipal Science and Technology Major Project
Shanghai Pujiang Program
National Natural Science Foundation of China

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
730
Total Downloads

Downloads (Last 12 months)355
Downloads (Last 6 weeks)43

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xu ZChen NLi GLi JZhu HZhu Z(2025)The mitigation of heterogeneity in temporal scale among different cortical regions for EEG emotion recognitionKnowledge-Based Systems10.1016/j.knosys.2024.112826309(112826)Online publication date: Jan-2025
https://doi.org/10.1016/j.knosys.2024.112826
Dong YJing CMahmud MNg MWang S(2024)Enhancing cross-subject emotion recognition precision through unimodal EEG: a novel emotion preceptor modelBrain Informatics10.1186/s40708-024-00245-811:1Online publication date: 18-Dec-2024
https://doi.org/10.1186/s40708-024-00245-8
Jiang WLan YLu BCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)REmoNet: Reducing Emotional Label Noise via Multi-regularized Self-supervisionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681406(2204-2213)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681406
Ma SZhang YZhang QChen YWang HJia ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)SleepMG: Multimodal Generalizable Sleep Staging with Inter-modal Balance of Classification and Domain DiscriminationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680854(4004-4013)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680854
Gou MYin HLu BZheng W(2024)Multi-modal Adversarial Regressive Transformer for Cross-subject Fatigue Detection2024 46th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)10.1109/EMBC53108.2024.10782078(1-4)Online publication date: 15-Jul-2024
https://doi.org/10.1109/EMBC53108.2024.10782078
Liu XJiang WZheng WLu B(2024)MoGE: Mixture of Graph Experts for Cross-subject Emotion Recognition via Decomposing EEG2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)10.1109/BIBM62325.2024.10822354(3515-3520)Online publication date: 3-Dec-2024
https://doi.org/10.1109/BIBM62325.2024.10822354
Ma TLiu XZheng WLu B(2024)Emotion Recognition from Eye Movements Using Multi-way Autoregressive Model2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)10.1109/BIBM62325.2024.10821739(1078-1084)Online publication date: 3-Dec-2024
https://doi.org/10.1109/BIBM62325.2024.10821739
Zhao MCheng CFeng L(2024)Multimodal Physiological Signal Emotion Recognition Method Based on Attention Mechanism2024 International Conference on Artificial Intelligence, Deep Learning and Neural Networks (AIDLNN)10.1109/AIDLNN65358.2024.00043(222-226)Online publication date: 20-Sep-2024
https://doi.org/10.1109/AIDLNN65358.2024.00043
Dong YChen XShen YNg MQian TWang S(2024)Multi-modal Mood Reader: Pre-trained Model Empowers Cross-Subject Emotion RecognitionNeural Computing for Advanced Applications10.1007/978-981-97-7007-6_13(178-192)Online publication date: 22-Sep-2024
https://doi.org/10.1007/978-981-97-7007-6_13
Zhu QZheng CZhang ZShao WZhang D(2023)Dynamic Confidence-Aware Multi-Modal Emotion RecognitionIEEE Transactions on Affective Computing10.1109/TAFFC.2023.334092415:3(1358-1370)Online publication date: 8-Dec-2023
https://dl.acm.org/doi/10.1109/TAFFC.2023.3340924

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten