Jiaxin Ma
ABSTRACT
Various studies have shown that the temporal information captured by conventional long-short-term memory (LSTM) networks is very useful for enhancing multimodal emotion recognition using encephalography (EEG) and other physiological signals. However, the dependency among multiple modalities and high-level temporal-feature learning using deeper LSTM networks is yet to be investigated. Thus, we propose a multimodal residual LSTM (MMResLSTM) network for emotion recognition. The MMResLSTM network shares the weights across the modalities in each LSTM layer to learn the correlation between the EEG and other physiological signals. It contains both the spatial shortcut paths provided by the residual network and temporal shortcut paths provided by LSTM for efficiently learning emotion-related high-level features. The proposed network was evaluated using a publicly available dataset for EEG-based emotion recognition, DEAP. The experimental results indicate that the proposed MMResLSTM network yielded a promising result, with a classification accuracy of 92.87% for arousal and 92.30% for valence.
- Salma Alhagry, Aly Aly, and Reda El-Khoribi. 2017. Emotion Recognition based on EEG using LSTM Recurrent Neural Network. International Journal of Advanced Computer Science and Applications 8, 10 (2017). https://doi.org/10.14569/IJACSA.2017.081046Google Scholar
Cross Ref
- Mouhannad Ali, Fadi Al Machot, Ahmad Haj Mosa, and Kyandoghere Kyamakya. 2016. A Novel EEG-based Emotion Recognition Approach for e-Healthcare Applications. In Proceedings of the 31st Annual ACM Symposium on Applied Computing (SAC '16). ACM, New York, NY, USA, 162--164. https://doi.org/10.1145/2851613.2851916Google ScholarDigital Library
- Pouya Bashivan, Irina Rish, Mohammed Yeasin, and Noel Codella. 2016. Learning representations from EEG with deep recurrent-convolutional neural networks. In International Conference on Learning Representations (ICLR).Google Scholar
- H. Becker, J. Fleureau, P. Guillotel, F. Wendling, I. Merlet, and L. Albera. 2018. Emotion recognition based on high-resolution EEG recordings and reconstructed brain sources. IEEE Transactions on Affective Computing(2018), 1--1. https://doi.org/10.1109/TAFFC.2017.2768030Google Scholar
- Carlos Busso, Zhigang Deng, Serdar Yildirim, Murtaza Bulut, Chul Min Lee,Abe Kazemzadeh, Sungbok Lee, Ulrich Neumann, and Shrikanth Narayanan. 2004. Analysis of Emotion Recognition Using Facial Expressions, Speech and Multimodal Information. In Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI '04). ACM, New York, NY, USA, 205--211. https://doi.org/10.1145/1027933.1027968Google ScholarDigital Library
- H. Candra, M. Yuwono, R. Chai, A. Handojoseno, I. Elamvazuthi, H. T. Nguyen,and S. Su. 2015. Investigation of window size in classification of EEG-emotion signal with wavelet entropy and support vector machine. In 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society(EMBC). 7250--7253. https://doi.org/10.1109/EMBC.2015.7320065Google Scholar
- Weitong Chen, Sen Wang, Xiang Zhang, Lina Yao, Lin Yue, Buyue Qian, and Xue Li. 2018. EEG-based Motion Intention Recognition via Multi-task RNNs. 279--287. https://doi.org/10.1137/1.9781611975321.32arXiv:https://epubs.siam.org/doi/pdf/10.1137/1.9781611975321.32Google Scholar
- Rahma Fourati, Boudour Ammar, Chaouki Aouiti, Javier Sanchez-Medina, and Adel M. Alimi. 2017. Optimized Echo State Network with Intrinsic Plasticity for EEG-Based Emotion Recognition. In Neural Information Processing, Derong Liu, Shengli Xie, Yuanqing Li, Dongbin Zhao, and El-Sayed M. El-Alfy (Eds.). Springer International Publishing, Cham, 718--727. https://doi.org/10.1007/978--3--319--70096-0_73Google Scholar
- Beatriz García-Martínez, Arturo Martínez-Rodrigo, Roberto Zangróniz, José Manuel Pastor, and Raúl Alcaraz. 2017. Symbolic Analysis of Brain Dynamics Detects Negative Stress. Entropy 19, 5 (2017). https://doi.org/10.3390/e19050196Google Scholar
- Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). IEEE, 6645--6649.Google ScholarCross Ref
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770--778. https://doi.org/10.1109/CVPR.2016.90Google Scholar
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory.Neural Comput. 9, 8 (Nov. 1997), 1735--1780. https://doi.org/10.1162/neco.1997.9.8.1735Google ScholarDigital Library
- Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37 (ICML'15). JMLR.org, 448--456. http://dl.acm.org/citation.cfm?id=3045118.3045167Google Scholar
- S. Katsigiannis and N. Ramzan. 2018. DREAMER: A Database for Emotion Recognition Through EEG and ECG Signals From Wireless Low-cost Off-the-Shelf Devices. IEEE Journal of Biomedical and Health Informatics 22, 1 (Jan 2018),98--107. https://doi.org/10.1109/JBHI.2017.2688239Google ScholarCross Ref
- B. H. Kim and S. Jo. 2018. Deep Physiological Affect Network for the Recognition of Human Emotions. IEEE Transactions on Affective Computing(2018), 1--1. https://doi.org/10.1109/TAFFC.2018.2790939Google Scholar
- S. Koelstra, C. Muhl, M. Soleymani, J. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras. 2012. DEAP: A Database for Emotion Analysis using Physiological Signals. IEEE Transactions on Affective Computing3, 1 (Jan 2012), 18--31. https://doi.org/10.1109/T-AFFC.2011.15Google ScholarDigital Library
- Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450(2016).Google Scholar
- Wenqian Lin, Chao Li, and Shouqian Sun. 2017. Deep Convolutional NeuralNetwork for Emotion Recognition Using EEG and Peripheral Physiological Signal. In Image and Graphics, Yao Zhao, Xiangwei Kong, and David Taubman(Eds.). Springer International Publishing, Cham, 385--394. https://doi.org/10.1007/978--3--319--71589--6_33Google Scholar
- Wei Liu, Wei-Long Zheng, and Bao-Liang Lu. 2016. Emotion Recognition Using Multimodal Deep Learning. In Proceedings of the 23rd International Conference onNeural Information Processing - Volume 9948. Springer-Verlag, Berlin, Heidelberg, 521--529. https://doi.org/10.1007/978--3--319--46672--9_58Google ScholarDigital Library
- Y. Liu and O. Sourina. 2013. EEG Databases for Emotion Recognition. In 2013 International Conference on Cyberworlds. 302--309. https://doi.org/10.1109/CW.2013.52Google Scholar
- Yifei Lu, Wei-Long Zheng, Binbin Li, and Bao-Liang Lu. 2015. Combining Eye Movements and EEG to Enhance Emotion Recognition. In Proceedings of the 24th International Conference on Artificial Intelligence (IJCAI'15). AAAI Press, 1170--1176. http://dl.acm.org/citation.cfm?id=2832249.2832411Google ScholarDigital Library
- Pallavi Pandey and K. R. Seeja. 2019. Emotional State Recognition with EEGSignals Using Subject Independent Approach. InData Science and Big Data Analytics, Durgesh Kumar Mishra, Xin-She Yang, and Aynur Unal (Eds.). Springer Singapore, Singapore, 117--124. https://doi.org/10.1007/978--981--10--7641--1_10Google Scholar
- V. Pham, T. Bluche, C. Kermorvant, and J. Louradour. 2014. Dropout Improves Recurrent Neural Networks for Handwriting Recognition. In 2014 14th Inter-national Conference on Frontiers in Handwriting Recognition. 285--290. https://doi.org/10.1109/ICFHR.2014.55Google Scholar
- L. Piho and T. Tjahjadi. 2018. A mutual information based adaptive windowing of informative EEG for emotion recognition. IEEE Transactions on Affective Computing(2018), 1--1. https://doi.org/10.1109/TAFFC.2018.2840973Google Scholar
- Jie-Lin Qiu, Wei Liu, and Bao-Liang Lu. 2018. Multi-view Emotion Recognition Using Deep Canonical Correlation Analysis. In Proceedings of the 25th International Conference on Neural Information Processing (ICONIP'18). 221--231.https://doi.org/10.1007/978--3-030-04221--9_20Google ScholarDigital Library
- Jimmy Ren, Yongtao Hu, Yu-Wing Tai, Chuan Wang, Li Xu, Wenxiu Sun, and Qiong Yan. 2016. Look, Listen and Learn - a Multimodal LSTM for Speaker Identification. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI'16). AAAI Press, 3581--3587. http://dl.acm.org/citation.cfm?id=3016387.3016407Google ScholarDigital Library
- M. Soleymani, S. Asghari-Esfeden, M. Pantic, and Y. Fu. 2014. Continuous emotion detection using EEG signals and facial expressions. In 2014 IEEE International Conference on Multimedia and Expo (ICME). 1--6. https://doi.org/10.1109/ICME.2014.6890301Google ScholarCross Ref
- M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic. 2012. A Multimodal Databasefor Affect Recognition and Implicit Tagging. IEEE Transactions on Affective Computing 3, 1 (Jan 2012), 42--55. https://doi.org/10.1109/T-AFFC.2011.25Google ScholarDigital Library
- Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training Very Deep Networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2 (NIPS'15). MIT Press, Cambridge, MA, USA, 2377--2385. http://dl.acm.org/citation.cfm?id=2969442.2969505Google Scholar
- Akara Supratak, Hao Dong, Chao Wu, and Yike Guo. 2017. DeepSleepNet: aModel for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG. IEEE Transactions on Neural Systems and Rehabilitation Engineering 25, 11 (Nov 2017), 1998--2008. https://doi.org/10.1109/TNSRE.2017.2721116Google ScholarCross Ref
- C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,V. Vanhoucke, and A. Rabinovich. 2015. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1--9. https://doi.org/10.1109/CVPR.2015.7298594Google ScholarCross Ref
- Hao Tang, Wei Liu, Wei-Long Zheng, and Bao-Liang Lu. 2017. Multimodal Emotion Recognition Using Deep Neural Networks. In Neural Information Processing, Derong Liu, Shengli Xie, Yuanqing Li, Dongbin Zhao, and El-Sayed M. El-Alfy(Eds.). Springer International Publishing, Cham, 811--819. https://doi.org/10.1007/978--3--319--70093--9_86Google Scholar
- Gyanendra K. Verma and Uma Shanker Tiwary. 2017. Affect Representation and Recognition in 3D Continuous Valence--arousal--dominance Space.Multimedia Tools Appl.76, 2 (Jan. 2017), 2159--2183. https://doi.org/10.1007/s11042-015--3119-yGoogle Scholar
- O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. 2015. Show and tell: A neuralimage caption generator. In 2015 IEEE Conference on Computer Vision and PatternRecognition (CVPR). 3156--3164. https://doi.org/10.1109/CVPR.2015.7298935Google Scholar
- Xiao-Wei Wang, Dan Nie, and Bao-Liang Lu. 2014. Emotional state classification from EEG data using machine learning approach. Neurocomputing 129 (Apr.2014), 94--106. https://doi.org/10.1016/j.neucom.2013.06.046Google Scholar
- Zhong Yin, Mengyuan Zhao, Yongxiong Wang, Jingdong Yang, and Jianhua Zhang. 2017. Recognition of emotions using multimodal physiological signalsand an ensemble deep learning model. Computer Methods and Programs in Biomedicine140 (2017), 93--110. https://doi.org/10.1016/j.cmpb.2016.12.005Google Scholar
- Wei-Long Zheng, Wei Liu, Yifei Lu, Bao-Liang Lu, and Andrzej Cichocki. 2018. Emotionmeter: A multimodal framework for recognizing human emotions. IEEE Transactions on Cybernetics 49, 3 (Feb. 2018), 1110--1122. https://doi.org/10.1109/TCYB.2018.2797176Google Scholar
- Wei-Long Zheng and Bao-Liang Lu. 2015. Investigating Critical Frequency Bandsand Channels for EEG-Based Emotion Recognition with Deep Neural Networks.IEEE Transactions on Autonomous Mental Development 7, 3 (Sep. 2015), 162--175.https://doi.org/10.1109/TAMD.2015.2431497Google Scholar
- Wei-Long Zheng, Jia-Yi Zhu, and Bao-Liang Lu. 2018. Identifying Stable Patternsover Time for Emotion Recognition from EEG. IEEE Transactions on Affective Computing(2018), 1--1. https://doi.org/10.1109/TAFFC.2017.2712143Google Scholar
- Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. 2017. Recurrent Highway Networks. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.), Vol. 70. PMLR, International Convention Centre, Sydney, Australia, 4189--4198. http://proceedings.mlr.press/v70/zilly17a.htmlGoogle Scholar
Index Terms
- Emotion Recognition using Multimodal Residual LSTM Network
Recommendations
Emotion Recognition in Conversations Using Brain and Physiological Signals
IUI '22: Proceedings of the 27th International Conference on Intelligent User InterfacesEmotions are complicated psycho-physiological processes that are related to numerous external and internal changes in the body. They play an essential role in human-human interaction and can be important for human-machine interfaces. Automatically ...
Student's Emotion Recognition using Multimodality and Deep Learning
The goal of emotion detection is to find and recognise emotions in text, speech, gestures, facial expressions, and more. This paper proposes an effective multimodal emotion recognition system based on facial expressions, sentence-level text, and voice. ...
Investigating Gender Differences of Brain Areas in Emotion Recognition Using LSTM Neural Network
Neural Information ProcessingAbstractIn this paper, we investigate key brain areas of men and women using electroencephalography (EEG) data on recognising three emotions, namely happy, sad and neutral. Considering that emotion changes over time, Long Short-Term Memory (LSTM) neural ...
Comments