Abstract
Emotion is an intrinsic part of human nature, and it plays a significant role in how we (as humans) think and behave, which drives us to make decisions and take actions. Emotion can be recognized by processing different types of data. Since the identification of human behavior with just a single form of expression is typically hard and thus it make emotion recognition is a challenging task. Recent approaches have concentrated on a single modality of conversation for emotion recognition. However, purely relying on the single modality (a form of expression) of data may not capture emotion in-depth when multi-party and multimodality are involved in the conversion. To fill this gap, we have used Multimodal EmotionLines Dataset (MELD), a broader and enhanced version of the EmotionLine dataset. MELD dataset is built from Friends, an American television sitcom aired by NBC. In Emotion recognition, we propose to detect and recognize different emotions through text, video, and audio modalities. We extended the emotion analysis based on video modality from MELD dataset with reduced number of extracted pixel features. Our experiment results show that with video modality, we can still maintain an F-score of 59 with comparatively less features and reduced testing time by a factor of 88.89%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Poria, S., Majumder, N., Mihalcea, R., Hovy, E.: Emotion recognition in conversation: research challenges, datasets, and recent advances. IEEE Access 7, 100943–100953 (2019)
Hazarika, D.: Conversational memory network for emotion recognition in dyadic dialogue videos. In: Association for Computational Linguistics, vol. 1, pp. 2122–2132 (2018)
Hazarika, D., Poria, S., Zimmermann, R., Mihalcea, R.: Emotion Recognition in Conversations with Transfer Learning from Generative Conversation Modeling arXiv:1910.04980 (2019)
Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., Mihalcea, R.: MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations, pp. 527-536 (2018). https://doi.org/10.18653/v1/P19-1050
Hazarika, D., Poria, S., Mihalcea, R., Cambria, E., Zimmermann, R.: ICON: interactive conversational memory network for multimodal emotion detection. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2594–2604 (2018)
Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., Cambria, E.: DialogueRNN: An Attentive RNN for Emotion Detection in Conversations arXiv:1811.00405 (2018)
Ghosal, D., Majumder, N., Poria, S., Chhaya, N., Gelbukh, A.F.: DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation arXiv:1908.11540 (2019)
Tractica. https://www.businesswire.com/news/home/20180307005985/en/
Zhang, Y., Jin, R., Zhou, Z.-H.: Understanding bag-of-words model: a statistical framework. Int. J. Mach. Learn. Cybern. 1, 43–52 (2010). https://doi.org/10.1007/s13042-010-0001-0
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kharat, A., Patel, A., Bhatt, D., Parikh, N., Rathore, H. (2021). Emotion Recognition Using Multimodalities. In: Abraham, A., Hanne, T., Castillo, O., Gandhi, N., Nogueira Rios, T., Hong, TP. (eds) Hybrid Intelligent Systems. HIS 2020. Advances in Intelligent Systems and Computing, vol 1375. Springer, Cham. https://doi.org/10.1007/978-3-030-73050-5_31
Download citation
DOI: https://doi.org/10.1007/978-3-030-73050-5_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-73049-9
Online ISBN: 978-3-030-73050-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)