Abstract
Development of automated systems to recognize human emotions can enhance the quality of delivery of public health service to a great extent. Due to this reason, extensive researches have been started in the recent years to recognize human emotions. This investigation proposes a novel method of human emotion recognition using analyses of both physiological and text-based features. Two physiological features (facial expressions and heart rate variability) have been considered in the present investigation. Variability in facial expressions and heart rate corresponding to different emotions have been collected by showing a bunch of emotion motivation sample movie clips corresponding to each type of emotions. In text-based analysis, various subjective information related to each of the emotional categories have been analyzed from different blogs written by persons in different emotions. The physiological and text analysis-based features have been combined and studied in Recurrent Neural Network (RNN)-based classification platform, one of the deep learning methods. Two different models of RNN—Long Short-Term Memory and Bidirectional Long Short-Term Memory have been used for this purpose. The performance of the proposed system has been evaluated using one public as well as one self-generated dataset and it yields a high recognition accuracy.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
The datasets generated during the current study are available from the corresponding author on reasonable request.
References
Ghosh R, Sinha D (2019) Human emotion detection based on questionnaire and text analysis. Int J Work Organ Emotion 10(1):66–89
Soleymani M, Esfeden SA, Fu Y, Pantic M (2016) Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Trans Affect Comput 7(1):17–28
Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12
Zhang Y, Ji Q (2005) Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Trans Pattern Anal Mach Intell 27(5):699–714
Majumder A, Behera L, Subramanian VK (2014) Emotion recognition from geometric facial features using self-organizing map. Pattern Recognit 47:1282–1293
Jiang R, Ho ATS, Cheheb I, Maadeed NA, Maadeed SA, Bouridane A (2017) Emotion recognition from scrambled facial images via many graph embedding. Pattern Recognit 67:245–251
Nardelli M, Valenza G, Greco A, Lanata A, Scilingo EP (2015) Recognizing emotions induced by affective sounds through heart rate variability. IEEE Trans Affect Comput 6(4):385–394
Valenza G, Lanata A, Scilingo EP (2012) Oscillations of heart rate and respiration synchronize during affective visual stimulation. IEEE Trans Inf Technol Biomed 16(4):683–690
Li W, Xu H (2014) Text-based emotion classification using emotion cause extraction. Expert Syst Appl 41(4):1742–1749
Gao K, Xu H, Wang J (2015) A rule-based approach to emotion cause detection for Chinese micro-blogs. Expert Syst Appl 42:4517–4528
Gao Y, Zhu W (2016) Detecting affective states from text based on a multi-component emotion model. Comput Speech Lang 36:42–57
Mehrabian A (2007) Nonverbal communication. Aldine De Gruyter, Berlin
Camm AJ, Malik M, Bigger JT, Breithardt G, Cerutti S, Cohen RJ, Coumel P, Fallen EL, Kennedy HL, Kleiger RE, Lombardi F, Malliani A, Moss AJ, Rottman JN, Schmidt G, Schwartz PJ, Singer DH (1996) Heart rate variability: standards of measurement, physiological interpretation, and clinical use. Circulation 93(5):1043–1065
Ghosh R, Vamsi C, Kumar P (2019) RNN based online handwritten word recognition in Devanagari and Bengali scripts using horizontal zoning. Pattern Recognit 92:203–218
Zhang L, Tjondronegoro D (2011) Facial expression recognition using facial movement features. IEEE Trans Affect Comput 2(4):219–229
Kanjo E, Younis E, Ang CS (2019) Deep learning analysis of mobile physiological, environmental and location sensor data for emotion detection. Inf Fusion 49:46–56
Nicolaou M, Gunes H, Pantic M (2011) Continuous prediction of spontaneous affect from multiple cues and modalities in valence arousal space. IEEE Trans Affect Comput 2(2):92–105
McKeown G, Valstar M, Cowie R, Pantic M, Schroder M (2012) The SEMAINE database: annotated multimodal records of emotionally colored conversations between a person and a limited agent. IEEE Trans Affect Comput 3(1):5–17
Nicolaou MA, Gunes H, Pantic M (2012) Output-associative RVM regression for dimensional and continuous emotion prediction. Image Vis Comput 30(3):186–196
Baltrusaitis T, Banda N, Robinson P (2013) Dimensional affect recognition using continuous conditional random fields, In: Proceedings of the 10th IEEE international conference on automatic face and gesture recognition, Sanghai, China, pp. 1–8
Firdaus S, Ding C, Sadeghian A (2019) Topic specific emotion detection for retweet prediction. Int J Mach Learn Cybern 10(8):2071–2083
Chakraborty A, Konar A, Chakraborty UK, Chatterjee A (2009) Emotion recognition from facial expressions and its control using fuzzy logic. IEEE Trans Syst Man Cybern 39(4):726–743
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ghosh, R., Sinha, D. Human emotion recognition by analyzing facial expressions, heart rate and blogs using deep learning method. Innovations Syst Softw Eng 20, 499–507 (2024). https://doi.org/10.1007/s11334-022-00471-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11334-022-00471-5