Skip to main content
Springer Nature Link
Log in

Text-based emotion recognition using contextual phrase embedding model

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, the proposed approach categories the sentences in the dataset into the various topical documents using the TE-LSTM+SC model. As well as, the model generates semantic words related to topics that are fed into the word embedding like Skip-Gram and FrameNet to build the domain-specific lexicon. The topically related sentences in each document are contextually grouped using Skip-Phrase. Each sentence in contextual group is given to Semantic Role Labelling (SRL). SRL indentify the essential predicate-argument structures with the semantic labels like verb (V) tag or ARGM-NEG or ARGM-PRP or ARGM-CAU or structures with the semantic labels like verb (V) tag or ARGM-NEG or ARGM-PRP or ARGM-CAU or ARGM-MNR or ARGM-MOD. The selected predicate-argument structures are aggregated into a linear layer to form a semantic embedding. Simultaneously, the predicate-argument embedding is segmented to sub words by BERT. The sub-words are transformed to word level through a convolutional layer to acquire the contextual word representation. Finally, semantic embedding and word representation are integrated to efficiently find the emotion of the given sentence. The experimental result proved that the proposed approach outperforms all the state-of-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

The ISEAR, CrowdFlower, TEC and Tales, ECE and ECPE dataset utilized in this work for experimental evaluation purpose are publicly available datasets introduced by [10, 33, 43, 58, 62] and [74], respectively. The data generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Alm CO, Roth D, Sproat R (2005) Emotions from text: machine learning for text-based emotion prediction. In: Proceedings of the conference on human language technology and empirical methodsin natural language processing- HLT’05. Association for computational linguistics, pp 579–586. https://doi.org/10.3115/1220575.1220648

  2. Alshaabi T, Van Oort CM, Fudolig MI, Arnold MV, Danforth CM, Dodds PS (2022) Augmenting semantic lexicons using word embeddings and transfer learning. Front Artif Intell 4:783778. https://doi.org/10.3389/frai.2021.783778

    Article  Google Scholar 

  3. Araque O, Gatti L, Staiano J, Guerini M (2019) Depechemood++: a bilingual emotion lexicon built through simple yet powerful techniques. IEEE Trans Affect Comput:1–1. https://doi.org/10.1109/TAFFC.2019.2934444

  4. Baker CF, Fillmore CJ, Lowe JB (1998) The berkeley framenet project. In: 36th Annual meeting of the association for computational linguistics and 17th international conference on computational linguistics. Association for computational linguistics, vol 1, pp 86–90. https://doi.org/10.3115/980845.980860

  5. Batbaatar E, Li M, Ryu KH (2019) Semantic-Emotion Neural network for emotion recognition from text. IEEE Access 7:111866–111878. https://doi.org/10.1109/ACCESS.2019.2934529

    Article  Google Scholar 

  6. Bostan LAM, Klinger R (2018) An analysis of annotated corpora for emotion classification in text, vol 16

  7. Cai D, He X, Han J (2011) Locally consistent concept factorization for document clustering. IEEE Trans Knowl Data Eng 23(6):902–913. https://doi.org/10.1109/TKDE.2010.165

    Article  Google Scholar 

  8. Cai Y, Huang Q, Lin Z, Xu J, Chen Z, Li Q (2020) Recurrent neural network with pooling operation and attention mechanism for sentiment analysis: a multi-task learning approach. Knowl-Based Syst 203:105856. https://doi.org/10.1016/j.knosys.2020.105856

    Article  Google Scholar 

  9. Chen Z, Huang D, Wang Y, Chen L (2018) Fast and light manifold CNN based 3D facial expression recognition across pose variations. In: Proceedings of the 26th ACM international conference on multimedia. MM ’18. Association for computing machinery, pp 229–238. https://doi.org/10.1145/3240508.3240568

  10. Crowdflower (2016) Sentiment analysis in text- dataset by crowdflower. https://data.world/crowdflower/sentiment-analysis-in-text

  11. Cui W, Li B, Yang X, Wang W, Wang B, Zhang X (2018) An adaptive hierarchical compositional model for phrase embedding, pp 4144–4151

  12. Deerwester S, Dumais ST, Furnas GW, Landauer TK, Harshman R (1990) Indexing by latent semantic analysis. J Am Soc Inf Sci 41(6):391–407. https://doi.org/10.1002/(SICI)1097-4571(199009)41:6%3C391::AID-ASI1%3E3.0.CO;2-9

    Article  Google Scholar 

  13. Devlin J, Chang M-W, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, vol 1 (long and short papers). Association for computational linguistics, Minneapolis, Minnesota, pp 4171–4186. https://doi.org/10.18653/v1/N19-1423

  14. Ding Z, He H, Zhang M, Xia R (2019) From independent prediction to re-ordered prediction: integrating relative position and global label information to emotion cause identification. arXiv:1906.01230

  15. Ding M, Yang H, Zhou C, Tang J (2020) CogLTX: applying BERT to long texts. vol 13

  16. Dowdy S, Wearden S, Chilko D (2004) Statistics for research: 441, 3rd edn. Wiley-Interscience

  17. Er MJ, Zhang Y, Wang N, Pratama M (2016) Attention pooling-based convolutional neural network for sentence modelling. Inf Sci 373:388–403. https://doi.org/10.1016/j.ins.2016.08.084

    Article  MATH  Google Scholar 

  18. Gómez-Adorno H., Posadas-Durán J-P, Sidorov G, Pinto D (2018) Document embeddings learned on various types of n-grams for cross-topic authorship attribution. Computing 100(7):741–756. https://doi.org/10.1007/s00607-018-0587-8

    Article  Google Scholar 

  19. Guan C, Cheng Y, Zhao H (2019) Semantic role labeling with associated memory network. In: Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, vol 1 (long and short papers). Association for computational linguistics, pp 3361–3371. https://doi.org/10.18653/v1/N19-1340

  20. Gui L, Hu J, He Y, Xu R, Lu Q, Du J (2017) A question answering approach to emotion cause extraction. arXiv:1708.05482

  21. Hashimoto K, Miwa M, Tsuruoka Y, Chikayama T (2013) Simple customization of recursive neural networks for semantic relation classification. In: Proceedings of the 2013 conference on empirical methods in natural language processing. Association for computational linguistics, pp 1372–1376. https://aclanthology.org/D13-1137

  22. He L, Lee K, Lewis M, Zettlemoyer L (2017) Deep semantic role labeling: what works and what’s next. In: Proceedings of the 55th annual meeting of the association for computational linguistics(vol 1: long papers). Association for computational linguistics, pp 473–483. https://doi.org/10.18653/v1/P17-1044

  23. Huang C, Trabelsi A, Zaïane O (2019) ANA At SemEval-2019 task 3: contextual emotion detection in conversations through hierarchical LSTMs and BERT. In: Proceedings of the 13th international workshop on semantic evaluation. Association for computational linguistics, pp 49–53. https://doi.org/10.18653/v1/S19-2006

  24. Ishiwatari T, Yasuda Y, Miyazaki T, Goto J (2020) Relation-aware graph attention networks with relational position encodings for emotion recognition in conversations. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP). Association for computational linguistics, pp 7360–7370. https://doi.org/10.18653/v1/2020.emnlp-main.597

  25. Jo Y, Lee L, Palaskar S (2017) Combining LSTM and latent topic modeling for mortality prediction. arXiv:1709.02842

  26. Kalchbrenner N, Grefenstette E, Blunsom P (2014) A convolutional neural network for modelling sentences. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (vol 1: long papers). Association for computational linguistics, pp 655–665. https://doi.org/10.3115/v1/P14-1062

  27. Kaliyar RK, Goswami A, Narang P, Sinha S (2020) FNDNEt – a deep convolutional neural network for fake news detection. Cognit Syst Res 61:32–44. https://doi.org/10.1016/j.cogsys.2019.12.005

    Article  Google Scholar 

  28. Keikha M, Khonsari A, Oroumchian F (2009) Rich document representation and classification: an analysis. Knowl-Based Syst 22(1):67–71. https://doi.org/10.1016/j.knosys.2008.06.002

    Article  Google Scholar 

  29. Kim Y (2014) Convolutional neural networks for sentence classification. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). Association for computational linguistics, pp 1746–1751. https://doi.org/10.3115/v1/D14-1181

  30. Lai S, Liu K, He S, Zhao J (2016) How to generate a good word embedding. IEEE Intell Syst 31(6):5–14. https://doi.org/10.1109/MIS.2016.45

    Article  Google Scholar 

  31. Le QV, Mikolov T (2014) Distributed Representations of Sentences and Documents. arXiv:1405.4053. [cs]

  32. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  33. Lee SYM, Chen Y, Huang C-R (2010) A text-driven rule-based system for emotion cause detection, vol 9

  34. Lee JY, Dernoncourt F, Szolovits P (2017) MIT At SemEval-2017 task10: relation extraction with convolutional neural networks. In: Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017). Association for computational linguistics, pp 978–984. https://doi.org/10.18653/v1/S17-2171

  35. Li J, Ji D, Li F, Zhang M, Liu Y (2020) Hitrans: a transformer-based context- and speaker-sensitive model for emotion detection in conversations. In: Proceedings of the 28th international conference on computational linguistics. International committee on computational linguistics, pp 4190–4200. https://doi.org/10.18653/v1/2020.coling-main.370

  36. Li J, Luong T, Jurafsky D (2015) A hierarchical neural autoencoder for paragraphs and documents. In: Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing(vol 1: long papers). Association for computational linguistics, Beijing, China, pp 1106–1115. https://doi.org/10.3115/v1/P15-1107

  37. Li X, Song K, Feng S, Wang D, Zhang Y (2018) A Co-Attention neural network model for emotion cause analysis with emotional context awareness. In: Proceedings of the 2018 conference on empirical methods in natural language processing. Association for computational linguistics, pp 4752–4757. https://doi.org/10.18653/v1/D18-1506

  38. Li C, Wang J, Wang H, Zhao M, Li W, Deng X (2020) Visual-texual emotion analysis with deep coupled video and danmu neural networks. IEEE Trans Multimed 22(6):1634–1646. https://doi.org/10.1109/TMM.2019.2946477

    Article  Google Scholar 

  39. Liu J, Shang J, Wang C, Ren X, Han J (2015) Mining quality phrases from massive text corpora. In: Proceedings of the 2015 ACM SIGMOD international conference on management of data. SIGMOD ’15. Association for computing machinery, pp 1729–1744. https://doi.org/10.1145/2723372.2751523

  40. Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv:1301.3781. [cs]

  41. Mikolov T, Sutskever I, Chen K, Corrado G, Dean J (2013) Distributed representations of words and phrases and their compositionality. arXiv:1310.4546. [cs, stat]

  42. Minaee S, Kalchbrenner N, Cambria E, Nikzad N, Chenaghlu M, Gao J (2021) Deep learning based text classification: a comprehensive review. arXiv:2004.03705. [cs, stat]

  43. Mohammad S (2012) #Emotional tweets 10

  44. Mundra S, Sen A, Sinha M, Mannarswamy S, Dandapat S, Roy S (2017) Fine-grained emotion detection in contact center chat utterances. In: Kim J, Shim K, Cao L, Lee J-G, Lin X, Moon Y-S (eds) Advances in knowledge discovery and data mining. Springer international publishing, vol 10235, pp 337–349. https://doi.org/10.1007/978-3-319-57529-2_27

  45. Nag PK, Priya RV (2021) Contextual BI -directional attention flow with embeddings from language models: a generative approach to emotion detection. In: Advances in robotics - 5th international conference of the robotics society. AIR 2021. Association for computing machinery, pp 1–6. https://doi.org/10.1145/3478586.3478629

  46. Nguyen TH, Grishman R (2015) Relation extraction: Perspective from convolutional neural networks. In: Proceedings of the 1st workshop on vector space modeling for natural language processing. Association for computational linguistics, pp 39–48. https://doi.org/10.3115/v1/W15-1506

  47. Nonis F, Barbiero P, Cirrincione G, Olivetti EC, Marcolin F, Vezzetti E (2021) Understanding abstraction in deep CNN: an application on facial emotion recognition. In: Esposito A, Faundez-Zanuy M, Morabito FC, Pasero E (eds) Progresses in artificial intelligence and neural systems. Smart innovation, systems and technologies. Springer, pp 281–290. https://doi.org/10.1007/978-981-15-5093-5_26

  48. Oyedotun OK, Demisse G, Shabayek AER, Aouada D, Ottersten B (2017) Facial expression recognition via joint deep learning of rgb-depth map latent representations. In: 2017 IEEE international conference on computer vision workshops (ICCVW), pp 3161–3168. https://doi.org/10.1109/ICCVW.2017.374

  49. Palangi H, Deng L, Shen Y, Gao J, He X, Chen J, Song X, Ward R (2016) Deep sentence embedding using long short-term memory networks: analysis and application to information retrieval. IEEE/ACM Trans Audio Speech Lang Process 24(4):694–707. https://doi.org/10.1109/TASLP.2016.2520371

    Article  Google Scholar 

  50. Palmer M, Gildea D, Kingsbury P (2005) The proposition bank: an annotated corpus of semantic roles. Computat Linguistics 31(1):71–106. https://doi.org/10.1162/0891201053630264

    Article  Google Scholar 

  51. Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. arXiv:1802.05365. [cs]

  52. Plaza-del-Arco FM, Halat S, Padó S, Klinger R (2022) Multi-task learning with sentiment, emotion, and target detection to recognize hate speech and offensive language. arXiv:2109.10255

  53. Posadas Durán J, Gomez Adorno H, Sidorov G, Batyrshin I, Pinto D, Chanona-Hernández L (2017) Application of the distributed document representation in the authorship attribution task for small corpora. Soft Comput, vol 21. https://doi.org/10.1007/s00500-016-2446-x

  54. Riaz M, Girju JR (2014) In-depth exploitation of noun and verb semantics to identify causation in verb-noun pairs. In: Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL). Association for computational linguistics, pp 161–170. https://doi.org/10.3115/v1/W14-4322

  55. Roth M, Lapata M (2016) Neural semantic role labeling with dependency path embeddings. arXiv:1605.07515. [cs]

  56. Ruppenhofer J, Elsworth M, Petruck MRL, Johnson CR, Scheffczyk J (2006) FrameNet II: extended theory and practice. California: international computer science Institue

  57. Santos WR, Santos WR, Paes PP, Ferreira-Silva IA, Santos AP, Vercese N, Machado DRL, De Paula FJA, Donadi EA, Navarro AM, Fernandes APM (2015) Impact of strength trainingon bone mineral density in patients infected with HIV exhibiting lipodystrophy. J Strength Condition Res 29 (12):3466–3471. https://doi.org/10.1519/JSC.0000000000001001

    Article  Google Scholar 

  58. Scherer KR, Wallbott HG (1994) Evidence for universality and cultural variation of differential emotion response patterning. J Pers Soc Psychol 66(2):310–328. https://doi.org/10.1037/0022-3514.66.2.310

    Article  Google Scholar 

  59. Shang J, Liu J, Jiang M, Ren X, Voss C, Han J (2017) Automated phrase mining from massive text corpora. IEEE Trans Knowl Data Eng, vol PP. https://doi.org/10.1109/TKDE.2018.2812203

  60. Singh N, Roy N, Gangopadhyay A (2019) Analyzing the emotions of crowd for improving the emergency response services. Pervasive Mobile Comput 58:101018. https://doi.org/10.1016/j.pmcj.2019.04.009

    Article  Google Scholar 

  61. Socher R, Huval B, Manning CD, Ng AY (2012) Semantic compositionality through recursive matrix-vector spaces. In: Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning. Association for computational linguistics, pp 1201–1211. https://aclanthology.org/D12-1110

  62. Sun M, Huang X, Ji H, Liu Z, Liu Y (eds) (2019) Chinese computational linguistics: 18th china national conference, CCL2019, 18–20 Oct 2019, proceedings. Lecture notes in computer science, vol. 11856. https://doi.org/10.1007/978-3-030-32381-3. Springer international publishing, Kunming

  63. Sun Y, Yen GG, Yi Z (2019) Evolving unsupervised deep neural networks for learning meaningful representations, vol 23. https://doi.org/10.1109/TEVC.2018.2808689

  64. Tai KS, Socher R, Manning CD (2015) Improved semantic representations from tree-structured long short-term memory networks. In: Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing(vol 1: long papers). Association for computational linguistics, pp 1556–1566. https://doi.org/10.3115/v1/P15-1150

  65. Tang H, Ji D, Zhou Q (2020) Joint multi-level attentional model for emotion detection and emotion-cause pair extraction. Neurocomputing 409:329–340. https://doi.org/10.1016/j.neucom.2020.03.105

    Article  Google Scholar 

  66. Turian J, Ratinov L-A, Bengio Y (2010) Word representations: a simple and general method for semi-supervised learning. In: Proceedings of the 48th annual meeting of the association for computational linguistics. Association for computational linguistics, pp 384–394. https://aclanthology.org/P10-1040

  67. Violante MG, Marcolin F, Vezzetti E, Ulrich L, Billia G, Di Grazia L (2019) 3D facial expression recognition for defining users’ inner requirements–an emotional design case study. Appl Sci 9(11):2218. https://doi.org/10.3390/app9112218

    Article  Google Scholar 

  68. Vishnu Priya R (2019) Emotion recognition from geometric fuzzy membership functions. Multimed Tools Appl 78(13):17847–17878. https://doi.org/10.1007/s11042-018-6954-9

    Article  Google Scholar 

  69. Wei C, Luo S, Guo J, Wu Z, Pan L (2017) Discriminative locally document embedding: learning a smooth affine map by approximation of the probabilistic generative structure of subspace. Knowl-Based Syst 121:41–57. https://doi.org/10.1016/j.knosys.2017.01.012

    Article  Google Scholar 

  70. Wieting J, Bansal M, Gimpel K, Livescu K (2015) From paraphrase databaseto compositional paraphrase model and back. Trans Assoc Computat Ling 3:345–358. https://doi.org/10.1162/tacl_a_00143

    Google Scholar 

  71. Winata GI, Madotto A, Lin Z, Shin J, Xu Y, Xu P, Fung P (2019) CAIRE_HKUST at SemEval-2019 task 3: hierarchical attention for dialogue emotion classification. In: Proceedings of the 13th international workshop on semantic evaluation. Association for computational linguistics, pp 142–147. https://doi.org/10.18653/v1/S19-2021

  72. Wong DF, Lu Y, Chao LS (2016) Bilingual recursive neural network based data selection for statistical machine translation. Knowl-Based Syst 108:15–24. https://doi.org/10.1016/j.knosys.2016.05.003

    Article  Google Scholar 

  73. Wu Y, Zhao S, Li W (2020) Phrase2vec: phrase embedding based on parsing. Inf Sci 517:100–127. https://doi.org/10.1016/j.ins.2019.12.031

    Article  Google Scholar 

  74. Xia R, Ding Z (2019) Emotion-cause pair extraction: a new task to emotion analysis in texts. arXiv:1906.01267. [cs]

  75. Xiao J (2019) Figure Eight at SemEval-2019 task 3: ensemble of transfer learning methods for contextual emotion detection. In: Proceedings of the 13th international workshop on semantic evaluation. Association for computational linguistics, pp 220–224. https://doi.org/10.18653/v1/S19-2036

  76. Yang Y, Nie F, Xu D, Luo J, Zhuang Y, Pan Y (2012) A multimedia retrieval framework based on semi-supervised ranking and relevance feedback. IEEE Trans Pattern Anal Mach Intell 34(4):723–742. https://doi.org/10.1109/TPAMI.2011.170

    Article  Google Scholar 

  77. Yu X, Rong W, Zhang Z, Ouyang Y, Xiong Z (2019) Multiple level hierarchical Network-Based clause selection for emotion cause extraction. IEEE Access 7:9071–9079. https://doi.org/10.1109/ACCESS.2018.2890390

    Article  Google Scholar 

  78. Zeng D, Liu K, Lai S, Zhou G, Zhao J (2014) Relation classification via convolutional deep neural network. In: Proceedings of COLING 2014, the 25th international conference on computational linguistics: technical papers. Dublin city university and association for computational linguistics, Dublin, Ireland, pp 2335–2344. https://aclanthology.org/C14-1220

  79. Zhang W, Li Y, Wang S (2019) Learning document representation via topic-enhanced LSTMmodel. Knowl-Based Syst 174:194–204. https://doi.org/10.1016/j.knosys.2019.03.007

    Article  Google Scholar 

  80. Zhang C, Xie L, Aizezi Y, Gu X (2019) User multi-modal emotional intelligence analysis method based on deep learning in social network big data environment. IEEE Access 7:181758–181766. https://doi.org/10.1109/ACCESS.2019.2959831

    Article  Google Scholar 

  81. Zhao R, Mao K (2018) Fuzzy bag-of-words model for document representation. IEEE Trans Fuzzy Syst 26(2):794–804. https://doi.org/10.1109/TFUZZ.2017.2690222

    Article  Google Scholar 

Download references

Acknowledgements

The authors, thanks to all the anonymous reviewers of Multimedia Tools and Applications Journal for their constructive remarks and fruitful suggestions to improve the manuscript.

Author information

Authors and Affiliations

  1. Department of Computer Applications, National Institute of Technology, Tiruchirappalli, Tamil Nadu, India

    Vishnu Priya R.

  2. Department of Mathematics, Bioinformatics and Computer Applications, Maulana Azad National Institute of Technology (MANIT), Link Road No. 3, Bhopal, 462003, Madhya Pradesh, India

    Prashant Kumar Nag

Authors
  1. Vishnu Priya R.
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Prashant Kumar Nag
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Vishnu Priya R..

Ethics declarations

Conflict of Interests

The authors declare that there are no conflicts of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

R., V.P., Nag, P.K. Text-based emotion recognition using contextual phrase embedding model. Multimed Tools Appl 82, 35329–35355 (2023). https://doi.org/10.1007/s11042-023-14524-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14524-9

Keywords