Skip to main content
SpringerLink
Log in

CLARIN-Emo: Training Emotion Recognition Models Using Human Annotation and ChatGPT

  • Conference paper
  • First Online:
Computational Science – ICCS 2023 (ICCS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14073))

Included in the following conference series:

Abstract

In this paper, we investigate whether it is possible to automatically annotate texts with ChatGPT or generate both artificial texts and annotations for them. We prepared three collections of texts annotated with emotions at the level of sentences and/or whole documents. CLARIN-Emo contains the opinions of real people, manually annotated by six linguists. Stockbrief-GPT consists of real human articles annotated by ChatGPT. ChatGPT-Emo is an artificial corpus created and annotated entirely by ChatGPT. We present an analysis of these corpora and the results of Transformer-based methods fine-tuned on these data. The results show that manual annotation can provide better-quality data, especially in building personalized models.

This work was financed by the European Regional Development Fund as a part of the 2014–2020 Smart Growth Operational Programme: (1) POIR.04.02.00-00C002/19 (AN,ŁR); (2) POIR.01.01.01-00-0615/21 (BK); (3) POIR.01.01.01-00-0288/22 (JK); (4) the statutory funds of the Department of Artificial Intelligence, Wroclaw University of Science and Technology; (5) the European Union under the Horizon Europe, grant no. 101086321 (OMINO).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Acheampong, F.A., Wenyu, C., Nunoo-Mensah, H.: Text-based emotion detection: advances, challenges, and opportunities. Eng. Rep. 2(7), e12189 (2020)

    Google Scholar 

  2. Antaki, F., Touma, S., Milad, D., El-Khoury, J., Duval, R.: Evaluating the performance of ChatGPT in ophthalmology: an analysis of its successes and shortcomings. medRxiv (2023)

    Google Scholar 

  3. Baran, J., Kocoń, J.: Linguistic knowledge application to neuro-symbolic transformers in sentiment analysis. In: 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 395–402. IEEE (2022)

    Google Scholar 

  4. Bravo-Marquez, F., Frank, E., Mohammad, S.M., Pfahringer, B.: Determining word-emotion associations from tweets by multi-label classification. In: 2016 IEEE/WIC/ACM International Conference on Web Intelligence (WI), pp. 536–539. IEEE (2016)

    Google Scholar 

  5. Chen, S.Y., Hsu, C.C., Kuo, C.C., Ku, L.W., et al.: EmotionLines: an emotion corpus of multi-party conversations. arXiv preprint arXiv:1802.08379 (2018)

  6. Chen, Y., Eger, S.: Transformers go for the LOLs: generating (humourous) titles from scientific abstracts end-to-end (2022)

    Google Scholar 

  7. Cohan, A., Beltagy, I., King, D., Dalvi, B., Weld, D.: Pretrained language models for sequential sentence classification. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3693–3699. Association for Computational Linguistics (2019)

    Google Scholar 

  8. Conneau, A., et al.: Unsupervised cross-lingual representation learning at scale. CoRR abs/1911.02116 (2019)

    Google Scholar 

  9. Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., Ravi, S.: GoEmotions: a dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547 (2020)

  10. Desai, S., Caragea, C., Li, J.J.: Detecting perceived emotions in hurricane disasters. arXiv preprint arXiv:2004.14299 (2020)

  11. Ekman, P.: Are there basic emotions? Psychol. Rev. 99(3), 550–553 (1992)

    Article  Google Scholar 

  12. Gao, C.A., et al.: Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv (2022)

    Google Scholar 

  13. Kazienko, P., et al.: Human-centred neural reasoning for subjective content processing: hate speech, emotions, and humor. Inf. Fusion (2023)

    Google Scholar 

  14. Kim, Y., Jeong, S.R., Ghani, I.: Text opinion mining to analyze news for stock market prediction. Int. J. Adv. Soft Comput. Appl. 6(1), 2074–8523 (2014)

    Google Scholar 

  15. Kocoń, J., et al.: Neuro-symbolic models for sentiment analysis. In: Groen, D., de Mulatier, C., Paszynski, M., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M.A. (eds.) ICCS 2022. LNCS, pp. 667–681. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-08754-7_69

    Chapter  Google Scholar 

  16. Kocoń, J., et al.: ChatGPT: jack of all trades, master of none. arXiv preprint arXiv:2302.10724 (2023)

  17. Kocoń, J., et al.: Learning personal human biases and representations for subjective tasks in natural language processing. In: 2021 IEEE International Conference on Data Mining (ICDM), pp. 1168–1173. IEEE (2021)

    Google Scholar 

  18. Kocoń, J., Miłkowski, P., Kanclerz, K.: MultiEmo: multilingual, multilevel, multidomain sentiment analysis corpus of consumer reviews. In: Paszynski, M., Kranzlmüller, D., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M.A. (eds.) ICCS 2021. LNCS, vol. 12743, pp. 297–312. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77964-1_24

    Chapter  Google Scholar 

  19. Kocoń, J., Miłkowski, P., Zaśko-Zielińska, M.: Multi-level sentiment analysis of PolEmo 2.0: extended corpus of multi-domain consumer reviews. In: Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 980–991 (2019)

    Google Scholar 

  20. Korczyński, W., Kocoń, J.: Compression methods for transformers in multidomain sentiment analysis. In: 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 419–426. IEEE (2022)

    Google Scholar 

  21. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach (2019)

    Google Scholar 

  22. Maynard, D.G., Greenwood, M.A.: Who cares about sarcastic tweets? Investigating the impact of sarcasm on sentiment analysis. In: LREC 2014 Proceedings. ELRA (2014)

    Google Scholar 

  23. Miłkowski, P., Gruza, M., Kazienko, P., Szołomicka, J., Woźniak, S., Kocoń, J.: Multi-model analysis of language-agnostic sentiment classification on multiemo data. In: Nguyen, N.T., Manolopoulos, Y., Chbeir, R., Kozierkiewicz, A., Trawiński, B. (eds.) ICCCI 2022. LNCS, pp. 163–175. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16014-1_14

    Chapter  Google Scholar 

  24. Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., Gao, J.: Deep learning based text classification: a comprehensive review (2020)

    Google Scholar 

  25. Mirzaee, H., Peymanfard, J., Moshtaghin, H.H., Zeinali, H.: ArmanEmo: a Persian dataset for text-based emotion detection. arXiv preprint arXiv:2207.11808 (2022)

  26. Ngo, A., Candri, A., Ferdinan, T., Kocoń, J., Korczynski, W.: StudEmo: a non-aggregated review dataset for personalized emotion recognition. In: Proceedings of the 1st Workshop on Perspectivist Approaches to NLP@ LREC2022, pp. 46–55 (2022)

    Google Scholar 

  27. Paul, M.J., et al.: Social media mining for public health monitoring and surveillance. In: Biocomputing 2016: Proceedings of the Pacific Symposium, pp. 468–479. World Scientific (2016)

    Google Scholar 

  28. Păvăloaia, V.D., Teodor, E.M., Fotache, D., Danileţ, M.: Opinion mining on social media data: sentiment analysis of user preferences. Sustainability 11(16), 4459 (2019)

    Article  Google Scholar 

  29. Phillips, T., Saleh, A., Glazewski, K.D., Hmelo-Silver, C.E., Mott, B., Lester, J.C.: Exploring the use of GPT-3 as a tool for evaluating text-based collaborative discourse. In: Companion Proceedings of the 12th, p. 54 (2022)

    Google Scholar 

  30. Plutchik, R.: A general psychoevolutionary theory of emotion. In: Theories of Emotion, pp. 3–33. Elsevier (1980)

    Google Scholar 

  31. Rashkin, H., Bosselut, A., Sap, M., Knight, K., Choi, Y.: Modeling Naive psychology of characters in simple commonsense stories. arXiv preprint arXiv:1805.06533 (2018)

  32. Reimers, N., Gurevych, I.: Making monolingual sentence embeddings multilingual using knowledge distillation. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (2020)

    Google Scholar 

  33. Rotsztejn, J., Hollenstein, N., Zhang, C.: ETH-DS3Lab at SemEval-2018 task 7: effectively combining recurrent and convolutional neural networks for relation classification and extraction. In: Proceedings of the 12th International Workshop on Semantic Evaluation, New Orleans, Louisiana, pp. 689–696. Association for Computational Linguistics (2018)

    Google Scholar 

  34. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter (2019)

    Google Scholar 

  35. Sprugnoli, R., et al.: Multiemotions-it: a new dataset for opinion polarity and emotion analysis for Italian. In: Proceedings of the Seventh Italian Conference on Computational Linguistics (CLiC-it 2020), pp. 402–408. Accademia University Press (2020)

    Google Scholar 

  36. Susnjak, T.: ChatGPT: the end of online exam integrity? arXiv preprint arXiv:2212.09292 (2022)

  37. Szołomicka, J., Kocon, J.: MultiAspectEmo: multilingual and language-agnostic aspect-based sentiment analysis. In: 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 443–450. IEEE (2022)

    Google Scholar 

  38. Tabone, W., de Winter, J.: Using ChatGPT for human-computer interaction research: a primer. Manuscript Submitted for Publication (2023)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Artificial Intelligence, Wrocław University of Science and Technology, Wrocław, Poland

    Bartłomiej Koptyra, Anh Ngo, Łukasz Radliński & Jan Kocoń

Authors
  1. Bartłomiej Koptyra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anh Ngo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Łukasz Radliński
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jan Kocoń
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bartłomiej Koptyra .

Editor information

Editors and Affiliations

  1. Czech Technical University in Prague, Prague, Czech Republic

    Jiří Mikyška

  2. University of Amsterdam, Amsterdam, The Netherlands

    Clélia de Mulatier

  3. AGH University of Science and Technology, Krakow, Poland

    Maciej Paszynski

  4. University of Amsterdam, Amsterdam, The Netherlands

    Valeria V. Krzhizhanovskaya

  5. University of Tennessee at Knoxville, Knoxville, TN, USA

    Jack J. Dongarra

  6. University of Amsterdam, Amsterdam, The Netherlands

    Peter M.A. Sloot

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Koptyra, B., Ngo, A., Radliński, Ł., Kocoń, J. (2023). CLARIN-Emo: Training Emotion Recognition Models Using Human Annotation and ChatGPT. In: Mikyška, J., de Mulatier, C., Paszynski, M., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds) Computational Science – ICCS 2023. ICCS 2023. Lecture Notes in Computer Science, vol 14073. Springer, Cham. https://doi.org/10.1007/978-3-031-35995-8_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-35995-8_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-35994-1

  • Online ISBN: 978-3-031-35995-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation