Skip to main content

Sentiment Analysis Based on Pretrained Language Models: Recent Progress

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1966))

Included in the following conference series:

  • 799 Accesses

Abstract

Pre-trained Language Models (PLMs) can be applied to downstream tasks with only fine-tuning, eliminating the need to train the model from scratch. In particular, PLMs have been utilised for Sentiment Analysis (SA), a process that detects, analyses, and extracts the polarity of text sentiments. To help researchers comprehensively understand the existing research on PLM-based SA, identify gaps, establish context, acknowledge previous work, and learn from methodologies, we present a literature review on the topic in this paper. Specifically, we brief the motivation of each method, offer a concise overview of these methods, compare their pros, cons, and performance, and identify the challenges for future research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    See Figs. 5 and 6 for the number and citation percentages of the references on various SA topics.

References

  1. Adhikari, A., Ram, A., Tang, R., Lin, J.: DocBERT: BERT for document classification. arXiv preprint arXiv:1904.08398 (2019)

  2. Barbieri, F., Anke, L.E., Camacho-Collados, J.: XLM-T: multilingual language models in twitter for sentiment analysis and beyond. In: Proceedings of the 13th Language Resources and Evaluation Conference, pp. 258–266 (2022)

    Google Scholar 

  3. Bensalah, N., et al.: Sentiment analysis in drug reviews based on improved pre-trained word embeddings. In: Ben Ahmed, M., Boudhir, A.A., Santos, D., Dionisio, R., Benaya, N. (eds.) SCA 2022. LNNS, vol. 629, pp. 87–96. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-26852-6_8

    Chapter  Google Scholar 

  4. Bhardwaj, M., Akhtar, M.S., Ekbal, A., Das, A., Chakraborty, T.: Hostility detection dataset in Hindi (2020). arXiv preprint arXiv:2011.03588

  5. Blitzer, J., Dredze, M., Pereira, F.: Biographies, Bollywood, boom-boxes and blenders: domain adaptation for sentiment classification. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 440–447 (2007)

    Google Scholar 

  6. Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)

    Google Scholar 

  7. Catelli, R., et al.: Cross lingual transfer learning for sentiment analysis of Italian tripadvisor reviews. Expert Syst. Appl. 209, 118246 (2022)

    Article  Google Scholar 

  8. Chakravarthi, B.R., Jose, N., Suryawanshi, S., Sherly, E., McCrae, J.P.: A sentiment analysis dataset for code-mixed Malayalam-English. In: LREC 2020 Workshop Language Resources and Evaluation Conference, p. 177 (2020)

    Google Scholar 

  9. Chakravarthi, B.R., Muralidaran, V.: Findings of the shared task on hope speech detection for equality, diversity, and inclusion. In: Proceedings of the 1st Workshop on Language Technology for Equality, Diversity and Inclusion, pp. 61–72 (2021)

    Google Scholar 

  10. Chakravarthi, B.R., Muralidaran, V., Priyadharshini, R., McCrae, J.P.: Corpus creation for sentiment analysis in code-mixed Tamil-English text. In: LREC 2020 Workshop Language Resources and Evaluation Conference (2020)

    Google Scholar 

  11. Chen, C., Chen, J., Liu, C., Yang, F., Wan, G., Xia, J.: MT-speech at SemEval-2022 task 10: Incorporating data augmentation and auxiliary task with cross-lingual pretrained language model for structured sentiment analysis. In: Proceedings of the 16th International Workshop on Semantic Evaluation, pp. 1329–1335 (2022)

    Google Scholar 

  12. Chen, Y.: Convolutional neural network for sentence classification. Master’s thesis, University of Waterloo (2015)

    Google Scholar 

  13. Chi, Z., et al.: Improving pretrained cross-lingual language models via self-labeled word alignment. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, vol. 1, pp. 3418–3430 (2021)

    Google Scholar 

  14. Conneau, A., et al.: Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440–8451 (2020)

    Google Scholar 

  15. Cui, J., Wang, Z., Ho, S.B., Cambria, E.: Survey on sentiment analysis: evolution of research methods and topics. Artif. Intell. Rev. 1–42 (2023)

    Google Scholar 

  16. Demner-Fushman, D., et al.: A dataset of 200 structured product labels annotated for adverse drug reactions. Sci. Data 5(1), 1–8 (2018)

    Article  Google Scholar 

  17. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186 (2019)

    Google Scholar 

  18. Dong, L., Wei, F., Tan, C., Tang, D., Zhou, M., Xu, K.: Adaptive recursive neural network for target-dependent Twitter sentiment classification. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, vol. 2, pp. 49–54 (2014)

    Google Scholar 

  19. Essebbar, A., Kane, B., Guinaudeau, O., Chiesa, V., Quénel, I., Chau, S.: Aspect based sentiment analysis using French pre-trained models. In: Proceedings of the 13th International Conference on Agents and Artificial Intelligence, pp. 519–525 (2021)

    Google Scholar 

  20. Gräßer, F., Kallumadi, S., Malberg, H., Zaunseder, S.: Aspect-based sentiment analysis of drug reviews applying cross-domain and cross-data learning. In: Proceedings of the 2018 International Conference on Digital Health, pp. 121–125 (2018)

    Google Scholar 

  21. Han, X., Wang, L.: A novel document-level relation extraction method based on BERT and entity information. IEEE Access 8, 96912–96919 (2020)

    Article  Google Scholar 

  22. Hande, A., Priyadharshini, R., Chakravarthi, B.R.: KanCMD: Kannada codemixed dataset for sentiment analysis and offensive language detection. In: Proceedings of the 3rd Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media, pp. 54–63 (2020)

    Google Scholar 

  23. Henry, S., Buchan, K., Filannino, M., Stubbs, A., Uzuner, O.: 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J. Am. Med. Inform. Assoc. 27(1), 3–12 (2020)

    Article  Google Scholar 

  24. Hossain, E., Sharif, O., Hoque, M.M.: NLP-CUET@ LT-EDI-EACL2021: multilingual code-mixed hope speech detection using cross-lingual representation learner. In: Proceedings of the 1st Workshop on Language Technology for Equality, Diversity and Inclusion, pp. 168–174 (2021)

    Google Scholar 

  25. Jafarian, H., Taghavi, A.H., Javaheri, A., Rawassizadeh, R.: Exploiting BERT to improve aspect-based sentiment analysis performance on Persian language. In: Proceedings of the 2021 7th International Conference on Web Research, pp. 5–8 (2021)

    Google Scholar 

  26. Jayanthi, S.M., Gupta, A.: Sj_aj@ dravidianlangtech-eacl2021: task-adaptive pre-training of multilingual BERT models for offensive language identification. In: Proceedings of the 1st Workshop on Speech and Language Technologies for Dravidian Languages, pp. 307–312 (2021)

    Google Scholar 

  27. Jiang, Q., Chen, L., Xu, R., Ao, X., Yang, M.: A challenge dataset and effective models for aspect-based sentiment analysis. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6280–6285 (2019)

    Google Scholar 

  28. Kakwani, D., et al.: IndicNLPSuite: monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4948–4961 (2020)

    Google Scholar 

  29. Kamal, O., Kumar, A., Vaidhya, T.: Hostility detection in Hindi leveraging pre-trained language models. In: Chakraborty, T., Shu, K., Bernard, H.R., Liu, H., Akhtar, M.S. (eds.) CONSTRAINT 2021. CCIS, vol. 1402, pp. 213–223. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-73696-5_20

    Chapter  Google Scholar 

  30. Ke, C., Xiong, Q., Wu, C., Liao, Z., Yi, H.: Prior-BERT and multi-task learning for target-aspect-sentiment joint detection. In: 2022 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, pp. 7817–7821 (2022)

    Google Scholar 

  31. Khan, L., Amjad, A., Ashraf, N., Chang, H.T.: Multi-class sentiment analysis of Urdu text using multilingual BERT. Sci. Rep. 12(1), 1–17 (2022)

    Google Scholar 

  32. Kong, J., Wang, J., Zhang, X.: Hierarchical BERT with an adaptive fine-tuning strategy for document classification. Knowl.-Based Syst. 238, 107872 (2022)

    Article  Google Scholar 

  33. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: ALBERT: a lite BERT for self-supervised learning of language representations. In: Proceedings of the 8th International Conference on Learning Representations (2020)

    Google Scholar 

  34. Leippold, M.: Sentiment spin: attacking financial sentiment with GPT-3. Technical report. 23-11, Swiss Finance Institute (2023)

    Google Scholar 

  35. Liao, W., Zeng, B., Yin, X., Wei, P.: An improved aspect-category sentiment analysis model for text sentiment analysis based on RoBERTa. Appl. Intell. 51(6), 3522–3533 (2021)

    Article  Google Scholar 

  36. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  37. Luo, X., Yin, S., Lin, P.: A survey of cross-lingual sentiment analysis based on pre-trained models. In: Proceedings of the 21st International Conference on Electronic Business, pp. 23–33 (2021)

    Google Scholar 

  38. Meena, A., Prabhakar, T.V.: Sentence level sentiment analysis in the presence of conjuncts using linguistic analysis. In: Amati, G., Carpineto, C., Romano, G. (eds.) ECIR 2007. LNCS, vol. 4425, pp. 573–580. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-71496-5_53

    Chapter  Google Scholar 

  39. Mensouri, D., Azmani, A., Azmani, M.: Combining RoBERTa pre-trained language model and NMF topic modeling technique to learn from customer reviews analysis. Int. J. Intell. Syst. Appl. Eng. 11(1), 39–49 (2023)

    Google Scholar 

  40. Moraes, R., Valiati, J.F., Neto, W.P.G.: Document-level sentiment classification: an empirical comparison between SVM and ANN. Expert Syst. Appl. 40(2), 621–633 (2013)

    Article  Google Scholar 

  41. Pontiki, M., et al.: SemEval-2016 task 5: aspect based sentiment analysis. In: Proceedings of the 10th International Workshop on Semantic Evaluation, pp. 19–30 (2016)

    Google Scholar 

  42. Pontiki, M., Galanis, D., Papageorgiou, H., Manandhar, S., Androutsopoulos, I.: SemEval-2015 task 12: aspect based sentiment analysis. In: Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pp. 486–495 (2015)

    Google Scholar 

  43. Pontiki, M., Galanis, D., Pavlopoulos, J., Papageorgiou, H., Androutsopoulos, I., Manandhar, S.: SemEval-2014 task 4: aspect based sentiment analysis. In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp. 27–35 (2014)

    Google Scholar 

  44. Sabri, N., Edalat, A., Bahrak, B.: Sentiment analysis of Persian-English code-mixed texts. In: Proceedings of 2021 26th International Computer Conference, Computer Society of Iran (CSICC), pp. 1–4 (2021)

    Google Scholar 

  45. Sarangi, P., Ganesan, S., Arora, P., Joshi, S.: AMEX AI labs at SemEval-2022 task 10: contextualized fine-tuning of BERT for structured sentiment analysis. In: Proceedings of the 16th International Workshop on Semantic Evaluation, pp. 1296–1304 (2022)

    Google Scholar 

  46. Sarker, A., Gonzalez, G.: A corpus for mining drug-related knowledge from Twitter chatter: language models and their utilities. Data Brief 10, 122–131 (2017)

    Article  Google Scholar 

  47. Sun, K., Luo, X., Luo, M.Y.: A survey of pretrained language models. In: Memmi, G., Yang, B., Kong, L., Zhang, T., Qiu, M. (eds.) KSEM 2022. LNCS, vol. 13369, pp. 442–456. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-10986-7_36

    Chapter  Google Scholar 

  48. Sweidan, A.H., El-Bendary, N., Al-Feel, H.: Sentence-level aspect-based sentiment analysis for classifying adverse drug reactions (ADRs) using hybrid ontology-XLNet transfer learning. IEEE Access 9, 90828–90846 (2021)

    Article  Google Scholar 

  49. Van Thin, D., Quoc Ngo, H., Ngoc Hao, D., Luu-Thuy Nguyen, N.: Exploring zero-shot and joint training cross-lingual strategies for aspect-based sentiment analysis based on contextualized multilingual language models. J. Inf. Telecommun. 1–23 (2023)

    Google Scholar 

  50. Venugopalan, M., Gupta, D.: An enhanced guided LDA model augmented with BERT based semantic strength for aspect term extraction in sentiment analysis. Knowl.-Based Syst. 108668 (2022)

    Google Scholar 

  51. Wan, H., Yang, Y., Du, J., Liu, Y., Qi, K., Pan, J.Z.: Target-aspect-sentiment joint detection for aspect-based sentiment analysis. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence, pp. 9122–9129 (2020)

    Google Scholar 

  52. Wang, D., Yang, A., Zhou, Y., Xie, F., Ouyang, Z., Peng, S.: Distillation language adversarial network for cross-lingual sentiment analysis. In: 2022 International Conference on Asian Language Processing (IALP), pp. 45–50 (2022)

    Google Scholar 

  53. Wang, H., Li, J., Wu, H., Hovy, E., Sun, Y.: Pre-trained language models and their applications. Engineering (2022)

    Google Scholar 

  54. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. In: Advances in Neural Information Processing Systems, vol. 32, pp. 5753–5763 (2019)

    Google Scholar 

  55. Yao, Y., et al.: DocRED: a large-scale document-level relation extraction dataset. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 764–777 (2019)

    Google Scholar 

  56. Yong, Q., Chen, C., Wang, Z., Xiao, R., Tang, H.: SGPT: semantic graphs based pre-training for aspect-based sentiment analysis. World Wide Web 1–14 (2023)

    Google Scholar 

  57. Zhang, K., et al.: Incorporating dynamic semantics into pre-trained language model for aspect-based sentiment analysis. In: Findings of the Association for Computational Linguistics, ACL 2022, pp. 3599–3610 (2022)

    Google Scholar 

  58. Zhou, J., Tian, J., Wang, R., Wu, Y., Xiao, W., He, L.: SentiX: a sentiment-aware pre-trained model for cross-domain sentiment analysis. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 568–579 (2020)

    Google Scholar 

  59. Zhou, X., Zhang, T., Cheng, C., Song, S.: Dynamic multichannel fusion mechanism based on a graph attention network and BERT for aspect-based sentiment classification. Appl. Intell. 1–14 (2022)

    Google Scholar 

  60. Zhou, X., et al.: Attention-based LSTM network for cross-lingual sentiment classification. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 247–256 (2016)

    Google Scholar 

  61. Zou, J., et al.: Aspect-level sentiment classification based on graph attention network with BERT. In: Sun, X., Zhang, X., Xia, Z., Bertino, E. (eds.) ICAIS 2022. CCIS, vol. 1586, pp. 231–244. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-06767-9_19

    Chapter  Google Scholar 

Download references

Acknowledgements

This work was partially supported by a Research Fund of Guangxi Key Lab of Multi-source Information Mining Security (22-A-01-02) and a Graduate Student Innovation Project of School of Computer Science, Engineering, Guangxi Normal University (JXXYYJSCXXM-2021-001) and the Middle-aged and Young Teachers’ Basic Ability Promotion Project of Guangxi (No. 2021KY0067).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xudong Luo .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yang, B., Luo, X., Sun, K., Luo, M.Y. (2024). Sentiment Analysis Based on Pretrained Language Models: Recent Progress. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1966. Springer, Singapore. https://doi.org/10.1007/978-981-99-8148-9_11

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8148-9_11

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8147-2

  • Online ISBN: 978-981-99-8148-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics