Abstract
Pre-trained Language Models (PLMs) can be applied to downstream tasks with only fine-tuning, eliminating the need to train the model from scratch. In particular, PLMs have been utilised for Sentiment Analysis (SA), a process that detects, analyses, and extracts the polarity of text sentiments. To help researchers comprehensively understand the existing research on PLM-based SA, identify gaps, establish context, acknowledge previous work, and learn from methodologies, we present a literature review on the topic in this paper. Specifically, we brief the motivation of each method, offer a concise overview of these methods, compare their pros, cons, and performance, and identify the challenges for future research.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Adhikari, A., Ram, A., Tang, R., Lin, J.: DocBERT: BERT for document classification. arXiv preprint arXiv:1904.08398 (2019)
Barbieri, F., Anke, L.E., Camacho-Collados, J.: XLM-T: multilingual language models in twitter for sentiment analysis and beyond. In: Proceedings of the 13th Language Resources and Evaluation Conference, pp. 258–266 (2022)
Bensalah, N., et al.: Sentiment analysis in drug reviews based on improved pre-trained word embeddings. In: Ben Ahmed, M., Boudhir, A.A., Santos, D., Dionisio, R., Benaya, N. (eds.) SCA 2022. LNNS, vol. 629, pp. 87–96. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-26852-6_8
Bhardwaj, M., Akhtar, M.S., Ekbal, A., Das, A., Chakraborty, T.: Hostility detection dataset in Hindi (2020). arXiv preprint arXiv:2011.03588
Blitzer, J., Dredze, M., Pereira, F.: Biographies, Bollywood, boom-boxes and blenders: domain adaptation for sentiment classification. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 440–447 (2007)
Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)
Catelli, R., et al.: Cross lingual transfer learning for sentiment analysis of Italian tripadvisor reviews. Expert Syst. Appl. 209, 118246 (2022)
Chakravarthi, B.R., Jose, N., Suryawanshi, S., Sherly, E., McCrae, J.P.: A sentiment analysis dataset for code-mixed Malayalam-English. In: LREC 2020 Workshop Language Resources and Evaluation Conference, p. 177 (2020)
Chakravarthi, B.R., Muralidaran, V.: Findings of the shared task on hope speech detection for equality, diversity, and inclusion. In: Proceedings of the 1st Workshop on Language Technology for Equality, Diversity and Inclusion, pp. 61–72 (2021)
Chakravarthi, B.R., Muralidaran, V., Priyadharshini, R., McCrae, J.P.: Corpus creation for sentiment analysis in code-mixed Tamil-English text. In: LREC 2020 Workshop Language Resources and Evaluation Conference (2020)
Chen, C., Chen, J., Liu, C., Yang, F., Wan, G., Xia, J.: MT-speech at SemEval-2022 task 10: Incorporating data augmentation and auxiliary task with cross-lingual pretrained language model for structured sentiment analysis. In: Proceedings of the 16th International Workshop on Semantic Evaluation, pp. 1329–1335 (2022)
Chen, Y.: Convolutional neural network for sentence classification. Master’s thesis, University of Waterloo (2015)
Chi, Z., et al.: Improving pretrained cross-lingual language models via self-labeled word alignment. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, vol. 1, pp. 3418–3430 (2021)
Conneau, A., et al.: Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440–8451 (2020)
Cui, J., Wang, Z., Ho, S.B., Cambria, E.: Survey on sentiment analysis: evolution of research methods and topics. Artif. Intell. Rev. 1–42 (2023)
Demner-Fushman, D., et al.: A dataset of 200 structured product labels annotated for adverse drug reactions. Sci. Data 5(1), 1–8 (2018)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186 (2019)
Dong, L., Wei, F., Tan, C., Tang, D., Zhou, M., Xu, K.: Adaptive recursive neural network for target-dependent Twitter sentiment classification. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, vol. 2, pp. 49–54 (2014)
Essebbar, A., Kane, B., Guinaudeau, O., Chiesa, V., Quénel, I., Chau, S.: Aspect based sentiment analysis using French pre-trained models. In: Proceedings of the 13th International Conference on Agents and Artificial Intelligence, pp. 519–525 (2021)
Gräßer, F., Kallumadi, S., Malberg, H., Zaunseder, S.: Aspect-based sentiment analysis of drug reviews applying cross-domain and cross-data learning. In: Proceedings of the 2018 International Conference on Digital Health, pp. 121–125 (2018)
Han, X., Wang, L.: A novel document-level relation extraction method based on BERT and entity information. IEEE Access 8, 96912–96919 (2020)
Hande, A., Priyadharshini, R., Chakravarthi, B.R.: KanCMD: Kannada codemixed dataset for sentiment analysis and offensive language detection. In: Proceedings of the 3rd Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media, pp. 54–63 (2020)
Henry, S., Buchan, K., Filannino, M., Stubbs, A., Uzuner, O.: 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. J. Am. Med. Inform. Assoc. 27(1), 3–12 (2020)
Hossain, E., Sharif, O., Hoque, M.M.: NLP-CUET@ LT-EDI-EACL2021: multilingual code-mixed hope speech detection using cross-lingual representation learner. In: Proceedings of the 1st Workshop on Language Technology for Equality, Diversity and Inclusion, pp. 168–174 (2021)
Jafarian, H., Taghavi, A.H., Javaheri, A., Rawassizadeh, R.: Exploiting BERT to improve aspect-based sentiment analysis performance on Persian language. In: Proceedings of the 2021 7th International Conference on Web Research, pp. 5–8 (2021)
Jayanthi, S.M., Gupta, A.: Sj_aj@ dravidianlangtech-eacl2021: task-adaptive pre-training of multilingual BERT models for offensive language identification. In: Proceedings of the 1st Workshop on Speech and Language Technologies for Dravidian Languages, pp. 307–312 (2021)
Jiang, Q., Chen, L., Xu, R., Ao, X., Yang, M.: A challenge dataset and effective models for aspect-based sentiment analysis. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6280–6285 (2019)
Kakwani, D., et al.: IndicNLPSuite: monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4948–4961 (2020)
Kamal, O., Kumar, A., Vaidhya, T.: Hostility detection in Hindi leveraging pre-trained language models. In: Chakraborty, T., Shu, K., Bernard, H.R., Liu, H., Akhtar, M.S. (eds.) CONSTRAINT 2021. CCIS, vol. 1402, pp. 213–223. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-73696-5_20
Ke, C., Xiong, Q., Wu, C., Liao, Z., Yi, H.: Prior-BERT and multi-task learning for target-aspect-sentiment joint detection. In: 2022 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, pp. 7817–7821 (2022)
Khan, L., Amjad, A., Ashraf, N., Chang, H.T.: Multi-class sentiment analysis of Urdu text using multilingual BERT. Sci. Rep. 12(1), 1–17 (2022)
Kong, J., Wang, J., Zhang, X.: Hierarchical BERT with an adaptive fine-tuning strategy for document classification. Knowl.-Based Syst. 238, 107872 (2022)
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: ALBERT: a lite BERT for self-supervised learning of language representations. In: Proceedings of the 8th International Conference on Learning Representations (2020)
Leippold, M.: Sentiment spin: attacking financial sentiment with GPT-3. Technical report. 23-11, Swiss Finance Institute (2023)
Liao, W., Zeng, B., Yin, X., Wei, P.: An improved aspect-category sentiment analysis model for text sentiment analysis based on RoBERTa. Appl. Intell. 51(6), 3522–3533 (2021)
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
Luo, X., Yin, S., Lin, P.: A survey of cross-lingual sentiment analysis based on pre-trained models. In: Proceedings of the 21st International Conference on Electronic Business, pp. 23–33 (2021)
Meena, A., Prabhakar, T.V.: Sentence level sentiment analysis in the presence of conjuncts using linguistic analysis. In: Amati, G., Carpineto, C., Romano, G. (eds.) ECIR 2007. LNCS, vol. 4425, pp. 573–580. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-71496-5_53
Mensouri, D., Azmani, A., Azmani, M.: Combining RoBERTa pre-trained language model and NMF topic modeling technique to learn from customer reviews analysis. Int. J. Intell. Syst. Appl. Eng. 11(1), 39–49 (2023)
Moraes, R., Valiati, J.F., Neto, W.P.G.: Document-level sentiment classification: an empirical comparison between SVM and ANN. Expert Syst. Appl. 40(2), 621–633 (2013)
Pontiki, M., et al.: SemEval-2016 task 5: aspect based sentiment analysis. In: Proceedings of the 10th International Workshop on Semantic Evaluation, pp. 19–30 (2016)
Pontiki, M., Galanis, D., Papageorgiou, H., Manandhar, S., Androutsopoulos, I.: SemEval-2015 task 12: aspect based sentiment analysis. In: Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pp. 486–495 (2015)
Pontiki, M., Galanis, D., Pavlopoulos, J., Papageorgiou, H., Androutsopoulos, I., Manandhar, S.: SemEval-2014 task 4: aspect based sentiment analysis. In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp. 27–35 (2014)
Sabri, N., Edalat, A., Bahrak, B.: Sentiment analysis of Persian-English code-mixed texts. In: Proceedings of 2021 26th International Computer Conference, Computer Society of Iran (CSICC), pp. 1–4 (2021)
Sarangi, P., Ganesan, S., Arora, P., Joshi, S.: AMEX AI labs at SemEval-2022 task 10: contextualized fine-tuning of BERT for structured sentiment analysis. In: Proceedings of the 16th International Workshop on Semantic Evaluation, pp. 1296–1304 (2022)
Sarker, A., Gonzalez, G.: A corpus for mining drug-related knowledge from Twitter chatter: language models and their utilities. Data Brief 10, 122–131 (2017)
Sun, K., Luo, X., Luo, M.Y.: A survey of pretrained language models. In: Memmi, G., Yang, B., Kong, L., Zhang, T., Qiu, M. (eds.) KSEM 2022. LNCS, vol. 13369, pp. 442–456. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-10986-7_36
Sweidan, A.H., El-Bendary, N., Al-Feel, H.: Sentence-level aspect-based sentiment analysis for classifying adverse drug reactions (ADRs) using hybrid ontology-XLNet transfer learning. IEEE Access 9, 90828–90846 (2021)
Van Thin, D., Quoc Ngo, H., Ngoc Hao, D., Luu-Thuy Nguyen, N.: Exploring zero-shot and joint training cross-lingual strategies for aspect-based sentiment analysis based on contextualized multilingual language models. J. Inf. Telecommun. 1–23 (2023)
Venugopalan, M., Gupta, D.: An enhanced guided LDA model augmented with BERT based semantic strength for aspect term extraction in sentiment analysis. Knowl.-Based Syst. 108668 (2022)
Wan, H., Yang, Y., Du, J., Liu, Y., Qi, K., Pan, J.Z.: Target-aspect-sentiment joint detection for aspect-based sentiment analysis. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence, pp. 9122–9129 (2020)
Wang, D., Yang, A., Zhou, Y., Xie, F., Ouyang, Z., Peng, S.: Distillation language adversarial network for cross-lingual sentiment analysis. In: 2022 International Conference on Asian Language Processing (IALP), pp. 45–50 (2022)
Wang, H., Li, J., Wu, H., Hovy, E., Sun, Y.: Pre-trained language models and their applications. Engineering (2022)
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V.: XLNet: generalized autoregressive pretraining for language understanding. In: Advances in Neural Information Processing Systems, vol. 32, pp. 5753–5763 (2019)
Yao, Y., et al.: DocRED: a large-scale document-level relation extraction dataset. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 764–777 (2019)
Yong, Q., Chen, C., Wang, Z., Xiao, R., Tang, H.: SGPT: semantic graphs based pre-training for aspect-based sentiment analysis. World Wide Web 1–14 (2023)
Zhang, K., et al.: Incorporating dynamic semantics into pre-trained language model for aspect-based sentiment analysis. In: Findings of the Association for Computational Linguistics, ACL 2022, pp. 3599–3610 (2022)
Zhou, J., Tian, J., Wang, R., Wu, Y., Xiao, W., He, L.: SentiX: a sentiment-aware pre-trained model for cross-domain sentiment analysis. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 568–579 (2020)
Zhou, X., Zhang, T., Cheng, C., Song, S.: Dynamic multichannel fusion mechanism based on a graph attention network and BERT for aspect-based sentiment classification. Appl. Intell. 1–14 (2022)
Zhou, X., et al.: Attention-based LSTM network for cross-lingual sentiment classification. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 247–256 (2016)
Zou, J., et al.: Aspect-level sentiment classification based on graph attention network with BERT. In: Sun, X., Zhang, X., Xia, Z., Bertino, E. (eds.) ICAIS 2022. CCIS, vol. 1586, pp. 231–244. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-06767-9_19
Acknowledgements
This work was partially supported by a Research Fund of Guangxi Key Lab of Multi-source Information Mining Security (22-A-01-02) and a Graduate Student Innovation Project of School of Computer Science, Engineering, Guangxi Normal University (JXXYYJSCXXM-2021-001) and the Middle-aged and Young Teachers’ Basic Ability Promotion Project of Guangxi (No. 2021KY0067).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yang, B., Luo, X., Sun, K., Luo, M.Y. (2024). Sentiment Analysis Based on Pretrained Language Models: Recent Progress. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1966. Springer, Singapore. https://doi.org/10.1007/978-981-99-8148-9_11
Download citation
DOI: https://doi.org/10.1007/978-981-99-8148-9_11
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8147-2
Online ISBN: 978-981-99-8148-9
eBook Packages: Computer ScienceComputer Science (R0)