Skip to main content
SpringerLink
Log in

An improved aspect-category sentiment analysis model for text sentiment analysis based on RoBERTa

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The aspect-category sentiment analysis can provide more and deeper information than the document-level sentiment analysis, because it aims to predict the sentiment polarities of different aspect categories in the same text. The main challenge of aspect-category sentiment analysis is that different aspect categories may present different polarities in the same text. Previous studies combine the Long Short-Term Memory (LSTM) and attention mechanism to predict the sentiment polarity of the given aspect category, but the LSTM-based methods are not really bidirectional text feature extraction methods. In this paper, we propose a multi-task aspect-category sentiment analysis model based on RoBERTa (Robustly Optimized BERT Pre-training Approach). Treating each aspect category as a subtask, we employ the RoBERTa based on deep bidirectional Transformer to extract features from both text and aspect tokens, and apply the cross-attention mechanism to guide the model to focus on the features most relevant to the given aspect category. According to the experimental results, the proposed model outperforms other models for comparison in aspect-category sentiment analysis. Furthermore, the influencing factors of our proposed model are also analyzed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Xu N, Mao W, Chen G (2018) A co-memory network for multimodal sentiment analysis. In: The 41st international ACM SIGIR conference on research & development in information retrieval, pp 929–932

  2. Hussein D M E D M (2018) A survey on sentiment analysis challenges. J King Saud Univ-Eng Sci 30(4):330–338

    Google Scholar 

  3. Pang B, Lee L (2008) Opinion mining and sentiment analysis. Found Trends Inf Retr 2 (1):1–135

    Article  Google Scholar 

  4. Xue W, Li T (2018) Aspect based sentiment analysis with gated convolutional networks. In: Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers). Association for Computational Linguistics, Melbourne, pp 2514–2523

  5. Sailunaz K, Alhajj R (2019) Emotion and sentiment analysis from twitter text. J Comput Sci 36:101003

    Article  Google Scholar 

  6. Luo L -X (2019) Network text sentiment analysis method combining lda text representation and gru-cnn. Pers Ubiquitous Comput 23(3-4):405–412

    Article  Google Scholar 

  7. Wang Y, Huang M, Zhu X, Zhao L (2016) Attention-based lstm for aspect-level sentiment classification. In: Proceedings of the 2016 conference on empirical methods in natural language processing, pp 606–615

  8. Liu J, Zhang Y (2017) Attention modeling for targeted sentiment. In: Proceedings of the 15th conference of the European chapter of the association for computational linguistics: volume 2, short papers, pp 572–577

  9. Wang S I, Manning CD (2012) bigrams Baselines Simple, good sentiment and topic classification. In: Proceedings of the 50th annual meeting of the association for computational linguistics (volume 2: short papers), pp 90–94

  10. Fu X, Liu G, Guo Y, Wang Z (2013) Multi-aspect sentiment analysis for chinese online social reviews based on topic modeling and hownet lexicon. Knowl-Based Syst 37:186–195

    Article  Google Scholar 

  11. Schouten K, Van Der Weijde O, Frasincar F, Dekker R (2017) Supervised and unsupervised aspect category detection for sentiment analysis with co-occurrence data. IEEE Trans Cybern 48(4):1263–1275

    Article  Google Scholar 

  12. Tang F, Fu L, Yao B, Xu W (2019) Aspect based fine-grained sentiment analysis for online reviews. Inf Sci 488:190–204

    Article  Google Scholar 

  13. Kiritchenko S, Zhu X, Cherry C, Mohammad S (2014) Nrc-canada-2014: detecting aspects and sentiment in customer reviews. In: Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014), pp 437–442

  14. Afzaal M, Usman M, Fong A (2019) Tourism mobile app with aspect-based sentiment classification framework for tourist reviews. IEEE Trans Consum Electron 65(2):233–242

    Article  Google Scholar 

  15. Mikolov T, Sutskever I, Chen K, Corrado G S, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp 3111–3119

  16. Yao H, Liu H, Zhang P (2018) A novel sentence similarity model with word embedding based on convolutional neural network. Concurr Comput: Pract Exp 30(23):e4415

    Article  Google Scholar 

  17. Esposito M, Damiano E, Minutolo A, De Pietro G, Fujita H (2020) Hybrid query expansion using lexical resources and word embeddings for sentence retrieval in question answering. Inf Sci 514:88–105

    Article  Google Scholar 

  18. Kim Y (2014) Convolutional neural networks for sentence classification. In: Empirical methods in natural language processing, pp 1746–1751

  19. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  20. Liu N, Shen B (2020) Aspect-based sentiment analysis with gated alternate neural network. Knowl-Based Syst 188:105010

    Article  Google Scholar 

  21. Song M, Park H, Shin K -S (2019) Attention-based long short-term memory network using sentiment lexicon embedding for aspect-level sentiment analysis in Korean. Inf Process Manag 56(3):637–653

    Article  Google Scholar 

  22. Al-Smadi M, Talafha B, Al-Ayyoub M, Jararweh Y (2019) Using long short-term memory deep neural networks for aspect-based sentiment analysis of arabic reviews. Int J Mach Learn Cybern 10 (8):2163–2175

    Article  Google Scholar 

  23. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Łukasz K, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems , pp 5998–6008

  24. Devlin J, Chang M, Lee K, Toutanova K (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: North American chapter of the association for computational linguistics, pp 4171–4186

  25. Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: a robustly optimized bert pretraining approach. arXiv: Computation and language

  26. Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R (2019) Albert: a lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942

  27. Rao D, Ravichandran D (2009) Semi-supervised polarity lexicon induction. In: Proceedings of the 12th conference of the European chapter of the ACL (EACL 2009), pp 675–682

  28. Vo D T, Zhang Y (2015) Target-dependent twitter sentiment classification with rich automatic features. In: Twenty-fourth international joint conference on artificial intelligence , pp 1347–1353

  29. Ge B, Li F -F, Guo S -L, Tang D -Q (2010) Word’s semantic similarity computation method based on hownet. Jisuanji Yingyong Yanjiu 27(9):3329–3333

    Google Scholar 

  30. Miller G A (1995) Wordnet: a lexical database for english. Commun ACM 38(11):39–41

    Article  Google Scholar 

  31. Wang Y, Wang M, Fujita H (2020) Word sense disambiguation: a comprehensive knowledge exploitation framework. Knowl-Based Syst 190:105030

    Article  Google Scholar 

  32. El Hannach H, Benkhalifa M (2018) Wordnet based implicit aspect sentiment analysis for crime identification from twitter. Int J Adv Comput Sci Appl 9(12):150–159

    Google Scholar 

  33. Tang D, Qin B, Liu T (2016) Aspect level sentiment classification with deep memory network. In: Empirical methods in natural language processing, pp 214–224

  34. Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

  35. He R, Lee W S, Ng H T, Dahlmeier D (2018) Effective attention modeling for aspect-level sentiment classification. In: Proceedings of the 27th international conference on computational linguistics, pp 1121–1131

  36. Fan F, Feng Y, Zhao D (2018) Multi-grained attention network for aspect-level sentiment classification. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 3433–3442

  37. Pota M, Esposito M, De Pietro G, Fujita H (2020) Best practices of convolutional neural networks for question classification. Appl Sci 10(14):4710

    Article  Google Scholar 

  38. Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. In: North American chapter of the association for computational linguistics, vol 1, pp 2227–2237

  39. Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-training (2018). https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf

  40. Sun C, Huang L, Qiu X (2019) Utilizing bert for aspect-based sentiment analysis via constructing auxiliary sentence. In: North American chapter of the association for computational linguistics, pp 380–385

  41. Xu H, Liu B, Shu L, Yu P (2019) BERT post-training for review reading comprehension and aspect-based sentiment analysis. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, vol 1 (long and short papers). Association for Computational Linguistics, Minneapolis, pp 2324–2335

  42. Gao Z, Feng A, Song X, Wu X (2019) Target-dependent sentiment classification with bert. IEEE Access 7:154290–154299

    Article  Google Scholar 

  43. Hoang M, Bihorac OA, Rouces J (2019) Aspect-based sentiment analysis using bert. In: NEAL proceedings of the 22nd Nordic conference on computional linguistics (NoDaLiDa), September 30–October 2, Turku, Finland, number 167. Linköping University Electronic Press, pp 187–196

  44. Cui Y, Chen Z, Wei S, Wang S, Liu T, Hu G (2017) Attention-over-attention neural networks for reading comprehension. In: Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers). Association for Computational Linguistics, Vancouver, pp 593–602

  45. Pennington J, Socher R, Manning C D (2014) Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543

  46. Yao L, Mao C, Luo Y (2019) Graph convolutional networks for text classification. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 7370–7377

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 61701122), the Natural Science Foundation of Guangdong Province, China (2017A030313431, 2018A030310540), the Science and Technology Program of Guangzhou, China (No. 201804010238), and the Science and Technology Plan Project of Guangdong Province, China (No.2016B010108004).

Author information

Authors and Affiliations

  1. School of Computers, Guangdong University of Technology, Guangzhou, 510006, China

    Wenxiong Liao, Bi Zeng & Pengfei Wei

  2. School of Automation, Guangdong University of Technology, Guangzhou, 510006, China

    Xiuwen Yin

Authors
  1. Wenxiong Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bi Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiuwen Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pengfei Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiuwen Yin.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liao, W., Zeng, B., Yin, X. et al. An improved aspect-category sentiment analysis model for text sentiment analysis based on RoBERTa. Appl Intell 51, 3522–3533 (2021). https://doi.org/10.1007/s10489-020-01964-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01964-1

Keywords

Search

Navigation