Skip to main content
SpringerLink
Log in

SA-ASBA: a hybrid model for aspect-based sentiment analysis using synthetic attention in pre-trained language BERT model with extreme gradient boosting

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Aspect-based sentiment analysis (ABSA) is a granular-level sentiment analysis task that aims to detect the sentiment polarities of a specified aspect in the text. This research shows excessive curiosity in modelling target and context through attention networks to attain effective feature representations for sentiment detection works. We have proposed a synthetic attention in bidirectional encoder representations from transformers (SA-BERT) with an extreme gradient boosting (XGBoost) classifier to classify sentiment polarity in the review dataset. The proposed model generates dynamic word vector encoding of the aspect and corresponding context of the reviews. Then, the aspect and context of the reviews are meaningfully represented by a transformer that can input the vector word in parallel. After that, the model uses the synthetic attention mechanism to learn essential parts of context and aspects in reviews. Finally, the model places overall representation in the sentiment classification layer to predict sentiment polarity. Both proposed SA-BERT and SA-BERT-XGBoost models achieved the highest accuracy (92.02 and 93.71%) on the restaurant16 and highest F-1 scores (81.19 and 81.64%) on the restaurant14 dataset, respectively. The average accuracy and F1 scores are approximately 2 and 3.04% higher than the baseline models (DLCF-DCA-CDM, R-GAT+BERT, ASGCN-DG, AEN-BERT and BERT-PT). Therefore, proposed models outperform in comparison with baseline models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Data availability

All data generated or analysed during this study are included in the published articles [13, 57] and [58].

References

  1. Pang B, Lee L (2008) Opinion mining and sentiment analysis. Found Trends Inf Retr 2:1–2

    Article  Google Scholar 

  2. Guixian X, Zhang Z, Zhang T, Shaona Yu, Meng Y, Chen S (2022) Aspect-level sentiment classification based on attention-BiLSTM model and transfer learning. Know-Based Syst 245:108586

    Article  Google Scholar 

  3. Mayur W, Sekhara RAC, Chaitanya K (2022) A survey on sentiment analysis methods, applications, and challenges. Artif Intell Rev 7:1–50

    Google Scholar 

  4. Schouten K, Frasincar F (2015) Survey on aspect-level sentiment analysis. IEEE Trans Know Data Eng 28(3):813–830

    Article  Google Scholar 

  5. Dou Z-Y (2017) Capturing user and product information for document level sentiment analysis with deep memory network. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 521–526

  6. Chakraborty K, Bhattacharyya S, Bag R (2020) A survey of sentiment analysis from social media data. IEEE Trans Comput Soc Syst 7(2):450–464

    Article  Google Scholar 

  7. Sun J, Han P, Cheng Z, Enming W, Wang W (2020) Transformer based multi-grained attention network for aspect-based sentiment analysis. IEEE Access 8:211152–211163

    Article  Google Scholar 

  8. Ayetiran EF, Eniafe Festus Ayetiran (2022) Attention-based aspect sentiment classification using enhanced learning through CNN-BiLSTM networks. Know-Based Syst 252:109409

    Article  Google Scholar 

  9. Zhao P, Hou L, Ou W (2020) Modeling sentiment dependencies with graph convolutional networks for aspect-level sentiment classification. Know-Based Syst 193:105443

    Article  Google Scholar 

  10. Xiao Z, Xin X, Xing H, Song F, Wang X, Zhao B (2021) A federated learning system with enhanced feature extraction for human activity recognition. Know-Based Syst 229:107338

    Article  Google Scholar 

  11. Guangtao X, Liu P, Zhu Z, Liu J, Fuyong X (2021) Attention-enhanced graph convolutional networks for aspect-based sentiment classification with multi-head attention. Appl Sci 11(8):3640

    Article  Google Scholar 

  12. Anand D, Naorem D (2016) Semi-supervised aspect based sentiment analysis for movies using review filtering. Procedia Comput Sci 84:86–93

    Article  Google Scholar 

  13. Kiritchenko S, Zhu X, Cherry C, Mohammad SM (2014) Detecting aspects and sentiment in customer reviews. In: 8th International Workshop on Semantic Evaluation (SemEval), pp 437–442

  14. Poria S, Ofek N, Gelbukh A, Hussain A, Rokach L(2014) Dependency tree-based rules for concept-level aspect-based sentiment analysis. In: Semantic Web Evaluation Challenge, 41–47. Springer: Cham

  15. Weichselbraun A, Gindl S, Scharl A (2013) Extracting and grounding contextualized sentiment lexicons. IEEE Intell Syst 28(2):39–46

    Article  Google Scholar 

  16. Gao Z, Li Z, Luo J, Li X (2022) Short text aspect-based sentiment analysis based on CNN+ BiGRU. Appl Sci 12(5):2707

    Article  Google Scholar 

  17. Wang X, Li F, Zhang Z, Guangluan X, Zhang J, Sun X (2021) A unified position-aware convolutional neural network for aspect based sentiment analysis. Neurocomputing 450:91–103

    Article  Google Scholar 

  18. Bai Q, Zhou J, He L (2022) PG-RNN: using position-gated recurrent neural networks for aspect-based sentiment classification. J Supercomput 78(3):4073–4094

    Article  Google Scholar 

  19. Zhang B, Li X, Xiaofei X, Leung K-C, Chen Z, Ye Y (2020) Knowledge guided capsule attention network for aspect-based sentiment analysis. IEEE/ACM Trans Audio, Speech, Lang Process 28:2538–2551

    Article  Google Scholar 

  20. Lin Y, Wang C, Song H, Li Y (2021) Multi-head self-attention transformation networks for aspect-based sentiment analysis. IEEE Access 9:8762–8770

    Article  Google Scholar 

  21. Liang B, Hang S, Gui L, Cambria E, Ruifeng X (2022) Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional networks. Knowl-Based Syst 235:107643

    Article  Google Scholar 

  22. Majumder N, Poria S, Gelbukh A, Akhtar MS, Cambria E, Ekbal A(2018) Iarm: Inter-aspect relation modeling with memory networks in aspect-based sentiment analysis. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 3402–3411

  23. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30

  24. Xing H, Xiao Z, Zhan D, Luo S, Dai P, Li K (2022) Selfmatch: Robust semisupervised time-series classification with self-distillation. Int J Intell Syst

  25. Zhang P, Huang X, Li M, Xue Yu (2021) Hybridization between neural computing and nature-inspired algorithms for a sentence similarity model based on the attention mechanism. ACM Trans Asian Low-Resour Lang Inf Process(TALLIP) 20(1):1–21

    Article  Google Scholar 

  26. Yadav RK, Jiao L, Goodwin M, Granmo O-C (2021) Positionless aspect based sentiment analysis using attention mechanism. Know-Based Syst 226:107136

    Article  Google Scholar 

  27. Huang B, Ou Y, Carley KM (2018) Aspect level sentiment classification with attention-over-attention neural networks. In: International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation. pp 197–206. Springer: Cham

  28. Xiao Z, Xin X, Xing H, Luo S, Dai P, Zhan D (2021) RTFN: a robust temporal feature network for time series classification. Inf Sci 571:65–86

    Article  MathSciNet  Google Scholar 

  29. Wang Y, Huang M, Zhu X, Zhao L(2016) Attention-based LSTM for aspect-level sentiment classification. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pP 606–615

  30. Tang D, Qin B, Liu T(2016) Aspect level sentiment classification with deep memory network. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp 214–224, Austin, Texas, November 2016a. Association for Computational Linguistics. https://doi.org/10.18653/v1/D16-1021

  31. Ma D, Li S, Zhang X, Wang H(2017) Interactive attention networks for aspect-level sentiment classification. In: IJCAI’17: Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI’17, pp 4068-4074. AAAI Press,. ISBN 9780999241103

  32. Bensoltane R, Zaki T (2022) Towards Arabic aspect-based sentiment analysis: a transfer learning-based approach. Soc Netw Anal Min 12(1):1–16

    Article  Google Scholar 

  33. Cai X, Cao H, Ma J, Li M, Zhuang X(2021) Aspect level sentiment classification with semantic distance attention networks. In: 2021 2nd International Conference on Computing, Networks and Internet of Things, pp 1–5

  34. Al-Smadi M, Talafha B, Al-Ayyoub M, Jararweh Y (2019) Using long short-term memory deep neural networks for aspect-based sentiment analysis of Arabic reviews. Int J Mach Learn Cybern 10(8):2163–2175

    Article  Google Scholar 

  35. Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. CoRR, abs/1802.05365

  36. Alec R, Karthik N, Tim S, Ilya S (2018) Improving language understanding by generative pre-training. The university of british columbia vancouver campus, Vancouver

    Google Scholar 

  37. Ullah H, Ahmad B, Sana I, Sattar A, Khan A, Akbar S, Asghar MZ (2021) Comparative study for machine learning classifier recommendation to predict political affiliation based on online reviews. CAAI Trans Intell Technol 6(3):251–264

    Article  Google Scholar 

  38. Karimi A, Rossi L, Prati A(2021) Adversarial training for aspect-based sentiment analysis with bert. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp 8797–8803. IEEE

  39. Song Y, Wang J, Jiang T, Liu Z, Rao Y(2019) Attentional encoder network for targeted sentiment classification. arXiv preprint arXiv:1902.09314

  40. Do HH, Prasad PWC, Maag A, Alsadoon A (2019) Deep learning for aspect-based sentiment analysis: a comparative review. Expert Syst Appl 118:272–299

    Article  Google Scholar 

  41. Zhou J, Huang JX, Chen Q, Qinmin Vivian H, Wang T, He L (2019) Deep learning for aspect-level sentiment classification: survey, vision, and challenges. IEEE Access 7:78454–78483

    Article  Google Scholar 

  42. Li N, Chow CY, Zhang JD (2020) SEML: a semi-supervised multi-task learning framework for aspect-based sentiment analysis. IEEE Access 8:189287–189297

    Article  Google Scholar 

  43. Chen Z, Qian T(2019) Transfer capsule network for aspect level sentiment classification. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp 547–556

  44. Jindian S, Shanshan Yu, Luo D (2020) Enhancing aspect-based sentiment analysis with capsule network. IEEE Access 8:100551–100561

    Article  Google Scholar 

  45. Zhao F, Wu Z, Dai X(2020b) Attention transfer network for aspect-level sentiment classification. In: Proceedings of the 28th International Conference on Computational Linguistics, pp 811–821

  46. Dong L, Wei F, Tan C, Tang D, Zhou M, Xu K(2014) Adaptive recursive neural network for target-dependent twitter sentiment classification. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (volume 2: Short papers), pp 49–54

  47. Tang D, Qin B, Feng X, Liu T(2016b) Effective LSTMs for target-dependent sentiment classification. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp 3298–3307, Osaka, Japan, December. The COLING 2016 Organizing Committee. https://aclanthology.org/C16-1311

  48. Tay Y, Tuan LA, Hui SC(2018) Learning to attend via word-aspect associative fusion for aspect-based sentiment analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence

  49. Ma Y, Peng H, Cambria E(2018) Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM. In: Proceedings of the AAAI Conference on Artificial Intelligence

  50. Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423

  51. Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R(2020) ALBERT: A lite BERT for self-supervised learning of language representations. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, https://openreview.net/forum?id=H1eA7AEtvS

  52. Li B, Pan F, Shou Z, Zhang H(2021) Aspect based sentiment analysis of catering field reviews via roberta-aoa model. In: J Phys: Conf Ser. pp 012064. IOP Publishing

  53. Tay Y, Bahri D, Metzler D, Juan DC, Zhao Z, Zheng C(2021) Synthesizer: Rethinking self-attention for transformer models. In: International Conference on Machine Learning, pp 10183–10192. PMLR

  54. Kitaev N, Kaiser Ł, Levskaya A(2020) Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451

  55. Aziz RHH, Dimililer N (2021) Sentixgboost: enhanced sentiment analysis in social media posts with ensemble XGBoost classifier. J Chin Inst Eng 44(6):562–572

    Article  Google Scholar 

  56. Pontiki M, Galanis D, Pavlopoulos J, Papageorgiou H, Androutsopoulos I, Manandhar S (2014) SemEval-2014 task 4: Aspect based sentiment analysis. In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp 27–35, Dublin, Ireland, August. Association for Computational Linguistics. https://doi.org/10.3115/v1/S14-2004

  57. Pontiki M, Galanis D, Papageorgiou H, Manandhar S, Androutsopoulos I (2015) Semeval-2015 task 12: Aspect based sentiment analysis. In: Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pp 486–495

  58. Pontiki M, Galanis D, Papageorgiou H, Androutsopoulos I, Manandhar S, Al-Smadi M, Al-Ayyoub M, Zhao Y, Qin B, De C, Orphé et al. (2016) Semeval-2016 task 5: Aspect based sentiment analysis. In: International Workshop on Semantic Evaluation, pp 19–30

  59. Yang C, Zhang H, Jiang B, Li K (2019) Aspect-based sentiment analysis with alternating coattention networks. Inf Process Manag 56(3):463–478

    Article  Google Scholar 

  60. Fan F, Feng Y, Zhao D(2018) Multi-grained attention network for aspect-level sentiment classification. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 3433–3442

  61. He R, Lee WS, Ng HT, Dahlmeier D (2018) Exploiting document knowledge for aspect-level sentiment classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp 579–585, Melbourne, Australia, July. Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-2092

  62. Zhang C, Li Q, Song D (2019) Aspect-based sentiment classification with aspect-specific graph convolutional networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp 4568–4578, Hong Kong, China, November. Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1464

  63. Xu H, Liu B, Shu L, Yu P(2019) BERT post-training for review reading comprehension and aspect-based sentiment analysis. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp 2324–2335, Minneapolis, Minnesota, June. Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1242

  64. Meng W, Wei Y, Liu P, Zhu Z, Yin H (2019) Aspect based sentiment analysis with feature enhanced attention CNN-BiLSTM. IEEE Access 7:167240–167249

    Article  Google Scholar 

  65. Zeng B, Han X, Zeng F, Xu R, Yang H(2019) Multifeature interactive fusion model for aspect-based sentiment analysis. Math Probl Eng

  66. Wang K, Shen W, Yang Y, Quan X, Wang R(2020) Relational graph attention network for aspect-based sentiment analysis. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 3229–3238, Online, Jul. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.295

  67. Qiannan X, Zhu L, Dai T, Yan C (2020) Aspect-based sentiment classification with multi-attention network. Neurocomputing 388:135–143

    Article  Google Scholar 

  68. Tang H, Ji D, Li C, Zhou Q(2020) Dependency graph enhanced dual-transformer structure for aspect-based sentiment classification. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 6578–6588

  69. Shuang K, Mengyu G, Li R, Loo J, Sen S (2021) Interactive POS-aware network for aspect-level sentiment classification. Neurocomputing 420:181–196

    Article  Google Scholar 

  70. Mayi X, Biqing Z, Heng Y, Junlong C, Jiatao C, Hongye L (2022) Combining dynamic local context focus and dependency cluster attention for aspect-level sentiment classification. Neurocomputing 478:49–69

    Article  Google Scholar 

Download references

Funding

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Motilal Nehru National Institute of Technology Allahabad, Prayagraj, 211004, India

    Arvind Mewada & Rupesh Kumar Dewang

Authors
  1. Arvind Mewada
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rupesh Kumar Dewang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arvind Mewada.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mewada, A., Dewang, R.K. SA-ASBA: a hybrid model for aspect-based sentiment analysis using synthetic attention in pre-trained language BERT model with extreme gradient boosting. J Supercomput 79, 5516–5551 (2023). https://doi.org/10.1007/s11227-022-04881-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04881-x

Keywords

Search

Navigation