Abstract
In the sentiment analysis tasks, the idea of introducing external information to improve the prediction performance is sprouting up. The Topic Sentiment Joint Model shows that as the carrier of fine-grained sentiment, topics have an auxiliary effect on sentiment analysis. However, the current research on fine-grained view focuses on recognizing sentiment, but pays little attention to their impact on the overall feeling. To use topic information to assist sentiment analysis, this paper proposes a topic-enhanced sentiment analysis model. Through the multi-task learning framework, the topic information is learned and used to guide the model for sentiment classification. The multi-stage learning strategy guarantees the accuracy of prior knowledge. With the help of the alternating co-attention mechanism, the essential topics and emotional expression are concerned. In this paper, the labels of Chinese bank forum data and restaurant review data are automatically transformed to obtain supervised topic information. The experimental results show that the model has achieved state-of-the-art over the baseline in two datasets. The Kappa coefficient of sentiment prediction has increased by more than 1%. The model also has apparent advantages in interpretability and noise resistance.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data and source code is available at https://github.com/WithMeteor/TescaBERT
References
Ali, S.M., Noorian, Z., Bagheri, E., & et al. (2020). Topic and sentiment aware microblog summarization for twitter. Journal of Intelligent Information Systems, 54(1), 129–156. https://doi.org/10.1007/s10844-018-0521-8.
Blei, D.M., Ng, A.Y., & Jordan, M.I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022. https://doi.org/10.1162/jmlr.2003.3.4-5.993.
Bu, J., Ren, L., Zheng, S., & et al. (2021). Asap: A chinese review dataset towards aspect category sentiment analysis and rating prediction. arXiv:2103.06605.
Cambria, E., Poria, S., Gelbukh, A., & et al. (2018). Sentiment analysis is a big suitcase. Intelligent Systems, IEEE, 32(6), 74–80. https://doi.org/10.1109/MIS.2017.4531228.
Chintalapudi, N., Battineni, G., Di Canio, M., & et al. (2021). Text mining with sentiment analysis on seafarers’ medical documents. International Journal of Information Management Data Insights, 1(1), 1–9. https://doi.org/10.1016/j.jjimei.2020.100005.
Cho, K., Van Merriënboer, B., Gulcehre, C., & et al. (2014). Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv:1406.1078.
Cui, Y., Che, W., Liu, T., & et al. (2021). Pre-training with whole word masking for Chinese Bert. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 3504–3514. https://doi.org/10.1109/TASLP.2021.3124365.
Davidov, D., Tsur, O., & Rappoport, A. (2010). Enhanced sentiment learning using twitter hashtags and smileys. In Coling 2010: Posters (pp. 241–249).
Devlin, J., Chang, M.-W., Lee, K., & et al. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805.
Fu, X., Sun, X., Wu, H., & et al. (2018). Weakly supervised topic sentiment joint model with word embeddings. Knowledge-Based Systems, 147, 43–54. https://doi.org/10.1016/j.knosys.2018.02.012.
Hamouda, A., & Rohaim, M. (2011). Reviews classification using sentiwordnet lexicon. In World congress on computer science and information technology, (Vol. 23 pp. 104–105).
Haque, T.U., Saber, N.N., & Shah, F.M. (2018). Sentiment analysis on large scale amazon product reviews. In 2018 IEEE International Conference on Innovative Research and Development (ICIRD). https://doi.org/10.1109/ICIRD.2018.8376299 (pp. 1–6). IEEE.
Hofmann, T. (1999). Probabilistic latent semantic indexing. In Proceedings of the 22nd Annual International ACM SIGIR conference on research and development in information retrieval. https://doi.org/10.1145/312624.312649 (pp. 50–57).
Johnson, R., & Zhang, T. (2017). Deep pyramid convolutional neural networks for text categorization. In Proceedings of the 55th annual meeting of the association for computational linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/P17-1052 (pp. 562–570).
Joulin, A., Grave, E., Bojanowski, P., & et al. (2016). Bag of tricks for efficient text classification. arXiv:1607.01759.
Ke, P., Ji, H., Liu, S., & et al. (2019). Sentilare: Sentiment-aware language representation learning with linguistic knowledge. arXiv:1911.02493.
Khan, F.H., Qamar, U., & Bashir, S. (2016). esap: A decision support framework for enhanced sentiment analysis and polarity classification. Information Sciences, 367, 862–873. https://doi.org/10.1016/j.ins.2016.07.028.
Kim, Y. (2014). Convolutional neural networks for sentence classification. Eprint Arxiv. https://doi.org/10.3115/v1/D14-1181.
Li, M., Chen, L., Zhao, J., & et al. (2021). Sentiment analysis of chinese stock reviews based on bert model. Applied Intelligence, 51(7), 5016–5024. https://doi.org/10.1007/s10489-020-02101-8.
Li, F., Huang, M., & Zhu, X. (2010). Sentiment analysis with global topics and local dependency. In Proceedings of the AAAI conference on artificial intelligence. https://doi.org/10.1609/aaai.v24i1.7523, (Vol. 24 pp. 1371–1376).
Li, C., Zhang, J., Sun, J.-T., & et al. (2013). Sentiment topic model with decomposed prior. In Proceedings of the 2013 SIAM international conference on data mining. https://doi.org/10.1137/1.9781611972832.85 (pp. 767–775). SIAM.
Li, S., Zhao, Z., Hu, R., & et al. (2018). Analogical reasoning on chinese morphological and semantic relations. arXiv:1805.06504.
Lin, C., He, Y., Everson, R., & et al. (2011). Weakly supervised joint sentiment-topic detection from text. IEEE Transactions on Knowledge and Data Engineering, 24(6), 1134–1145. https://doi.org/10.1109/TKDE.2011.48.
Liu, P., Qiu, X., Chen, X., & et al. (2015). Multi-timescale long short-term memory neural network for modelling sentences and documents. In Proceedings of the 2015 conference on empirical methods in natural language processing. https://doi.org/10.18653/v1/D15-1280 (pp. 2326–2335).
Lu, J., Yang, J., Batra, D., & et al. (2016). Hierarchical co-attention for visual question answering. Advances in Neural Information Processing Systems (NIPS) 2. https://doi.org/10.48550/arXiv.1606.00061.
McAuley, J.J., & Leskovec, J. (2013). From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In Proceedings of the 22nd international conference on World Wide Web. https://doi.org/10.1145/2488388.2488466(pp. 897–908).
Mei, Q., Ling, X., Wondra, M., & et al. (2007). Topic sentiment mixture: modeling facets and opinions in weblogs. In Proceedings of the 16th international conference on World Wide Web. https://doi.org/10.1145/1242572.1242596 (pp. 171–180).
Nazir, A., Yuan, R., Wu, L., & et al. (2020). Issues and challenges of aspect-based sentiment analysis: A comprehensive survey. IEEE Transactions on Affective Computing, PP(99), 1–1. https://doi.org/10.1109/TAFFC.2020.2970399.
Ohana, B., & Tierney, B. (2009). Sentiment classification of reviews using sentiwordnet. Proceedings of IT&T, 22–23. https://doi.org/10.13140/2.1.4547.0089.
Ramaswamy, S.L., & Jayakumar, C. (2022). Recognet-lstm+cnn: a hybrid network with attention mechanism for aspect categorization and sentiment classification. Journal of Intelligent Information Systems, 58(2), 379–404. https://doi.org/10.1007/s10844-021-00692-3.
Ren, Y., Wang, R., & Ji, D. (2016). A topic-enhanced word embedding for twitter sentiment classification. Information Sciences, 369, 188–198. https://doi.org/10.1016/j.ins.2016.06.040.
Sharma, P., & Moh, T.-S. (2016). Prediction of indian election using sentiment analysis on hindi twitter. In 2016 IEEE international conference on big data (big Data). https://doi.org/10.1109/BigData.2016.7840818 (pp. 1966–1971). IEEE.
Singh, L.G., & Singh, S.R. (2021). Empirical study of sentiment analysis tools and techniques on societal topics. Journal of Intelligent Information Systems, 56(2), 379–407. https://doi.org/10.1007/s10844-020-00616-7.
Socher, R., Perelygin, A., Wu, J., & et al. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing (pp. 1631–1642).
Sun, Y., Wang, S., Li, Y., & et al. (2019). Ernie: Enhanced representation through knowledge integration. arXiv:1904.09223.
Tai, K.S., Socher, R., & Manning, C.D. (2015). Improved semantic representations from tree-structured long short-term memory networks. arXiv:1503.00075.
Tian, H., Gao, C., Xiao, X., & et al. (2020). Skep: Sentiment knowledge enhanced pre-training for sentiment analysis. arXiv:2005.05635.
Wang, B. (2018). Disconnected recurrent neural networks for text categorization. In Proceedings of the 56th annual meeting of the association for computational linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/P18-1215 (pp. 2311–2320).
Wang, Y., Huang, M., Zhu, X., & et al. (2016). Attention-based lstm for aspect-level sentiment classification. In Proceedings of the 2016 conference on empirical methods in natural language processing. https://doi.org/10.18653/v1/D16-1058 (pp. 606–615).
Wang, R., Li, Z., Cao, J., & et al. (2019). Convolutional recurrent neural networks for text classification. In 2019 international joint conference on neural networks (IJCNN). https://doi.org/10.1109/IJCNN.2019.8852406 (pp. 1–6). IEEE.
Wang, X., Wei, F., Liu, X., & et al. (2011). Topic sentiment analysis in twitter: a graph-based hashtag sentiment classification approach. In Proceedings of the 20th ACM international conference on information and knowledge management. https://doi.org/10.1145/2063576.2063726 (pp. 1031–1040).
Wu, D., Yang, R.X., & Shen, C. (2021). Sentiment word co-occurrence and knowledge pair feature extraction based lda short text clustering algorithm. Journal of Intelligent Information Systems, 56(1), 1–23. https://doi.org/10.1007/s10844-020-00597-7.
Xuan, J., Lu, J., Zhang, G., & et al. (2017). Nonparametric relational topic models through dependent gamma processes. IEEE Transactions on Knowledge & Data Engineering, 29(7), 1357–1369. https://doi.org/10.1109/TKDE.2016.2636182.
Yadav, A., & Vishwakarma, D.K. (2020). Sentiment analysis using deep learning architectures: a review. Artificial Intelligence Review, 53(6), 4335–4385. https://doi.org/10.1007/s10462-019-09794-5.
Yang, Z., Yang, D., Dyer, C., & et al. (2016). Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American chapter of the association for computational linguistics: Human language technologies. https://doi.org/10.18653/v1/N16-1174 (pp. 1480–1489).
Yang, C., Zhang, H., Jiang, B., & et al. (2019). Aspect-based sentiment analysis with alternating coattention networks. Information Processing & Management, 56(3), 463–478. https://doi.org/10.1016/j.ipm.2018.12.004.
Yin, D., Meng, T., & Chang, K.-W. (2020). Sentibert: A transferable transformer-based architecture for compositional sentiment semantics. arXiv:2005.04114.
Yue, L., Chen, W., Li, X., & et al. (2019). A survey of sentiment analysis in social media. Knowledge and Information Systems, 60(2), 617–663. https://doi.org/10.1007/s10115-018-1236-4.
Acknowledgements
This research is supported by the Science and Technology Research Program of the Department of Science and Technology of Henan Province (approval No.: 222102210081)
Author information
Authors and Affiliations
Contributions
Shiyu Wang: Data curation, Writing- Original draft preparation, Software. Gang Zhou: Conceptualization, Methodology, Validation. Jicang Lu: Writing - Reviewing and Editing, Investigation, Supervision. Jing Chen: Funding acquisition, Investigation, Supervision. Yi Xia: Formal analysis, Visualization.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not Applicable.
Consent for publication
Not Applicable.
Competing interests
The authors declare no competing interests.
Conflict of interests
The authors declare that they have no conflicts of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Jicang Lu and Jing Chen contributed equally to this work.
Appendix 1: Training Algorithm
Appendix 1: Training Algorithm
The algorithm for training TescaBERT is described as Algorithm 1.
Training of TescaBERT.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, S., Zhou, G., Lu, J. et al. Topic enhanced sentiment co-attention BERT. J Intell Inf Syst 60, 175–197 (2023). https://doi.org/10.1007/s10844-022-00749-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10844-022-00749-x