Abstract
Multi-target stance detection in tweets aims to detect the stance of given texts towards a specific target entity. Most existing models on stance detection consider word embedding as input, however, recent developments pointed out that it would be beneficial to incorporate feature-based information appropriately. Motivated by the strong performance of the pre-trained models in many Natural Language Processing field, and n-gram features that have been proved to be effective in prior competition, we present a novel combination module to obtain both advantages. This paper has proposed a pre-trained model integrated with n-gram features module (PMINFM) to better utilize multi-scale feature representation information and semantic features. Then connect it to a Bidirectional Long Short-Term Memory networks with target-specific attention mechanism. The experimental results show that our proposed model outperforms other baseline models in the SemEval-2016 stance detection dataset and achieves state-of-the-art performance.
Supported by National Key Research and Development Program of China No. 2018YFC1604000, Fundamental Research Funds for the Central Universities No. 2042017gf0035.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aldayel, A., Magdy, W.: Your stance is exposed! analysing possible factors for stance detection on social media. Proc. ACM Hum.-Comput. Interact. 3(CSCW), 1–20 (2019)
Alghanmi, I., Espinosa-Anke, L., Schockaert, S.: Combining BERT with static word embeddings for categorizing social media (2020)
Anand, P., Walker, M., Abbott, R., Tree, J.E.F., Bowmani, R., Minor, M.: Cats rule and dogs drool!: classifying stance in online debate. In: Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2011), pp. 1–9 (2011)
Augenstein, I., Rocktäschel, T., Vlachos, A., Bontcheva, K.: Stance detection with bidirectional conditional encoding. arXiv preprint arXiv:1606.05464 (2016)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Baly, R., et al.: A characterization study of Arabic twitter data with a benchmarking for state-of-the-art opinion mining models. In: Proceedings of the Third Arabic Natural Language Processing Workshop, pp. 110–118 (2017)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Du, J., Xu, R., He, Y., Gui, L.: Stance classification with target-specific neural attention networks. In: International Joint Conferences on Artificial Intelligence (2017)
Dulhanty, C., Deglint, J.L., Daya, I.B., Wong, A.: Taking a stance on fake news: towards automatic disinformation assessment via deep bidirectional transformer language models for stance detection. arXiv preprint arXiv:1911.11951 (2019)
Li, Y., Caragea, C.: Multi-task stance detection with sentiment and stance lexicons. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6300–6306 (2019)
Liu, Y., et al.: Roberta: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
Mohammad, S., Kiritchenko, S., Sobhani, P., Zhu, X., Cherry, C.: SemEval-2016 task 6: detecting stance in tweets. In: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pp. 31–41 (2016)
Mohammad, S.M., Sobhani, P., Kiritchenko, S.: Stance and sentiment in tweets. ACM Trans. Internet Technol. (TOIT) 17(3), 1–23 (2017)
Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Prakash, A., Madabushi, H.T.: Incorporating count-based features into pre-trained models for improved stance detection. arXiv preprint arXiv:2010.09078 (2020)
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)
Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)
Somasundaran, S., Wiebe, J.: Recognizing stances in ideological on-line debates. In: Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pp. 116–124 (2010)
Sun, Q., Wang, Z., Zhu, Q., Zhou, G.: Stance detection with hierarchical attention network. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 2399–2409 (2018)
Vlad, G.A., Tanase, M.A., Onose, C., Cercel, D.C.: Sentence-level propaganda detection in news articles with transfer learning and BERT-BiLSTM-capsule model. In: Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pp. 148–154 (2019)
Walker, M., Anand, P., Abbott, R., Grant, R.: Stance classification using dialogic properties of persuasion. In: Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 592–596 (2012)
Wang, L., Cardie, C.: Improving agreement and disagreement identification in online discussions with a socially-tuned sentiment lexicon. arXiv preprint arXiv:1606.05706 (2016)
Wei, P., Mao, W., Zeng, D.: A target-guided neural memory model for stance detection in twitter. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2018)
Wei, W., Zhang, X., Liu, X., Chen, W., Wang, T.: pkudblab at SemEval-2016 task 6: a specific convolutional neural network system for effective stance detection. In: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pp. 384–388 (2016)
Yang, R., Xie, W., Liu, C., Yu, D.: BLCU\_NLP at SemEval-2019 task 7: an inference chain-based GPT model for rumour evaluation. In: Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 1090–1096 (2019)
Yang, Y., Wu, B., Zhao, K., Guo, W.: Tweet stance detection: a two-stage DC-BiLSTM model based on semantic attention. In: 2020 IEEE Fifth International Conference on Data Science in Cyberspace (DSC), pp. 22–29. IEEE (2020)
Zhang, C., Yamana, H.: WUY at SemEval-2020 task 7: combining BERT and Naïve Bayes-SVM for humor assessment in edited news headlines. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 1071–1076 (2020)
Zhou, Y., Cristea, A.I., Shi, L.: Connecting targets to tweets: semantic attention-based model for target-specific stance detection. In: Bouguettaya, A., et al. (eds.) WISE 2017, Part I. LNCS, vol. 10569, pp. 18–32. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68783-4_2
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, P., Ye, K., Cui, X. (2021). Integrating N-Gram Features into Pre-trained Model: A Novel Ensemble Model for Multi-target Stance Detection. In: Farkaš, I., Masulli, P., Otte, S., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2021. ICANN 2021. Lecture Notes in Computer Science(), vol 12893. Springer, Cham. https://doi.org/10.1007/978-3-030-86365-4_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-86365-4_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86364-7
Online ISBN: 978-3-030-86365-4
eBook Packages: Computer ScienceComputer Science (R0)