Skip to main content
SpringerLink
Log in

Attention-Based Model for Accurate Stance Detection

  • Conference paper
  • First Online:
Text, Speech, and Dialogue (TSD 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13502))

Included in the following conference series:

  • 883 Accesses

Abstract

Effective representation learning is an essential building block for achieving many natural language processing tasks such as stance detection as performed implicitly by humans. Stance detection can assist in understanding how individuals react to certain information by revealing the user’s stance on a particular topic. In this work, we propose a new attention-based model for learning feature representations and show its effectiveness in the task of stance detection. The proposed model is based on transfer learning and multi-head attention mechanisms. Specifically, we use BERT and word2vec models to learn text representation vectors from the data and pass both of them simultaneously to the multi-head attention layer to help focus on the best learning features. We present five variations of the model, each with a different combination of BERT and word2vec embeddings for the query and value parameters of the attention layer. The performance of the proposed model is evaluated against multiple baseline and state-of-the-art models. The best of the five proposed variations of the model improved the accuracy on average by 0.4% and achieved 68.4% accuracy for multi-classification, while the best accuracy for binary classification is 86.1% with a 1.3% improvement.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://ieee-dataport.org/9221.

References

  1. Miao, L., Last, M., Litwak, M.: Tracking social media during the COVID-19 pandemic: the case study of lockdown in New York state. Expert Syst. Appl. 187, 115797 (2022)

    Google Scholar 

  2. Al-Ghadir, A.I., Azmi, A.M., Hussain, A.: A novel approach to stance detection in social media tweets by fusing ranked lists and sentiments. Inf. Fusion 67, 29–40 (2021)

    Article  Google Scholar 

  3. Allaway, E., Srikanth, M., McKeown, K.: Adversarial learning for zero-shot stance detection on social media. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online (2021)

    Google Scholar 

  4. Anand, P., Walker, M., Abbott, R., Tree, J.E.F., Bowmani, R., Minor, M.: Cats rule and dogs drool!: classifying stance in online debate. In: Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011), pp. 1–9 (2011)

    Google Scholar 

  5. Bao, J., Zhang, L., Han, B.: Collaborative attention network with word and n-gram sequences modeling for sentiment classification. In: Tetko, I.V., Kůrková, V., Karpov, P., Theis, F. (eds.) ICANN 2019. LNCS, vol. 11730, pp. 79–92. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30490-4_8

    Chapter  Google Scholar 

  6. Barbieri, F., Camacho-Collados, J., Neves, L., Espinosa-Anke, L.: TweetEval: unified benchmark and comparative evaluation for tweet classification. arXiv preprint arXiv:2010.12421 (2020)

  7. Bello-Orgaz, G., Hernandez-Castro, J., Camacho, D.: Detecting discussion communities on vaccination in twitter. Futur. Gener. Comput. Syst. 66, 125–136 (2017)

    Article  Google Scholar 

  8. Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996)

    MATH  Google Scholar 

  9. Crammer, K., Singer, Y.: On the algorithmic implementation of multiclass Kernel-based vector machines. J. Mach. Learn. Res. 2, 265–292 (2001)

    Google Scholar 

  10. Das, B., Krishnan, N.C., Cook, D.J.: Handling imbalanced and overlapping classes in smart environments prompting dataset. In: Yada, K. (ed.) Data Mining for Service. SBD, vol. 3, pp. 199–219. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-45252-9_12

    Chapter  Google Scholar 

  11. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota (2019)

    Google Scholar 

  12. Du, J., Xu, R., He, Y., Gui, L.: Stance classification with target-specific neural attention networks.In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (2017)

    Google Scholar 

  13. Ghafarian, S.H., Yazdi, H.S.: Identifying crisis-related informative tweets using learning on distributions. Inf. Process. Manage. 57(2), 102145 (2020)

    Article  Google Scholar 

  14. Glandt, K., Khanal, S., Li, Y., Caragea, D., Caragea, C.: Stance detection in COVID-19 tweets. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online (2021)

    Google Scholar 

  15. Khadhraoui, M., Bellaaj, H., Ammar, M.B., Hamam, H., Jmaiel, M.: Survey of BERT-base models for scientific text classification: COVID-19 case study. Appl. Sci. 12(6), 2891 (2022)

    Article  Google Scholar 

  16. Lai, S., Xu, L., Liu, K., Zhao, J.: Recurrent convolutional neural networks for text classification. In: AAAI2015: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 2267–2273 (2015)

    Google Scholar 

  17. Lanjewar, R., Mathurkar, S., Patel, N.: Implementation and comparison of speech emotion recognition system using Gaussian mixture model (GMM) and k-nearest neighbor (k-NN) techniques. Procedia Comput. Sci. 49, 50–57 (2015)

    Google Scholar 

  18. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)

    Google Scholar 

  19. Mingers, J.: An empirical comparison of pruning methods for decision tree induction. Mach. Learn. 4(2), 227–243 (1989)

    Article  Google Scholar 

  20. Ramadhan, W., Novianty, S.A., Setianingsih, S.C.: Sentiment analysis using multinomial logistic regression. In: 2017 International Conference on Control, Electronics, Renewable Energy and Communications (ICCREC), pp. 46–49. IEEE (2017)

    Google Scholar 

  21. Ren, J., Lee, S.D., Chen, X., Kao, B., Cheng, R., Cheung, D.: Naive Bayes classification of uncertain data. In: 2009 Ninth IEEE International Conference on Data Mining, pp. 944–949. IEEE (2009)

    Google Scholar 

  22. Song, Y., Wang, J., Jiang, T., Liu, Z., Rao, Y.: Targeted sentiment classification with attentional encoder network. In: Tetko, I.V., Kůrková, V., Karpov, P., Theis, F. (eds.) ICANN 2019. LNCS, vol. 11730, pp. 93–103. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30490-4_9

    Chapter  Google Scholar 

  23. Vizcarra, G., Mauricio, A., Mauricio, L.: A deep learning approach for sentiment analysis in spanish tweets. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11141, pp. 622–629. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01424-7_61

    Chapter  Google Scholar 

  24. Walker, M.A., Anand, P., Abbott, R., Tree, J.E.F., Martell, C., King, J.: That is your evidence?: classifying stance in online political debate. Decis. Support Syst. 53(4), 719–729 (2012)

    Article  Google Scholar 

  25. Wang, S.I., Manning, C.D.: Baselines and bigrams: simple, good sentiment and topic classification. In: Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 90–94 (2012)

    Google Scholar 

  26. Wei, J., Liao, J., Yang, Z., Wang, S., Zhao, Q.: BiLSTM with multi-polarity orthogonal attention for implicit sentiment analysis. Neurocomputing 383, 165–173 (2020)

    Article  Google Scholar 

  27. Wei, W., Zhang, X., Liu, X., Chen, W., Wang, T.: Pkudblab at SemEval-2016 task 6: a specific convolutional neural network system for effective stance detection. In: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pp. 384–388 (2016)

    Google Scholar 

  28. Wu, H., Qin, S., Nie, R., Cao, J., Gorbachev, S.: Effective collaborative representation learning for multilabel text categorization. In: IEEE Transactions on Neural Networks and Learning Systems (2021)

    Google Scholar 

  29. Zhou, J., Xu, W.: End-to-end learning of semantic role labeling using recurrent neural networks. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1127–1137 (2015)

    Google Scholar 

  30. Zhou, S., Lin, J., Tan, L., Liu, X.: Condensed convolution neural network by attention over self-attention for stance detection in twitter. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science and Engineering Department, Qatar University, Doha, Qatar

    Omama Hamad & Khaled Shaban

  2. Computer Science School, University of Adelaide, Adelaide, Australia

    Ali Hamdi

Authors
  1. Omama Hamad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ali Hamdi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Khaled Shaban
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Omama Hamad .

Editor information

Editors and Affiliations

  1. Faculty of Informatics, Masaryk University, Brno, Czech Republic

    Petr Sojka

  2. Faculty of Informatics, Masaryk University, Brno, Czech Republic

    Aleš Horák

  3. Faculty of Informatics, Masaryk University, Brno, Czech Republic

    Ivan Kopeček

  4. Faculty of Informatics, Masaryk University, Brno, Czech Republic

    Karel Pala

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hamad, O., Hamdi, A., Shaban, K. (2022). Attention-Based Model for Accurate Stance Detection. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds) Text, Speech, and Dialogue. TSD 2022. Lecture Notes in Computer Science(), vol 13502. Springer, Cham. https://doi.org/10.1007/978-3-031-16270-1_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16270-1_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16269-5

  • Online ISBN: 978-3-031-16270-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation