Skip to main content
SpringerLink
Log in

Did I See It Before? Detecting Previously-Checked Claims over Twitter

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13185))

Included in the following conference series:

Abstract

With the proliferation of fake news in the last few years, especially during the COVID-19 period, combating the spread of misinformation has become an urgent need. Although automated fact-checking systems were proposed recently, they leave much to be desired in terms of accuracy and explainability. Therefore, involving humans during verification could make the process much easier and more reliable. In this work, we propose an automated approach to detect claims that have been already manually-verified by professional fact-checkers. Our proposed approach uses recent powerful BERT variants as point-wise rerankers. Additionally, we study the impact of using different fields of the verified claim during training and inference phases. Experimental results show that our proposed pipeline outperforms the state-of-the-art approaches on two English and one Arabic datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.politifact.com/.

  2. 2.

    http://fullfact.org/.

  3. 3.

    We specifically use STSb-MPNet and Paraphrase-MPNet.

  4. 4.

    https://huggingface.co/sentence-transformers/msmarco-roberta-base-v2.

  5. 5.

    https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L12-v2.

  6. 6.

    https://www.sbert.net/docs/pretrained_models.html.

References

  1. Abdul-Mageed, M., Elmadany, A., Nagoudi, E.M.B.: ARBERT & MARBERT: deep bidirectional transformers for Arabic. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7088–7105, August 2021

    Google Scholar 

  2. Ali, Z.S., Mansour, W., Elsayed, T., Al-Ali, A.: AraFacts: the first large Arabic dataset of naturally occurring claims. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 231–236 (2021)

    Google Scholar 

  3. Amati, G., Amodeo, G., Bianchi, M., Gaibisso, C., Gambosi, G.: FUB, IASI-CNR and university of “Tor Vergata” at TREC 2008 blog track. In: The Seventeenth Text REtrieval Conference, TREC 2008, US (2008)

    Google Scholar 

  4. Antoun, W., Baly, F., Hajj, H.: AraBERT: Transformer-based model for Arabic language understanding. In: Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pp. 9–15 (2020)

    Google Scholar 

  5. Barrón-Cedeño, A., et al.: Overview of CheckThat! 2020 – automatic identification and verification of claims in social media. In: Proceedings of the 11th International Conference of the CLEF Association: Experimental IR Meets Multilinguality, Multimodality, and Interaction, CLEF 2020, pp. 215–236 (2020)

    Google Scholar 

  6. Benjamini, Y., Hochberg, Y.: Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Stat. Soc. Ser. B (Methodol.) 57(1), 289–300 (1995)

    MathSciNet  MATH  Google Scholar 

  7. Bouziane, M., Perrin, H., Cluzeau, A., Mardas, J., Sadeq, A.: Buster.AI at CheckThat! 2020: insights and recommendations to improve fact-checking. In: Cappellato et al. [8] (2020)

    Google Scholar 

  8. Cappellato, L., Eickhoff, C., Ferro, N., Névéol, A. (eds.): CLEF 2020 Working Notes. CEUR Workshop Proceedings. CEUR-WS.org (2020)

    Google Scholar 

  9. Chernyavskiy, A., Ilvovsky, D., Nakov, P.: Aschern at CheckThat! 2021: lambda-calculus of fact-checked claims. In: Faggioli et al. [12] (2021)

    Google Scholar 

  10. Chowdhury, S.A., Abdelali, A., Darwish, K., Soon-Gyo, J., Salminen, J., Jansen, B.J.: Improving Arabic text categorization using transformer training diversification. In: Proceedings of the Fifth Arabic Natural Language Processing Workshop, pp. 226–236 (2020)

    Google Scholar 

  11. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186 (2019)

    Google Scholar 

  12. Faggioli, G., Ferro, N., Joly, A., Maistro, M., Piroi, F. (eds.): CLEF 2021 Working Notes. Working Notes of CLEF 2021-Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings. CEUR-WS.org (2021)

    Google Scholar 

  13. Farha, I.A., Magdy, W.: Benchmarking transformer-based language models for Arabic sentiment and sarcasm detection. In: Proceedings of the Sixth Arabic Natural Language Processing Workshop, pp. 21–31 (2021)

    Google Scholar 

  14. Google: Fact check. https://toolbox.google.com/factcheck/explorer

  15. Gunther, R., Beck, P.A., Nisbet, E.C.: Fake news did have a significant impact on the vote in the 2016 election: original full-length version with methodological appendix, Ohio State University, Columbus, OH. Unpublished manuscript (2018)

    Google Scholar 

  16. Jelinek, F.: Interpolated estimation of Markov source parameters from sparse data. In: Proceedings of the Workshop on Pattern Recognition in Practice, 1980 (1980)

    Google Scholar 

  17. Kazemi, A., Garimella, K., Gaffney, D., Hale, S.: Claim matching beyond English to scale global fact-checking. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 4504–4517 (2021)

    Google Scholar 

  18. Lan, W., Chen, Y., Xu, W., Ritter, A.: GigaBERT: zero-shot transfer learning from English to Arabic. In: Proceedings of The 2020 Conference on Empirical Methods on Natural Language Processing (EMNLP) (2020)

    Google Scholar 

  19. Lavrenko, V., Croft, W.B.: Relevance based language models. In: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 120–127 (2001)

    Google Scholar 

  20. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  21. Macdonald, C., Tonellotto, N.: Declarative experimentation in information retrieval using PyTerrier. In: Proceedings of ICTIR 2020 (2020)

    Google Scholar 

  22. McDonald, T., et al.: The university of sheffield at CheckThat! 2020: claim identification and verification on Twitter. In: Cappellato et al. [8] (2020)

    Google Scholar 

  23. Nakov, P., et al.: Automated fact-checking for assisting human fact-checkers. In: International Joint Conferences on Artificial Intelligence (2021)

    Google Scholar 

  24. Nakov, P., et al.: Overview of the CLEF–2021 CheckThat! Lab on detecting check-worthy claims, previously fact-checked claims, and fake news. In: Candan, K.S., Ionescu, B., Goeuriot, L., Larsen, B., Müller, H., Joly, A., Maistro, M., Piroi, F., Faggioli, G., Ferro, N. (eds.) CLEF 2021. LNCS, vol. 12880, pp. 264–291. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85251-1_19

    Chapter  Google Scholar 

  25. Nogueira, R., Yang, W., Cho, K., Lin, J.: Multi-stage document ranking with BERT. arXiv:abs/1910.14424 (2019)

  26. Passaro, L., Bondielli, A., Lenci, A., Marcelloni, F.: UNIPI-NLE at CheckThat! 2020: approaching fact checking from a sentence similarity perspective through the lens of transformers. In: Cappellato et al. [8] (2020)

    Google Scholar 

  27. Reimers, N., Gurevych, I.: Sentence-BERT: sentence embeddings using siamese BERT-networks. arXiv preprint arXiv:1908.10084 (2019)

  28. Robertson, S., Zaragoza, H.: The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. 3(4), 333–389 (2009)

    Article  Google Scholar 

  29. Safaya, A.: Arabic-ALBERT, August 2020. https://doi.org/10.5281/zenodo.4718724

  30. Safaya, A., Abdullatif, M., Yuret, D.: Kuisail at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 2054–2059 (2020)

    Google Scholar 

  31. Shaar, S., Alam, F., Martino, G.D.S., Nakov, P.: The role of context in detecting previously fact-checked claims. arXiv preprint arXiv:2104.07423 (2021)

  32. Shaar, S., Babulkov, N., Da San Martino, G., Nakov, P.: That is a known lie: detecting previously fact-checked claims. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3607–3618 (2020)

    Google Scholar 

  33. Shaar, S., et al.: Overview of the CLEF-2021 CheckThat! lab task 2 on detecting previously fact-checked claims in tweets and political debates. In: Faggioli et al. [12] (2021)

    Google Scholar 

  34. Sheng, Q., Cao, J., Zhang, X., Li, X., Zhong, L.: Article reranking by memory-enhanced key sentence matching for detecting previously fact-checked claims. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 5468–5481 (2021)

    Google Scholar 

  35. Shu, K., Sliva, A., Wang, S., Tang, J., Liu, H.: Fake news detection on social media: a data mining perspective. ACM SIGKDD Explor. Newsl. 19(1), 22–36 (2017)

    Article  Google Scholar 

  36. Snowdon, R.: Meta reverse image search. https://github.com/vivithemage/mrisa (2017)

  37. Song, K., Tan, X., Qin, T., Lu, J., Liu, T.Y.: MPNet: masked and permuted pre-training for language understanding. In: Advances in Neural Information Processing Systems, vol. 33, pp. 16857–16867. Curran Associates, Inc. (2020)

    Google Scholar 

  38. Tchechmedjiev, A., et al.: ClaimsKG: a knowledge graph of fact-checked claims. In: Ghidini, C., et al. (eds.) ISWC 2019. LNCS, vol. 11779, pp. 309–324. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30796-7_20

    Chapter  Google Scholar 

  39. Thorne, J., Vlachos, A.: Automated fact checking: task formulations, methods and future directions. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 3346–3359 (2018)

    Google Scholar 

  40. Vo, N., Lee, K.: Where Are the Facts? Searching for fact-checked information to alleviate the spread of fake news. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7717–7731 (2020)

    Google Scholar 

  41. Vosoughi, S., Roy, D., Aral, S.: The spread of true and false news online. Science 359(6380), 1146–1151 (2018)

    Article  Google Scholar 

  42. Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: MiniLM: deep self-attention distillation for task-agnostic compression of pre-trained transformers. In: Advances in Neural Information Processing Systems, vol. 33, pp. 5776–5788. Curran Associates, Inc. (2020)

    Google Scholar 

Download references

Acknowledgments

This work was made possible by NPRP grant# NPRP11S-1204-170060 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors.

Author information

Authors and Affiliations

  1. Computer Science and Engineering Department, Qatar University, Doha, Qatar

    Watheq Mansour, Tamer Elsayed & Abdulaziz Al-Ali

  2. KINDI Center for Computing Research, Qatar University, Doha, Qatar

    Abdulaziz Al-Ali

Authors
  1. Watheq Mansour
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tamer Elsayed
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abdulaziz Al-Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Watheq Mansour .

Editor information

Editors and Affiliations

  1. Martin Luther University Halle-Wittenberg, Halle, Germany

    Matthias Hagen

  2. Leiden University, Leiden, The Netherlands

    Suzan Verberne

  3. University of Glasgow, Glasgow, UK

    Craig Macdonald

  4. University of Duisburg-Essen, Essen, Germany

    Christin Seifert

  5. University of Stavanger, Stavanger, Norway

    Krisztian Balog

  6. Norwegian University of Science and Technology, Trondheim, Norway

    Kjetil Nørvåg

  7. University of Stavanger, Stavanger, Norway

    Vinay Setty

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mansour, W., Elsayed, T., Al-Ali, A. (2022). Did I See It Before? Detecting Previously-Checked Claims over Twitter. In: Hagen, M., et al. Advances in Information Retrieval. ECIR 2022. Lecture Notes in Computer Science, vol 13185. Springer, Cham. https://doi.org/10.1007/978-3-030-99736-6_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-99736-6_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-99735-9

  • Online ISBN: 978-3-030-99736-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation