Skip to main content

Checkworthiness in Automatic Claim Detection Models: Definitions and Analysis of Datasets

  • Conference paper
  • First Online:
Disinformation in Open Online Media (MISDOOM 2020)

Abstract

Public, professional and academic interest in automated fact-checking has drastically increased over the past decade, with many aiming to automate one of the first steps in a fact-check procedure: the selection of so-called checkworthy claims. However, there is little agreement on the definition and characteristics of checkworthiness among fact-checkers, which is consequently reflected in the datasets used for training and testing checkworthy claim detection models. After elaborate analysis of checkworthy claim selection procedures in fact-check organisations and analysis of state-of-the-art claim detection datasets, checkworthiness is defined as the concept of having a spatiotemporal and context-dependent worth and need to have the correctness of the objectivity it conveys verified. This is irrespective of the claim’s perceived veracity judgement by an individual based on prior knowledge and beliefs. Concerning the characteristics of current datasets, it is argued that the data is not only highly imbalanced and noisy, but also too limited in scope and language. Furthermore, we believe that the subjective concept of checkworthiness might not be a suitable filter for claim detection.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Check: Cambridge Dictionary. Cambridge University Press (2019). https://dictionary.cambridge.org/dictionary/english/check. Accessed 23 Dec 2019

  2. Worth: Oxford English Dictionary. Oxford University Press (2019). http://www.oed.com/view/Entry/230376. Accessed 6 Dec 2019

  3. Worthiness: Oxford English Dictionary. Oxford University Press (2019). http://www.oed.com/view/Entry/230389. Accessed 6 Dec 2019

  4. Worthiness: Cambridge Dictionary. Cambridge University Press (2019). https://dictionary.cambridge.org/dictionary/english/worthiness. Accessed 6 Dec 2019

  5. Adair, B.: Duke study finds fact-checking growing around the world. Duke University Reporter’s Lab (April 2014). https://reporterslab.org/duke-study-finds-fact-checking-growing-around-the-world/

  6. Alter, A.L., Oppenheimer, D.M.: Uniting the tribes of fluency to form a metacognitive nation. Pers. Soc. Psychol. Rev. 13(3), 219–235 (2009)

    Article  Google Scholar 

  7. Atanasova, P., et al.: Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims. Task 1: check-worthiness. In: CEUR Workshop Proceedings, Avignon, France (2018)

    Google Scholar 

  8. Atanasova, P., Nakov, P., Karadzhov, G., Mohtarami, M., Da San Martino, G.: Overview of the CLEF-2019 CheckThat! Lab on automatic identification and verification of claims. Task 1: check-worthiness. In: CEUR Workshop Proceedings, Lugano, Switzerland (2019)

    Google Scholar 

  9. Cruz, N.P., Taboada, M., Mitkov, R.: A machine-learning approach to negation and speculation detection for sentiment analysis. J. Assoc. Inf. Sci. Technol. 67(9), 2118–2136 (2016)

    Article  Google Scholar 

  10. Dechêne, A., Stahl, C., Hansen, J., Wänke, M.: The truth about the truth: a meta-analytic review of the truth effect. Pers. Soc. Psychol. Rev. 14(2), 238–257 (2010)

    Article  Google Scholar 

  11. Dobric Holan, A.: The principles of the truth-o-meter: politifact’s methodology for independent fact-checking (2018). https://www.politifact.com/truth-o-meter/article/2018/feb/12/principles-truth-o-meter-politifacts-methodology-i/#How%20we%20choose%20claims

  12. FactCheck.org: Our process (2019). https://www.factcheck.org/our-process/. Accessed 16 Dec 2019

  13. Favano, L., Carman, M., Lanzi, P.: TheEarthIsFlat’s submission to CLEF’19 CheckThat! challenge. In: CLEF 2019 Working Notes, Working Notes of CLEF 2019 Conference and Labs of the Evaluation Forum (2019)

    Google Scholar 

  14. Fazio, L.K., Brashier, N.M., Payne, B.K., Marsh, E.J.: Knowledge does not protect against illusory truth. J. Exp. Psychol.: Gen. 144(5), 993 (2015)

    Article  Google Scholar 

  15. Fullfact.org: Effectiveness - full fact (2019). https://fullfact.org/about/effectiveness/. Accessed 6 Dec 2019

  16. Gencheva, P., Nakov, P., Màrquez, L., Barrón-Cedeño, A., Koychev, I.: A context-aware approach for detecting worth-checking claims in political debates. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pp. 267–276 (2017)

    Google Scholar 

  17. Graves, L.: Deciding what’s true: fact-checking journalism and the new ecology of news. Ph.D. thesis, Columbia University (2013)

    Google Scholar 

  18. Graves, L., Nyhan, B., Reifler, J.: Why do journalists fact-check (2016). https://www.dartmouth.edu/~nyhan/journalist-fact-checking.pdf

  19. Halliday, M.A., Horowitz, R., Samuels, S.J.: Comprehending Oral and Written Language. Emeral Group, Bingley (1987)

    Google Scholar 

  20. Hansen, C., Hansen, C., Simonsen, J., Lioma, C.: Neural weakly supervised fact check-worthiness detection with contrastive sampling-based ranking loss. In: 20th Working Notes of CLEF Conference and Labs of the Evaluation Forum, CLEF 2019 Conference and Labs of the Evaluation Forum, vol. 2380 (2019)

    Google Scholar 

  21. Hansen, C., Hansen, C., Simonsen, J.G., Lioma, C.: The Copenhagen team participation in the check-worthiness task of the competition of automatic identification and verification of claims in political debates of the CLEF-2018 CheckThat! Lab. In: CLEF (Working Notes) (2018)

    Google Scholar 

  22. Hasanain, M., Suwaileh, R., Elsayed, T., Barrón-Cedeno, A., Nakov, P.: Overview of the CLEF-2019 CheckThat! Lab on automatic identification and verification of claims. Task 2: evidence and factuality. In: CEUR Workshop Proceedings, Lugano, Switzerland (2019)

    Google Scholar 

  23. Hassan, N., Arslan, F., Li, C., Tremayne, M.: Toward automated fact-checking: detecting check-worthy factual claims by ClaimBuster. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1803–1812. ACM (2017)

    Google Scholar 

  24. Hassan, N., Li, C., Tremayne, M.: Detecting check-worthy factual claims in presidential debates. In: Proceedings of the 24th ACM International Conference on Information and Knowledge Management, pp. 1835–1838. ACM (2015)

    Google Scholar 

  25. Jimenez, D., Li, C.: An empirical study on identifying sentences with salient factual statements. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2018)

    Google Scholar 

  26. Kazemian, B., Hashemi, S.: Critical discourse analysis of Barack Obama’s 2012 speeches: views from systemic functional linguistics and rhetoric. Theory Pract. Lang. Stud. (TPLS) 4(6), 1178–1187 (2014)

    Google Scholar 

  27. Levy, R., Bilu, Y., Hershcovich, D., Aharoni, E., Slonim, N.: Context dependent claim detection. In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 1489–1500 (2014)

    Google Scholar 

  28. Nakov, P., et al.: Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims. In: Bellot, P., et al. (eds.) CLEF 2018. LNCS, vol. 11018, pp. 372–387. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98932-7_32

    Chapter  Google Scholar 

  29. Patwari, A., Goldwasser, D., Bagchi, S.: Tathya: A multi-classifier system for detecting check-worthy statements in political debates. In: Proceedings of the 2017 ACM Conference on Information and Knowledge Management, pp. 2259–2262. ACM (2017)

    Google Scholar 

  30. Qian, Z., Li, P., Zhu, Q., Zhou, G., Luo, Z., Luo, W.: Speculation and negation scope detection via convolutional neural networks. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 815–825 (2016)

    Google Scholar 

  31. Scholl, S.G., Greifeneder, R., Bless, H.: When fluency signals truth: prior successful reliance on fluency moderates the impact of fluency on truth judgments. J. Behav. Decis. Mak. 27(3), 268–280 (2014)

    Article  Google Scholar 

  32. Snopes.com: Transparancy (2019). https://www.snopes.com/transparency/. Accessed 6 Dec 2019

  33. Stencel, M., Luther, J.: Reporter’s lab fact-checking tally tops 200. Duke University Reporter’s Lab (October 2019). https://reporterslab.org/category/fact-checking/#article-2551

  34. Thorne, J., Vlachos, A., Cocarascu, O., Christodoulopoulos, C., Mittal, A.: The Fact Extraction and VERification (FEVER) shared task. In: Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pp. 1–9. Association for Computational Linguistics, Brussels (2018)

    Google Scholar 

  35. Truthorfiction.com: Our methodology and process (2019). https://www.truthorfiction.com/our-methodology-and-process/. Accessed 6 Dec 2019

  36. Zubiaga, A., Aker, A., Bontcheva, K., Liakata, M., Procter, R.: Detection and resolution of rumours in social media: a survey. ACM Comput. Surv. (CSUR) 51(2), 32 (2018)

    Article  Google Scholar 

  37. Zuo, C., Karakas, A., Banerjee, R.: A hybrid recognition system for check-worthy claims using heuristics and supervised learning. In: CLEF (Working Notes) (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Liesbeth Allein or Marie-Francine Moens .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Allein, L., Moens, MF. (2020). Checkworthiness in Automatic Claim Detection Models: Definitions and Analysis of Datasets. In: van Duijn, M., Preuss, M., Spaiser, V., Takes, F., Verberne, S. (eds) Disinformation in Open Online Media. MISDOOM 2020. Lecture Notes in Computer Science(), vol 12259. Springer, Cham. https://doi.org/10.1007/978-3-030-61841-4_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-61841-4_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-61840-7

  • Online ISBN: 978-3-030-61841-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics