Abstract
Public, professional and academic interest in automated fact-checking has drastically increased over the past decade, with many aiming to automate one of the first steps in a fact-check procedure: the selection of so-called checkworthy claims. However, there is little agreement on the definition and characteristics of checkworthiness among fact-checkers, which is consequently reflected in the datasets used for training and testing checkworthy claim detection models. After elaborate analysis of checkworthy claim selection procedures in fact-check organisations and analysis of state-of-the-art claim detection datasets, checkworthiness is defined as the concept of having a spatiotemporal and context-dependent worth and need to have the correctness of the objectivity it conveys verified. This is irrespective of the claim’s perceived veracity judgement by an individual based on prior knowledge and beliefs. Concerning the characteristics of current datasets, it is argued that the data is not only highly imbalanced and noisy, but also too limited in scope and language. Furthermore, we believe that the subjective concept of checkworthiness might not be a suitable filter for claim detection.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Check: Cambridge Dictionary. Cambridge University Press (2019). https://dictionary.cambridge.org/dictionary/english/check. Accessed 23 Dec 2019
Worth: Oxford English Dictionary. Oxford University Press (2019). http://www.oed.com/view/Entry/230376. Accessed 6 Dec 2019
Worthiness: Oxford English Dictionary. Oxford University Press (2019). http://www.oed.com/view/Entry/230389. Accessed 6 Dec 2019
Worthiness: Cambridge Dictionary. Cambridge University Press (2019). https://dictionary.cambridge.org/dictionary/english/worthiness. Accessed 6 Dec 2019
Adair, B.: Duke study finds fact-checking growing around the world. Duke University Reporter’s Lab (April 2014). https://reporterslab.org/duke-study-finds-fact-checking-growing-around-the-world/
Alter, A.L., Oppenheimer, D.M.: Uniting the tribes of fluency to form a metacognitive nation. Pers. Soc. Psychol. Rev. 13(3), 219–235 (2009)
Atanasova, P., et al.: Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims. Task 1: check-worthiness. In: CEUR Workshop Proceedings, Avignon, France (2018)
Atanasova, P., Nakov, P., Karadzhov, G., Mohtarami, M., Da San Martino, G.: Overview of the CLEF-2019 CheckThat! Lab on automatic identification and verification of claims. Task 1: check-worthiness. In: CEUR Workshop Proceedings, Lugano, Switzerland (2019)
Cruz, N.P., Taboada, M., Mitkov, R.: A machine-learning approach to negation and speculation detection for sentiment analysis. J. Assoc. Inf. Sci. Technol. 67(9), 2118–2136 (2016)
Dechêne, A., Stahl, C., Hansen, J., Wänke, M.: The truth about the truth: a meta-analytic review of the truth effect. Pers. Soc. Psychol. Rev. 14(2), 238–257 (2010)
Dobric Holan, A.: The principles of the truth-o-meter: politifact’s methodology for independent fact-checking (2018). https://www.politifact.com/truth-o-meter/article/2018/feb/12/principles-truth-o-meter-politifacts-methodology-i/#How%20we%20choose%20claims
FactCheck.org: Our process (2019). https://www.factcheck.org/our-process/. Accessed 16 Dec 2019
Favano, L., Carman, M., Lanzi, P.: TheEarthIsFlat’s submission to CLEF’19 CheckThat! challenge. In: CLEF 2019 Working Notes, Working Notes of CLEF 2019 Conference and Labs of the Evaluation Forum (2019)
Fazio, L.K., Brashier, N.M., Payne, B.K., Marsh, E.J.: Knowledge does not protect against illusory truth. J. Exp. Psychol.: Gen. 144(5), 993 (2015)
Fullfact.org: Effectiveness - full fact (2019). https://fullfact.org/about/effectiveness/. Accessed 6 Dec 2019
Gencheva, P., Nakov, P., Mà rquez, L., Barrón-Cedeño, A., Koychev, I.: A context-aware approach for detecting worth-checking claims in political debates. In: Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pp. 267–276 (2017)
Graves, L.: Deciding what’s true: fact-checking journalism and the new ecology of news. Ph.D. thesis, Columbia University (2013)
Graves, L., Nyhan, B., Reifler, J.: Why do journalists fact-check (2016). https://www.dartmouth.edu/~nyhan/journalist-fact-checking.pdf
Halliday, M.A., Horowitz, R., Samuels, S.J.: Comprehending Oral and Written Language. Emeral Group, Bingley (1987)
Hansen, C., Hansen, C., Simonsen, J., Lioma, C.: Neural weakly supervised fact check-worthiness detection with contrastive sampling-based ranking loss. In: 20th Working Notes of CLEF Conference and Labs of the Evaluation Forum, CLEF 2019 Conference and Labs of the Evaluation Forum, vol. 2380 (2019)
Hansen, C., Hansen, C., Simonsen, J.G., Lioma, C.: The Copenhagen team participation in the check-worthiness task of the competition of automatic identification and verification of claims in political debates of the CLEF-2018 CheckThat! Lab. In: CLEF (Working Notes) (2018)
Hasanain, M., Suwaileh, R., Elsayed, T., Barrón-Cedeno, A., Nakov, P.: Overview of the CLEF-2019 CheckThat! Lab on automatic identification and verification of claims. Task 2: evidence and factuality. In: CEUR Workshop Proceedings, Lugano, Switzerland (2019)
Hassan, N., Arslan, F., Li, C., Tremayne, M.: Toward automated fact-checking: detecting check-worthy factual claims by ClaimBuster. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1803–1812. ACM (2017)
Hassan, N., Li, C., Tremayne, M.: Detecting check-worthy factual claims in presidential debates. In: Proceedings of the 24th ACM International Conference on Information and Knowledge Management, pp. 1835–1838. ACM (2015)
Jimenez, D., Li, C.: An empirical study on identifying sentences with salient factual statements. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2018)
Kazemian, B., Hashemi, S.: Critical discourse analysis of Barack Obama’s 2012 speeches: views from systemic functional linguistics and rhetoric. Theory Pract. Lang. Stud. (TPLS) 4(6), 1178–1187 (2014)
Levy, R., Bilu, Y., Hershcovich, D., Aharoni, E., Slonim, N.: Context dependent claim detection. In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 1489–1500 (2014)
Nakov, P., et al.: Overview of the CLEF-2018 CheckThat! Lab on automatic identification and verification of political claims. In: Bellot, P., et al. (eds.) CLEF 2018. LNCS, vol. 11018, pp. 372–387. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98932-7_32
Patwari, A., Goldwasser, D., Bagchi, S.: Tathya: A multi-classifier system for detecting check-worthy statements in political debates. In: Proceedings of the 2017 ACM Conference on Information and Knowledge Management, pp. 2259–2262. ACM (2017)
Qian, Z., Li, P., Zhu, Q., Zhou, G., Luo, Z., Luo, W.: Speculation and negation scope detection via convolutional neural networks. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 815–825 (2016)
Scholl, S.G., Greifeneder, R., Bless, H.: When fluency signals truth: prior successful reliance on fluency moderates the impact of fluency on truth judgments. J. Behav. Decis. Mak. 27(3), 268–280 (2014)
Snopes.com: Transparancy (2019). https://www.snopes.com/transparency/. Accessed 6 Dec 2019
Stencel, M., Luther, J.: Reporter’s lab fact-checking tally tops 200. Duke University Reporter’s Lab (October 2019). https://reporterslab.org/category/fact-checking/#article-2551
Thorne, J., Vlachos, A., Cocarascu, O., Christodoulopoulos, C., Mittal, A.: The Fact Extraction and VERification (FEVER) shared task. In: Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pp. 1–9. Association for Computational Linguistics, Brussels (2018)
Truthorfiction.com: Our methodology and process (2019). https://www.truthorfiction.com/our-methodology-and-process/. Accessed 6 Dec 2019
Zubiaga, A., Aker, A., Bontcheva, K., Liakata, M., Procter, R.: Detection and resolution of rumours in social media: a survey. ACM Comput. Surv. (CSUR) 51(2), 32 (2018)
Zuo, C., Karakas, A., Banerjee, R.: A hybrid recognition system for check-worthy claims using heuristics and supervised learning. In: CLEF (Working Notes) (2018)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Allein, L., Moens, MF. (2020). Checkworthiness in Automatic Claim Detection Models: Definitions and Analysis of Datasets. In: van Duijn, M., Preuss, M., Spaiser, V., Takes, F., Verberne, S. (eds) Disinformation in Open Online Media. MISDOOM 2020. Lecture Notes in Computer Science(), vol 12259. Springer, Cham. https://doi.org/10.1007/978-3-030-61841-4_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-61841-4_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61840-7
Online ISBN: 978-3-030-61841-4
eBook Packages: Computer ScienceComputer Science (R0)