skip to main content
10.1145/3269206.3269288acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
short-paper

Re-ranking Web Search Results for Better Fact-Checking: A Preliminary Study

Published: 17 October 2018 Publication History

Abstract

Even though Web search engines play an important role in finding documents relevant to user queries, there is little to no attention given to how they perform in terms of usefulness for fact-checking claims. In this paper, we introduce a new research problem that addresses the ability of fact-checking systems to distinguish Web search results that are useful in discovering the veracity of claims from the ones that are not.We also propose a re-ranking method to improve ranking of search results for fact-checking. To evaluate our proposed method, we conducted a preliminary study for which we have developed a test collection that includes 22 claims and 20 manually-annotated Web search results for each. Our experiments show that the proposed method outperforms the baseline represented by the original ranking of search results. The contributions this improvement brings to real-world applications is two-fold: it will help human fact-checkers find useful documents for their task faster, and it will help automated fact-checking systems by pointing out which documents are useful and which are not.

References

[1]
David Corney, Dyaa Albakour, Miguel Martinez, and Samir Moussa. 2016. What do a Million News Articles Look like?. In Proceedings of the First International Workshop on Recent Trends in News Information Retrieval co-located with 38th European Conference on Information Retrieval (ECIR 2016), Padua, Italy, March 20, 2016. 42--47.
[2]
William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies. 1163--1168.
[3]
Georgi Karadzhov, Preslav Nakov, Lluis Marquez, Alberto Barron-Cedeno, and Ivan Koychev. 2017. Fully Automated Fact Checking Using External Sources. arXiv preprint arXiv:1710.00341 (2017).
[4]
Christina Lioma, Birger Larsen, Wei Lu, and Yong Huang. 2016. A study of factuality, objectivity and relevance: three desiderata in large-scale information retrieval?. In Proceedings of the 3rd IEEE/ACM International Conference on Big Data Computing, Applications and Technologies. ACM, 107--117.
[5]
Christina Lioma, Jakob Grue Simonsen, and Birger Larsen. 2017. Evaluation Measures for Relevance and Credibility in Ranked Lists. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval. ACM, 91--98.
[6]
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Association for Computational Linguistics (ACL) System Demonstrations . 55--60.
[7]
An T Nguyen1 Aditya Kharosekar Matthew and Lease1 Byron C Wallace. 2018. An Interpretable Joint Graphical Model for Fact-Checking from Crowds. (2018).
[8]
Ndapandula Nakashole and Tom M Mitchell. 2014. Language-aware truth assessment of fact candidates. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 1009--1019.
[9]
Nikolaos Pappas and Andrei Popescu-Belis. 2013. Sentiment Analysis of User Comments for One-Class Collaborative Filtering Over TED Talks. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval (SIGIR '13). 773--776.
[10]
William Yang Wang. 2017. "Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vol. 2. 422--426.

Cited By

View all
  • (2023)Designing and Evaluating Presentation Strategies for Fact-Checked ContentProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614841(751-761)Online publication date: 21-Oct-2023
  • (2023)Overview of the CLEF–2023 CheckThat! Lab on Checkworthiness, Subjectivity, Political Bias, Factuality, and Authority of News Articles and Their SourceExperimental IR Meets Multilinguality, Multimodality, and Interaction10.1007/978-3-031-42448-9_20(251-275)Online publication date: 11-Sep-2023
  • (2022)Overview of the CLEF–2022 CheckThat! Lab on Fighting the COVID-19 Infodemic and Fake News DetectionExperimental IR Meets Multilinguality, Multimodality, and Interaction10.1007/978-3-031-13643-6_29(495-520)Online publication date: 25-Aug-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge Management
October 2018
2362 pages
ISBN:9781450360142
DOI:10.1145/3269206
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. evaluation
  2. fake news
  3. learning to rank
  4. misinformation
  5. search engines
  6. test collection

Qualifiers

  • Short-paper

Funding Sources

Conference

CIKM '18
Sponsor:

Acceptance Rates

CIKM '18 Paper Acceptance Rate 147 of 826 submissions, 18%;
Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Designing and Evaluating Presentation Strategies for Fact-Checked ContentProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614841(751-761)Online publication date: 21-Oct-2023
  • (2023)Overview of the CLEF–2023 CheckThat! Lab on Checkworthiness, Subjectivity, Political Bias, Factuality, and Authority of News Articles and Their SourceExperimental IR Meets Multilinguality, Multimodality, and Interaction10.1007/978-3-031-42448-9_20(251-275)Online publication date: 11-Sep-2023
  • (2022)Overview of the CLEF–2022 CheckThat! Lab on Fighting the COVID-19 Infodemic and Fake News DetectionExperimental IR Meets Multilinguality, Multimodality, and Interaction10.1007/978-3-031-13643-6_29(495-520)Online publication date: 25-Aug-2022
  • (2022)Studying effectiveness of Web search for fact checkingJournal of the Association for Information Science and Technology10.1002/asi.2457773:5(738-751)Online publication date: 1-Apr-2022
  • (2019)Overview of the CLEF-2019 CheckThat! Lab: Automatic Identification and Verification of ClaimsExperimental IR Meets Multilinguality, Multimodality, and Interaction10.1007/978-3-030-28577-7_25(301-321)Online publication date: 3-Aug-2019
  • (2019)CheckThat! at CLEF 2019: Automatic Identification and Verification of ClaimsAdvances in Information Retrieval10.1007/978-3-030-15719-7_41(309-315)Online publication date: 7-Apr-2019

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media