Abstract
The proliferation of misinformation demands the development of automated fact-checking systems. Large language models (LLMs), are increasingly being used for academic, legal, and journalistic content writing. This underscores the critical importance of LLMs in distinguishing between factual accuracy and inaccuracy. Hence, understanding the capacities and limitations of LLMs in fact-checking tasks is essential for their usage in information space. While previous research showed the potential of LLMs in fact-checking English news headlines, the extent to which LLMs work well in other languages are mostly unexplored. In this paper, using data from a local fact-checking website, we investigate the performance of close- and open-source LLMs in fact-checking Farsi news headlines. Our results show that in none of the model combinations, the fact-checking accuracy of LLMs exceeds 55%, which is pretty low compared to results reported for English news. While fine-tuning shows promising results for performance gain, and should be explored further in future research, our results underscore the weakness of LLMs in low-resource languages such as Farsi, even when fine-tuned.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Achiam, J., et al.: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
Alizadeh, M., et al.: Open-source large language models outperform crowd workers and approach ChatGPT in text-annotation tasks. arXiv preprint arXiv:2307.02179 (2023)
Alizadeh, M., Shapiro, J.N., Buntain, C., Tucker, J.A.: Content-based features predict social media influence operations. Sci. Adv. 6(30), eabb5824 (2020)
Brown, T., et al.: Language models are few-shot learners. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)
Caramancion, K.M.: News verifiers showdown: a comparative performance evaluation of ChatGPT 3.5, ChatGPT 4.0, Bing AI, and bard in news fact-checking. arXiv preprint arXiv:2306.17176 (2023)
Choi, E.C., Ferrara, E.: Automated claim matching with large language models: empowering fact-checkers in the fight against misinformation. arXiv preprint arXiv:2310.09223 (2023)
Das, A., Liu, H., Kovatchev, V., Lease, M.: The state of human-centered NLP technology for fact-checking. Inf. Process. Manag. 60(2), 103219 (2023)
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: QloRA: efficient finetuning of quantized LLMs (2023)
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: QLoRA: efficient finetuning of quantized LLMs. In: Advances in Neural Information Processing Systems, vol. 36 (2024)
DeVerna, M.R., Yan, H.Y., Yang, K.C., Menczer, F.: Artificial intelligence is ineffective and potentially harmful for fact checking. arXiv preprint arXiv:2308.10800 (2023)
Gorrell, G., Bontcheva, K., Derczynski, L., Kochkina, E., Liakata, M., Zubiaga, A.: RumourEval 2019: determining rumour veracity and support for rumours. arXiv preprint arXiv:1809.06683 (2018)
Hoes, E., Altay, S., Bermeo, J.: Leveraging ChatGPT for efficient fact-checking (2023)
Hu, E.J., et al.: LoRA: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
Köhler, J., et al.: Overview of the CLEF-2022 CheckThat! Lab: task 3 on fake news detection. In: CLEF (Working Notes), pp. 404–421 (2022)
Köpf, A., et al.: OpenAssistant conversations – democratizing large language model alignment (2023)
Menczer, F., Crandall, D., Ahn, Y.Y., Kapadia, A.: Addressing the harms of AI-generated inauthentic content. Nat. Mach. Intell. 5(7), 679–680 (2023)
Nakov, P., et al.: Overview of the CLEF-2022 CheckThat! Lab task 1 on identifying relevant claims in tweets. In: 2022 Conference and Labs of the Evaluation Forum, CLEF 2022, pp. 368–392. CEUR Workshop Proceedings (CEUR-WS. org) (2022)
Nakov, P., et al.: Overview of the CLEF–2022 CheckThat! Lab on fighting the covid-19 infodemic and fake news detection. In: Barrón-Cedeño, A., et al. (eds.) CLEF 2022. LNCS, vol. 13390, pp. 495–520. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-13643-6_29
Nakov, P., Da San Martino, G., Alam, F., Shaar, S., Mubarak, H., Babulkov, N.: Overview of the CLEF-2022 CheckThat! Lab task 2 on detecting previously fact-checked claims (2022)
Pierri, F., DeVerna, M.R., Yang, K.C., Axelrod, D., Bryden, J., Menczer, F.: One year of covid-19 vaccine misinformation on twitter: longitudinal study (preprint) (2022)
Porter, E., Wood, T.J.: The global effectiveness of fact-checking: Evidence from simultaneous experiments in Argentina, Nigeria, South Africa, and the United Kingdom. Proc. Natl. Acad. Sci. 118(37), e2104235118 (2021)
Quelle, D., Bovet, A.: The perils and promises of fact-checking with large language models. Front. Artif. Intell. 7, 1341697 (2024)
Sawiński, M., et al.: OpenFact at CheckThat! 2023: head-to-head GPT vs. BERT-a comparative study of transformers language models for the detection of check-worthy claims. In: Working Notes of CLEF (2023)
Thorne, J., Vlachos, A., Christodoulopoulos, C., Mittal, A.: Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355 (2018)
Touvron, H., et al.: Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
Wei, J., et al.: Finetuned language models are zero-shot learners (2022)
von Werra, L., et al.: TRL: transformer reinforcement learning (2020). https://github.com/huggingface/trl
Yang, C., et al.: Large language models as optimizers. arXiv preprint arXiv:2309.03409 (2023)
Yang, K.C., Singh, D., Menczer, F.: Characteristics and prevalence of fake social media profiles with AI-generated faces. arXiv preprint arXiv:2401.02627 (2024)
Acknowledgments
We thank Factyar team (an Iranian NGO fact-checking website) for providing easy acess to their fact-checked data.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors declare no conflict of interest.
Appendix
Appendix
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Dehghani, S. et al. (2024). Leveraging Large Language Models for Fact-Checking Farsi News Headlines. In: Preuss, M., Leszkiewicz, A., Boucher, JC., Fridman, O., Stampe, L. (eds) Disinformation in Open Online Media. MISDOOM 2024. Lecture Notes in Computer Science, vol 15175. Springer, Cham. https://doi.org/10.1007/978-3-031-71210-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-71210-4_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-71209-8
Online ISBN: 978-3-031-71210-4
eBook Packages: Computer ScienceComputer Science (R0)