Cited By
View all- Li DZhu JFu XGuo XLiu YYang GLiu JZha Z(2024)Noise-Assisted Prompt Learning for Image Forgery Detection and LocalizationComputer Vision – ECCV 202410.1007/978-3-031-73247-8_2(18-36)Online publication date: 1-Nov-2024
ESCNet: Entity-enhanced and Stance Checking Network for Multi-modal Fact-Checking
Authors: 
Fanrui Zhang, 
Jiawei Liu, Jingyi Xie, Qiang Zhang, Yongchao Xu, Zheng-Jun ZhaAuthors Info & Claims
Fanrui Zhang, Jiawei Liu, Jingyi Xie, Qiang Zhang, Yongchao Xu, Zheng-Jun ZhaAuthors Info & ClaimsPages 2429 - 2440
Abstract
Recently, misinformation incorporating both texts and images has been disseminated more effectively than those containing text alone on social media, raising significant concerns for multi-modal fact-checking. Existing research makes contributions to multi-modal feature extraction and interaction, but fails to fully enhance the valuable semantic representations or excavate the intricate entity information. Besides, existing multi-modal fact-checking datasets are primarily focused on English and merely concentrate on a single type of misinformation, thereby neglecting a comprehensive summary and coverage of various types of misinformation. Taking these factors into account, we construct the first large-scale Chinese Multi-modal Fact-Checking (CMFC) dataset which encompasses 46,000 claims. The CMFC covers all types of misinformation for fact-checking and is divided into two sub-datasets, Collected Chinese Multi-modal Fact-Checking (CCMF) and Synthetic Chinese Multi-modal Fact-Checking (SCMF). To establish baseline performance, we propose a novel Entity-enhanced and Stance Checking Network (ESCNet), which includes Multi-modal Feature Extraction Module, Stance Transformer, and Entity-enhanced Encoder. The ESCNet jointly models stance semantic reasoning features and knowledge-enhanced entity pair features, in order to simultaneously learn effective semantic-level and knowledge-level claim representations. Our work offers the first step and establishes a benchmark for evidence-based, multi-type, multi-modal fact-checking.
Supplemental Material
MP4 File
Supplemental video
- Download
- 15.79 MB
References
[1]
Sahar Abdelnabi, Rakibul Hasan, and Mario Fritz. 2022. Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14940--14949.
[2]
Shruti Agarwal, Hany Farid, Yuming Gu, Mingming He, Koki Nagano, and Hao Li. 2019. Protecting World Leaders Against Deep Fakes. In Proceedings of the IEEE Conference Workshops on Computer Vision and Pattern Recognition. 38--48.
[3]
Rami Aly, Zhijiang Guo, M. Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, O. Cocarascu, and Arpit Mittal. 2021. FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information. In NeurIPS.
[4]
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of data. 1247--1250.
[5]
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. Proceedings of the Advances in Neural Information Processing Systems, Vol. 26 (2013).
[6]
Xiaojun Chen, Shengbin Jia, and Yang Xiang. 2020. A review: Knowledge reasoning over knowledge graph. Expert Systems with Applications, Vol. 141 (2020), 112948.
[7]
Zhendong Chen, Siu Cheung Hui, Fuzhen Zhuang, Lejian Liao, Fei Li, Meihuizi Jia, and Jiaqi Li. 2022. EvidenceNet: Evidence Fusion Network for Fact Verification. In Proceedings of the ACM Web Conference 2022. 2636--2645.
[8]
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for chinese bert. IEEE/ACM Transactions on Audio, Speech, and Language Processing, Vol. 29 (2021), 3504--3514.
[9]
Tri Setya Dermawan. 2019. Comparison of dijkstra dan floyd-warshall algorithm to determine the best route of train. International Journal on Informatics for Development, Vol. 7, 2 (2019), 54--58.
[10]
Abhishek Dhankar, Osmar R. Zaïane, and Francois Bolduc. 2022. UofA-Truth at Factify 2022 : Transformer And Transfer Learning Based Multi-Modal Fact-Checking. arxiv: 2203.07990 [cs.MM]
[11]
Wei-Wei Du, Hong-Wei Wu, Wei-Yao Wang, and Wen-Chih Peng. 2023. Team Triple-Check at Factify 2: Parameter-Efficient Large Foundation Models with Feature Representations for Multi-Modal Fact Verification. arXiv preprint arXiv:2302 (2023).
[12]
Mohamed H Gad-Elrab, Daria Stepanova, Jacopo Urbani, and Gerhard Weikum. 2019. Exfakt: A framework for explaining facts over knowledge graphs and text. In Proceedings of the twelfth ACM international conference on web search and data mining. 87--95.
[13]
Jie Gao, Hella-Franziska Hoffmann, Stylianos Oikonomou, David Kiskovski, and Anil Bandhakavi. 2021. Logically at Factify 2022: Multimodal Fact Verfication. ArXiv, Vol. abs/2112.09253 (2021).
[14]
Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, Vol. 10 (2022), 178--206.
[15]
Stefan Hougardy. 2010. The Floyd--Warshall algorithm on graphs with negative cycles. Inform. Process. Lett., Vol. 110, 8--9 (2010), 279--281.
[16]
Xuming Hu, Zhijiang Guo, Junzhe Chen, Lijie Wen, and Philip S Yu. 2023. Mr2: A benchmark for multimodal retrieval-augmented rumor detection in social media. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2901--2912.
[17]
Xuming Hu, Zhijiang Guo, Guanyu Wu, Aiwei Liu, Lijie Wen, and Philip S Yu. 2022. CHEF: A Pilot Chinese Dataset for Evidence-Based Fact-Checking. arXiv preprint arXiv:2206.11863 (2022).
[18]
Ayush Jaiswal, Ekraam Sabir, Wael AbdAlmageed, and Premkumar Natarajan. 2017. Multimedia semantic integrity assessment using joint embedding of images and text. In Proceedings of the 25th ACM international conference on Multimedia. 1465--1471.
[19]
Jiseong Kim and Key-sun Choi. 2020. Unsupervised fact checking by counter-weighted positive and negative evidential paths in a knowledge graph. In Proceedings of the 28th international conference on computational linguistics. 1677--1686.
[20]
Neema Kotonya and Francesca Toni. 2020. Explainable Automated Fact-Checking for Public Health Claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 7740--7754. https://doi.org/10.18653/v1/2020.emnlp-main.623
[21]
Dong Li, Jiaying Zhu, Menglu Wang, Jiawei Liu, Xueyang Fu, and Zheng-Jun Zha. 2023. Edge-Aware Regional Message Passing Controller for Image Forgery Localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8222--8232.
[22]
Grace Luo, Trevor Darrell, and Anna Rohrbach. 2021. Newsclippings: Automatic generation of out-of-context multimodal media. arXiv preprint arXiv:2104.05893 (2021).
[23]
Shreyash Mishra, S Suryavardan, Amrit Bhaskar, Parul Chopra, Aishwarya Reganti, Parth Patwa, Amitava Das, Tanmoy Chakraborty, Amit Sheth, Asif Ekbal, et al. 2022. Factify: A multi-modal fact verification dataset. In Proceedings of the First Workshop on Multimodal Fact-Checking and Hate Speech Detection (DE-FACTIFY).
[24]
Parth Patwa, Shreyash Mishra, S Suryavardan, Amrit Bhaskar, Parul Chopra, Aishwarya N. Reganti, Amitava Das, Tanmoy Chakraborty, A. Sheth, Asif Ekbal, and Chaitanya Ahuja. 2022. Benchmarking Multi-Modal Entailment for Fact Verification (short paper). In DE-FACTIFY@AAAI.
[25]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748--8763.
[26]
Ekraam Sabir, Wael AbdAlmageed, Yue Wu, and Prem Natarajan. 2018. Deep multimodal image-repurposing detection. In Proceedings of the 26th ACM international conference on Multimedia. 1337--1345.
[27]
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv, Vol. abs/1910.01108 (2019).
[28]
Karishma Sharma, Emilio Ferrara, and Yan Liu. 2022. Construction of Large-Scale Misinformation Labeled Datasets from Social Media Discourse using Label Refinement. In Proceedings of the ACM Web Conference 2022. 3755--3764.
[29]
Mengzhu Sun, Xi Zhang, Jianqiang Ma, Sihong Xie, Yazheng Liu, and S Yu Philip. 2023. Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection. IEEE Transactions on Knowledge and Data Engineering (2023).
[30]
S Suryavardan, Shreyash Mishra, Megha Chakraborty, Parth Patwa, Anku Rani, Aman Chadha, Aishwarya Reganti, Amitava Das, Amit Sheth, Manoj Chinnakotla, et al. 2023 a. Findings of factify 2: multimodal fake news detection. arXiv preprint arXiv:2307.10475 (2023).
[31]
S Suryavardan, Shreyash Mishra, Parth Patwa, Megha Chakraborty, Anku Rani, Aishwarya Reganti, Aman Chadha, Amitava Das, Amit Sheth, Manoj Chinnakotla, et al. 2023 b. Factify 2: A multimodal fake news and satire news dataset. arXiv preprint arXiv:2304.03897 (2023).
[32]
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355 (2018).
[33]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, Vol. 30 (2017).
[34]
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or Fiction: Verifying Scientific Claims. In Proc. of EMNLP. Association for Computational Linguistics, Online, 7534--7550. https://doi.org/10.18653/v1/2020.emnlp-main.609
[35]
Weizhi Xu, Junfei Wu, Qiang Liu, Shu Wu, and Liang Wang. 2022. Evidence-aware fake news detection with graph neural networks. In Proceedings of the ACM Web Conference 2022. 2501--2510.
[36]
Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cho, and Lifu Huang. 2022. End-to-end multimodal fact-checking and explanation generation: A challenging dataset and models. arXiv preprint arXiv:2205.12487 (2022).
[37]
Fanrui Zhang. 2023. Team Zhang at Factify 2: Unimodal Feature-Enhanced and Cross-Modal Correlation Learning for Multi-Modal Fact Verification. Technical Report. EasyChair.
[38]
Fanrui Zhang, Jiawei Liu, Qiang Zhang, Esther Sun, Jingyi Xie, and Zheng-Jun Zha. 2023 a. ECENet: Explainable and Context-Enhanced Network for Muti-modal Fact verification. In Proceedings of the 31st ACM International Conference on Multimedia. 1231--1240.
[39]
Qiang Zhang, Jiawei Liu, Fanrui Zhang, Jingyi Xie, and Zheng-Jun Zha. 2023 b. Hierarchical Semantic Enhancement Network for Multimodal Fake News Detection. In Proceedings of the 31st ACM International Conference on Multimedia. 3424--3433.
[40]
Yinuo Zhang, Zhulin Tao, Xi Wang, and Tongyue Wang. 2023 c. INO at Factify 2: Structure Coherence based Multi-Modal Fact Verification. arXiv preprint arXiv:2303.01510 (2023). io
Index Terms
- ESCNet: Entity-enhanced and Stance Checking Network for Multi-modal Fact-Checking
Recommendations
Fact-checking Effect on Viral Hoaxes: A Model of Misinformation Spread in Social Networks
WWW '15 Companion: Proceedings of the 24th International Conference on World Wide Webspread of misinformation, rumors and hoaxes. The goal of this work is to introduce a simple modeling framework to study the diffusion of hoaxes and in particular how the availability of debunking information may contain their diffusion. As traditionally ...
Linguistic Signals under Misinformation and Fact-Checking: Evidence from User Comments on Social Media
Misinformation and fact-checking are opposite forces in the news environment: the former creates inaccuracies to mislead people, while the latter provides evidence to rebut the former. These news articles are often posted on social media and attract user ...
The Fact-Checking Observatory: Reporting the Co-Spread of Misinformation and Fact-checks on Social Media
HT '23: Proceedings of the 34th ACM Conference on Hypertext and Social MediaIn the context of the Covid-19 pandemic and the Russian invasion of Ukraine, tracking how misinformation and fact-checks spread on social media is key for understanding where fact-checking efforts need to be focused and what demographics are most likely ...
Comments
Information & Contributors
Information
Published In
May 2024
4826 pages
ISBN:9798400701719
DOI:10.1145/3589334
- General Chairs:
- Tat-Seng Chua,
- Chong-Wah Ngo,
- Proceedings Chair:
- Roy Ka-Wei Lee,
- Program Chairs:
- Ravi Kumar,
- Hady W. Lauw
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Natural Science Foundation of China (NSFC) under Grants
Conference
WWW '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,899 of 8,196 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 372Total Downloads
- Downloads (Last 12 months)372
- Downloads (Last 6 weeks)56
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
Cited By
View all- Li DZhu JFu XGuo XLiu YYang GLiu JZha Z(2024)Noise-Assisted Prompt Learning for Image Forgery Detection and LocalizationComputer Vision – ECCV 202410.1007/978-3-031-73247-8_2(18-36)Online publication date: 1-Nov-2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in