skip to main content
10.1145/3672919.3672933acmotherconferencesArticle/Chapter ViewAbstractPublication PagescsaideConference Proceedingsconference-collections
research-article

Research on fake news detection based on CLIP multimodal mechanism

Published: 24 July 2024 Publication History

Abstract

Compared to text-only online fake news, combined graphic forms of fake news are more likely to gain people's trust. Most of these forms of fake news have been detected using multimodal feature fusion, with less attention being paid to the correlation between two modalities and the interaction within and between individual modalities. To address this problem, we propose a multimodal mechanism based on CLIP for fake news detection, referred to as CLIP-FND. First, the visual encoder and text encoder of the large-scale graphical pre-training model CLIP are used to unify and map the image and text data into the same feature space respectively. Then, the attention mechanism and the bilinear pooling method are used to fuse the text feature vector and the visual feature vector and input to the fake news detector for fake news detection. The experimental results show that the accuracy and F1 value of the model are much higher than the traditional multimodal fake news detection model on the publicly available dataset, which improves the fake news detection effect.

References

[1]
Shu, Kai, "Fake news detection on social media: A data mining perspective." ACM SIGKDD explorations newsletter 19.1 2017: 22-36.
[2]
Aldwairi, Monther, and Ali Alwahedi. "Detecting fake news in social media networks." Procedia Computer Science 141 2018: 215-222.
[3]
Kim, Antino, Patricia L. Moravec, and Alan R. Dennis. "Combating fake news on social media with source ratings: The effects of user and expert reputation ratings." Journal of Management Information Systems 36.3 2019: 931-968.
[4]
Shu, Kai, "Hierarchical propagation networks for fake news detection: Investigation and exploitation." Proceedings of the international AAAI conference on web and social media. Vol. 14. 2020.
[5]
Radford, Alec, "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.
[6]
Choi, Hyewon, "How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image." arXiv preprint arXiv:2204.05533 2022.
[7]
Castillo, Carlos, Marcelo Mendoza, and Barbara Poblete. "Information credibility on twitter." Proceedings of the 20th international conference on World wide web. 2011.
[8]
Qazvinian, Vahed, "Rumor has it: Identifying misinformation in microblogs." Proceedings of the 2011 conference on empirical methods in natural language processing. 2011.
[9]
Rashkin, Hannah, "Truth of varying shades: Analyzing language in fake news and political fact-checking." Proceedings of the 2017 conference on empirical methods in natural language processing. 2017.
[10]
Chen, Tong, "Call attention to rumors: Deep attention based recurrent neural networks for early rumor detection." Trends and Applications in Knowledge Discovery and Data Mining: PAKDD 2018 Workshops, BDASC, BDM, ML4Cyber, PAISI, DaMEMO, Melbourne, VIC, Australia, June 3, 2018, Revised Selected Papers 22. Springer International Publishing, 2018.
[11]
Zhang, Jiawei, Bowen Dong, and S. Yu Philip. "Deep diffusive neural network based fake news detection from heterogeneous social networks." 2019 IEEE international conference on big data (Big Data). IEEE, 2019.
[12]
Qi, Peng, "Exploiting multi-domain visual information for fake news detection." 2019 IEEE international conference on data mining (ICDM). IEEE, 2019.
[13]
Ma, Jing, Wei Gao, and Kam-Fai Wong. "Rumor detection on twitter with tree-structured recursive neural networks." Association for Computational Linguistics, 2018.
[14]
Jin, Zhiwei, "Multimodal fusion with recurrent neural networks for rumor detection on microblogs." Proceedings of the 25th ACM international conference on Multimedia. 2017.
[15]
Wang, Yaqing, "Eann: Event adversarial neural networks for multi-modal fake news detection." Proceedings of the 24th acm sigkdd international conference on knowledge discovery & data mining. 2018.
[16]
Khattar, Dhruv, "Mvae: Multimodal variational autoencoder for fake news detection." The world wide web conference. 2019.
[17]
Zhou, Xinyi, Jindi Wu, and Reza Zafarani. "Safe: similarity-aware multi-modal fake news detection 2020." Preprint. arXiv 200304981 2020: 2.
[18]
Song, Chenguang, "A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks." Information Processing & Management 58.1 2021: 102437.
[19]
Singhal, Shivangi, "Spotfake: A multi-modal framework for fake news detection." 2019 IEEE fifth international conference on multimedia big data (BigMM). IEEE, 2019.
[20]
Tuan, Nguyen Manh Duc, and Pham Quang Nhat Minh. "Multimodal fusion with BERT and attention mechanism for fake news detection." 2021 RIVF International Conference on Computing and Communication Technologies (RIVF). IEEE, 2021.
[21]
Vaswani, Ashish, "Attention is all you need." Advances in neural information processing systems 30 2017.
[22]
Devlin, Jacob, "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 2018.
[23]
Dosovitskiy, Alexey, "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 2020.
[24]
Gao, Peng, "Dynamic fusion with intra-and inter-modality attention flow for visual question answering." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.
[25]
Lin, Tsung-Yu, Aruni RoyChowdhury, and Subhransu Maji. "Bilinear CNN models for fine-grained visual recognition." Proceedings of the IEEE international conference on computer vision. 2015.
[26]
Boididou, Christina, "Verifying multimedia use at mediaeval 2015." MediaEval 2015. Vol. 1436. CEUR-WS, 2015.
[27]
Godbole, Shantanu, and Sunita Sarawagi. "Discriminative methods for multi-labeled classification." Advances in Knowledge Discovery and Data Mining: 8th Pacific-Asia Conference, PAKDD 2004, Sydney, Australia, May 26-28, 2004. Proceedings 8. Springer Berlin Heidelberg, 2004.

Index Terms

  1. Research on fake news detection based on CLIP multimodal mechanism

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    CSAIDE '24: Proceedings of the 2024 3rd International Conference on Cyber Security, Artificial Intelligence and Digital Economy
    March 2024
    676 pages
    ISBN:9798400718212
    DOI:10.1145/3672919
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 24 July 2024

    Permissions

    Request permissions for this article.

    Check for updates

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    CSAIDE 2024

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 61
      Total Downloads
    • Downloads (Last 12 months)61
    • Downloads (Last 6 weeks)7
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media