skip to main content
10.1145/3591106.3592250acmconferencesArticle/Chapter ViewAbstractPublication PagesicmrConference Proceedingsconference-collections
research-article

Graph Interactive Network with Adaptive Gradient for Multi-Modal Rumor Detection

Published: 12 June 2023 Publication History

Abstract

With more and more messages in the form of text and image being spread on the Internet, multi-modal rumor detection has become the focus of recent research. However, most of the existing methods simply concatenate or fuse image features with text features, which can not fully explore the interaction between modalities. Meanwhile, they ignore the convergence inconsistency problem between strong and weak modalities, that is, the dominant rumor text modality may inhibit the optimization of image modality. In this paper, we investigate multi-modal rumor detection from a novel perspective, and propose a Multi-modal Graph Interactive Network with Adaptive Gradient (MGIN-AG) to solve the problem of insufficient information mining within and between modalities, and alleviate the optimization imbalance. Specifically, we first construct fine-grained graph for each rumor text or image to explicitly capture the relation between text tokens or image patches in uni-modal. Then, the cross modal interaction graph between text and image is designed to implicitly mine the text-image interaction, especially focusing on the consistency and mutual enhancement between image patches and text tokens. Furthermore, we extract the embedded text in images as an important supplement to improve the performance of the model. Finally, a strategy of dynamically adjusting the model gradient is introduced to alleviate the under optimization problem of weak modalities in the multi-modal rumor detection task. Extensive experiments demonstrate the superiority of our model in comparison with the state-of-the-art baselines.

References

[1]
Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Journal of economic perspectives 31, 2 (2017), 211–36.
[2]
Tian Bian, Xi Xiao, Tingyang Xu, Peilin Zhao, Wenbing Huang, Yu Rong, and Junzhou Huang. 2020. Rumor detection on social media with bi-directional graph convolutional networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 549–556.
[3]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
[4]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
[5]
Yaqian Dun, Kefei Tu, Chen Chen, Chunyan Hou, and Xiaojie Yuan. 2021. Kan: Knowledge-aware attention network for fake news detection. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 81–89.
[6]
Z. He, C. Li, F. Zhou, and Y. Yang. 2021. Rumor Detection on Social Media with Event Augmentations. In SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval.
[7]
Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang, and Jiebo Luo. 2017. Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In Proceedings of the 25th ACM international conference on Multimedia. 795–816.
[8]
Nikhil Ketkar. 2017. Introduction to pytorch. In Deep learning with python. Springer, 195–208.
[9]
Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and Vasudeva Varma. 2019. Mvae: Multimodal variational autoencoder for fake news detection. In The world wide web conference. 2915–2921.
[10]
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).
[11]
Shimon Kogan, Tobias J Moskowitz, and Marina Niessner. 2019. Fake news: Evidence from financial markets. Available at SSRN 3237763 (2019).
[12]
Quanzhi Li, Qiong Zhang, and Luo Si. 2019. Rumor detection by exploiting user credibility information, attention and multi-task learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 1173–1179.
[13]
Yi-Ju Lu and Cheng-Te Li. 2020. GCAN: Graph-aware co-attention networks for explainable fake news detection on social media. arXiv preprint arXiv:2004.11648 (2020).
[14]
Jing Ma, Wei Gao, Prasenjit Mitra, Sejeong Kwon, Bernard J Jansen, Kam-Fai Wong, and Meeyoung Cha. 2016. Detecting rumors from microblogs with recurrent neural networks. (2016).
[15]
Jing Ma, Wei Gao, and Kam-Fai Wong. 2018. Detect rumor and stance jointly by neural multi-task learning. In Companion proceedings of the the web conference 2018. 585–593.
[16]
Jing Ma, Wei Gao, and Kam-Fai Wong. 2019. Detect rumors on twitter by promoting information campaigns with generative adversarial learning. In The World Wide Web Conference. 3049–3055.
[17]
Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. 2022. Balanced Multimodal Learning via On-the-fly Gradient Modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8238–8247.
[18]
Lahari Poddar, Wynne Hsu, Mong Li Lee, and Shruti Subramaniyam. 2018. Predicting stances in twitter conversations for detecting veracity of rumors: A neural approach. In 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 65–72.
[19]
Peng Qi, Juan Cao, Xirong Li, Huan Liu, Qiang Sheng, Xiaoyue Mi, Qin He, Yongbiao Lv, Chenyang Guo, and Yingchao Yu. 2021. Improving fake news detection by using an entity-enhanced framework to fuse diverse multimodal clues. In Proceedings of the 29th ACM International Conference on Multimedia. 1212–1220.
[20]
Shengsheng Qian, Jinguang Wang, Jun Hu, Quan Fang, and Changsheng Xu. 2021. Hierarchical multi-modal contextual attention network for fake news detection. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 153–162.
[21]
Kai Shu, Suhang Wang, and Huan Liu. 2017. Exploiting tri-relationship for fake news detection. arXiv preprint arXiv:1712.07709 8 (2017).
[22]
Mengzhu Sun, Xi Zhang, Jianqiang Ma, and Yazheng Liu. 2021. Inconsistency Matters: A Knowledge-guided Dual-inconsistency Network for Multi-modal Rumor Detection. In Findings of the Association for Computational Linguistics: EMNLP 2021. 1412–1423.
[23]
Tiening Sun, Zhong Qian, Sujun Dong, Peifeng Li, and Qiaoming Zhu. 2022. Rumor Detection on Social Media with Graph Adversarial Contrastive Learning. In Proceedings of the ACM Web Conference 2022. 2789–2797.
[24]
Tian Tian, Yudong Liu, Xiaoyu Yang, Yuefei Lyu, Xi Zhang, and Binxing Fang. 2020. QSAN: A quantum-probability based signed attention network for explainable false information detection. In Proceedings of the 29th ACM international conference on information & knowledge management. 1445–1454.
[25]
NGUYEN VAN HA, K Sugiyama, P Nakov, and MY Kan. 2020. FANG: Leveraging Social Context for Fake News Detection Using Graph Representation. (2020).
[26]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[27]
Weiyao Wang, Du Tran, and Matt Feiszli. 2020. What makes training multi-modal classification networks hard?. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12695–12705.
[28]
Yaqing Wang, Fenglong Ma, Zhiwei Jin, Ye Yuan, Guangxu Xun, Kishlay Jha, Lu Su, and Jing Gao. 2018. Eann: Event adversarial neural networks for multi-modal fake news detection. In Proceedings of the 24th acm sigkdd international conference on knowledge discovery & data mining. 849–857.
[29]
Youze Wang, Shengsheng Qian, Jun Hu, Quan Fang, and Changsheng Xu. 2020. Fake news detection via knowledge-driven multimodal graph convolutional networks. In Proceedings of the 2020 International Conference on Multimedia Retrieval. 540–547.
[30]
Penghui Wei, Nan Xu, and Wenji Mao. 2019. Modeling conversation structure and temporal dynamics for jointly predicting rumor stance and veracity. arXiv preprint arXiv:1909.08211 (2019).
[31]
Lianwei Wu, Yuan Rao, Yongqiang Zhao, Hao Liang, and Ambreen Nazir. 2020. DTCA: Decision tree-based co-attention networks for explainable claim verification. arXiv preprint arXiv:2004.13455 (2020).
[32]
Yang Wu, Pengwei Zhan, Yunjian Zhang, Liming Wang, and Zhen Xu. 2021. Multimodal fusion with co-attention networks for fake news detection. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. 2560–2569.
[33]
Rui Xia, Kaizhou Xuan, and Jianfei Yu. 2020. A State-independent and Time-evolving Network for Early Rumor Detection in Social Media. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020. 9042–9051.
[34]
Feng Yu, Qiang Liu, Shu Wu, Liang Wang, Tieniu Tan, 2017. A Convolutional Approach for Misinformation Identification. In IJCAI. 3901–3907.
[35]
Chunyuan Yuan, Qianwen Ma, Wei Zhou, Jizhong Han, and Songlin Hu. 2019. Jointly embedding the local and global relations of heterogeneous graph for rumor detection. In 2019 IEEE International Conference on Data Mining (ICDM). IEEE, 796–805.
[36]
Huaiwen Zhang, Shengsheng Qian, Quan Fang, and Changsheng Xu. 2021. Multi-modal meta multi-task learning for social media rumor detection. IEEE Transactions on Multimedia 24 (2021), 1449–1459.
[37]
Jiaqi Zheng, Xi Zhang, Sanchuan Guo, Quan Wang, Wenyu Zang, and Yongdong Zhang. 2022. MFAN: Multi-modal Feature-enhanced Attention Networks for Rumor Detection. In IJCAI. 2413–2419.

Cited By

View all
  • (2025)A unified framework for multi-modal rumor detection via multi-level dynamic interaction with evolving stancesInformation Processing & Management10.1016/j.ipm.2025.10406662:3(104066)Online publication date: May-2025
  • (2024)Fingerprinting in EEG Model IP Protection Using Diffusion ModelProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658057(120-128)Online publication date: 30-May-2024
  • (2024)MSynFD: Multi-hop Syntax Aware Fake News DetectionProceedings of the ACM Web Conference 202410.1145/3589334.3645468(4128-4137)Online publication date: 13-May-2024
  • Show More Cited By

Index Terms

  1. Graph Interactive Network with Adaptive Gradient for Multi-Modal Rumor Detection

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        ICMR '23: Proceedings of the 2023 ACM International Conference on Multimedia Retrieval
        June 2023
        694 pages
        ISBN:9798400701788
        DOI:10.1145/3591106
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 12 June 2023

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. fake news detection
        2. graph neural networks
        3. multi-modal fusion
        4. rumor detection
        5. social networks

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Funding Sources

        Conference

        ICMR '23
        Sponsor:

        Acceptance Rates

        Overall Acceptance Rate 254 of 830 submissions, 31%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)200
        • Downloads (Last 6 weeks)11
        Reflects downloads up to 10 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2025)A unified framework for multi-modal rumor detection via multi-level dynamic interaction with evolving stancesInformation Processing & Management10.1016/j.ipm.2025.10406662:3(104066)Online publication date: May-2025
        • (2024)Fingerprinting in EEG Model IP Protection Using Diffusion ModelProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658057(120-128)Online publication date: 30-May-2024
        • (2024)MSynFD: Multi-hop Syntax Aware Fake News DetectionProceedings of the ACM Web Conference 202410.1145/3589334.3645468(4128-4137)Online publication date: 13-May-2024
        • (2024)Graph Attention Network with Cross-Modal Interaction for Rumor Detection2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650542(1-8)Online publication date: 30-Jun-2024
        • (2024)PVCG: Prompt-Based Vision-Aware Classification and Generation for Multi-Modal Rumor DetectionICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10447285(11036-11040)Online publication date: 14-Apr-2024
        • (2024)DSMMInformation Processing and Management: an International Journal10.1016/j.ipm.2023.10352861:1Online publication date: 1-Feb-2024
        • (2024)Improving Multimodal Rumor Detection via Dynamic Graph ModelingPattern Recognition10.1007/978-3-031-78456-9_16(242-258)Online publication date: 3-Dec-2024
        • (2023)Contrastive Learning for Rumor Detection via Fitting Beta Mixture ModelProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615138(4160-4164)Online publication date: 21-Oct-2023
        • (2023)Multimodal Rumor Detection with Causal Graph Attention Network2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS60453.2023.00203(1429-1436)Online publication date: 17-Dec-2023

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media