Abstract
Multi-modal Fake News Detection (MFND), which aims to identify fake news by integrating texts and attached images, has attracted considerable attention in recent years. Existing works on MNFD have made a great progress by enhancing text-only fake news detection with visual information. However, most prior efforts focus on conducting multi-modal fusion yet largely ignore the significance of multi-modal representation, which is insufficient to explore various semantic interactions between images and texts. In this paper, we propose an instance-guided multi-modal graph fusion method by jointly modeling the intra- and inter-modality relationships between image and text. Specifically, considering that the content of multi-media news is always narrated around instances, we extract instance-level features of images to represent visual contents. After that, we construct a unified graph to enhance the multi-modal representation for improving fake news detection. In addition, we utilize multiple fusion layers to learn the graph embeddings, which is able to capture the intra-modality relationship within each modality and the inter-modality relationship between textual and visual instances simultaneously. Finally, we devise a fake news detector with hierarchical multi-modal representations to identify the fake news. Experimentation on two benchmark datasets demonstrates the superiority of our model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
If no noun is detected for a sentence, all textual and visual nodes are connected with fully connection.
References
Baltrusaitis, T., Ahuja, C., Morency, L.: Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41(2), 423–443 (2019)
He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. In: ICCV 2017, pp. 2980–2988 (2017)
Huang, Z., et al.: Audio-oriented multimodal machine comprehension via dynamic inter-and intra-modality attention. In: AAAI 2021, pp. 13098–13106 (2021)
Jin, Z., Cao, J., Guo, H., Zhang, Y., Luo, J.: Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: MM 2017, pp. 795–816 (2017)
Jin, Z., Cao, J., Zhang, Y., Zhou, J., Tian, Q.: Novel visual and statistical image features for microblogs news verification. IEEE Trans. Multimedia 19(3), 598–608 (2017)
Kenton, J.D.M.W.C., Toutanova, L.K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT 2019, pp. 4171–4186 (2019)
Khattar, D., Goud, J.S., Gupta, M., Varma, V.: MVAE: multimodal variational autoencoder for fake news detection. In: WWW 2019, pp. 2915–2921 (2019)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR 2015 (2015)
Li, P., Sun, X., Yu, H., Tian, Y., Yao, F., Xu, G.: Entity-oriented multi-modal alignment and fusion network for fake news detection. IEEE Trans. Multimedia (2021)
Lin, T., et al.: Microsoft COCO: common objects in context. CoRR abs/1405.0312 (2014)
Maigrot, C., Claveau, V., Kijak, E., Sicre, R.: Mediaeval 2016: A multimodal system for the verifying multimedia use task. In: MediaEval 2016: Verfiying Multimedia Use task (2016)
Qi, P., et al.: Improving fake news detection by using an entity-enhanced framework to fuse diverse multimodal clues. In: MM 2021, pp. 1212–1220 (2021)
Vaswani, A., et al.: Attention is all you need. In: NeurlPIS 2017, pp. 6000–6010 (2017)
Wang, Y., et al.: Eann: event adversarial neural networks for multi-modal fake news detection. In: SIGKDD 2018, pp. 849–857 (2018)
Wu, Y., Zhan, P., Zhang, Y., Wang, L., Xu, Z.: Multimodal fusion with co-attention networks for fake news detection. In: ACL-IJCNLP 2021, pp. 2560–2569 (2021)
Yin, Y., et al.: A novel graph-based multi-modal fusion encoder for neural machine translation. In: ACL 2020, pp. 3025–3035 (2020)
Yu, F., Liu, Q., Wu, S., Wang, L., Tan, T.: A convolutional approach for misinformation identification. In: IJCAI 2017, pp. 3901–3907 (2017)
Zhou, X., Wu, J., Zafarani, R.: \(\sf SAFE\): similarity-aware multi-modal fake news detection. In: PAKDD 2020, pp. 354–367 (2020)
Zhou, X., Zafarani, R.: A survey of fake news: fundamental theories, detection methods, and opportunities. ACM Comput. Surv. 53(5), 1–40 (2020)
Acknowledgement
The authors would like to thank the anonymous reviewers for their constructive comments. This research was supported by the National Natural Science Foundation of China (No. 61976247).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, J., Yang, Y., Liu, K., Xie, P., Liu, X. (2022). Instance-Guided Multi-modal Fake News Detection with Dynamic Intra- and Inter-modality Fusion. In: Gama, J., Li, T., Yu, Y., Chen, E., Zheng, Y., Teng, F. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2022. Lecture Notes in Computer Science(), vol 13280. Springer, Cham. https://doi.org/10.1007/978-3-031-05933-9_40
Download citation
DOI: https://doi.org/10.1007/978-3-031-05933-9_40
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-05932-2
Online ISBN: 978-3-031-05933-9
eBook Packages: Computer ScienceComputer Science (R0)