Skip to main content

Advertisement

Springer Nature Link
Log in

MCAN: multimodal cross-aware network for fake news detection by extracting semantic-physical feature consistency

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Social platforms are vital for information dissemination but also contribute to the spread of fake news, causing confusion and misinformation. To combat this, advancements in detection technology are crucial, particularly for posts that combine text and images, as they often present misleading information. However, current research often overlooks the extraction of key features from both modalities, missing critical elements like writing styles and image manipulations, hampering detection accuracy. In response, this work introduces the MCAN (Multimodal Cross-Aware Network), which freezes the parameters of BERT and ResNet50 to extract semantic features from text and images. It includes a text vocabulary network to analyze writing style differences and employs error level analysis to detect image manipulations. By integrating these features through a flexible multimodal fusion subnetwork with Bimodal Cross-Attention Blocks, MCAN effectively identifies fake news. Experimental results on two popular datasets demonstrate that MCAN outperforms existing baseline models in predictive accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

The program can be found on GitHub: https://github.com/yaozengzhang/MCAN.git.

References

  1. Allcott H, Gentzkow M (2017) Social media and fake news in the 2016 election. J Econ Perspect 31(2):211–236

    Article  Google Scholar 

  2. Meel P, Vishwakarma DK (2020) Fake news, rumor, information pollution in social media and web: a contemporary survey of state-of-the-arts, challenges and opportunities. Expert Syst Appl 153:112986

    Article  Google Scholar 

  3. Jin Z, Cao J, Zhang Y et al (2016) Novel visual and statistical image features for microblogs news verification. IEEE Trans Multimedia 19(3):598–608

    Article  Google Scholar 

  4. Sadeghi F, Bidgoly AJ, Amirkhani H (2022) Fake news detection on social media using a natural language inference approach. Multimedia Tools Appl 81(23):33801–33821

    Article  Google Scholar 

  5. Mallik A, Kumar S (2024) Word2Vec and LSTM based deep learning technique for context-free fake news detection. Multimedia Tools Appl 83(1):919–940

    Article  Google Scholar 

  6. Ma J, Gao W, Mitra P, et al. (2016) Detecting rumors from microblogs with recurrent neural networks. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI 2016). 3818–3824.

  7. Huang Z, Xiao X, Cao X (Eds.) (2017) Databases Theory and Applications: 28th Australasian Database Conference, ADC 2017; Brisbane, QLD, Australia, September 25–28, 2017; Proceedings, Springer

  8. Cui L, Wang S, Lee D (2019) SAME: sentiment-aware multi-modal embedding for detecting fake news. In: Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining. New York, NY, USA: Association for Computing Machinery, pp. 41–48.

  9. Zeng J, Zhang Y, Ma X (2021) Fake news detection for epidemic emergencies via deep correlations between text and images. Sustain Cities Soc 66:102652

    Article  Google Scholar 

  10. Giachanou A, Zhang G, Rosso P (2020) Multimodal multi-image fake news detection. In: 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA). pp 647–654

  11. Qian S, Wang J, Hu J, et al. (2021) Hierarchical multi-modal contextual attention network for fake news detection. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, NY, USA: Association for Computing Machinery, pp 153–162

  12. Ma J, Gao W, Wong K-F (2019) Detect rumors on twitter by promoting information campaigns with generative adversarial learning. In: The World Wide Web Conference. New York, NY, USA: Association for Computing Machinery, pp 3049–3055

  13. Xue J, Wang Y, Tian Y et al (2021) Detecting fake news by exploring the consistency of multimodal data. Inf Process Manage 58(5):102610

    Article  Google Scholar 

  14. Alkhodair SA, Ding SHH, Fung BCM et al (2020) Detecting breaking news rumors of emerging topics in social media. Inf Process Manage 57(2):102018

    Article  Google Scholar 

  15. Zhang X, Cao J, Li X, et al. (2021) Mining dual emotion for fake news detection. In: Proceedings of the Web Conference 2021. New York, NY, USA: Association for Computing Machinery, pp 3465–3476

  16. Yang Z, Wang C, Zhang F, et al. (2015) Emerging rumor identification for social media with hot topic detection. In: 2015 12th Web Information System and Application Conference (WISA). pp 53–58

  17. Rubin VL, Chen Y, Conroy NK (2015) Deception detection for news: three types of fakes. Proc Associat Inf Sci Technol 52(1):1–4

    Article  Google Scholar 

  18. Ruchansky N, Seo S, Liu Y (2021) CSI: a hybrid deep model for fake news detection. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. New York, NY, USA: Association for Computing Machinery, pp 797–806

  19. Zhang C, Gupta A, Kauten C et al (2019) Detecting fake news for reducing misinformation risks using analytics approaches. Eur J Oper Res 279(3):1036–1052

    Article  Google Scholar 

  20. Chen T, Li X, Yin H et al (2018) Call attention to rumors: deep attention based recurrent neural networks for early rumor detection. In: Ganji M, Rashidi L, Fung BCM et al (eds) Trends and applications in knowledge discovery and data mining. Springer International Publishing, Cham, pp 40–52

    Chapter  Google Scholar 

  21. Yu F, Liu Q, Wu S, et al. (2017) A Convolutional Approach for Misinformation Identification, Twenty-Sixth International Joint Conference on Artificial Intelligence. https://doi.org/10.24963/ijcai.2017/545.

  22. Kausar N, AliKhan A, Sattar M (2022) Towards better representation learning using hybrid deep learning model for fake news detection. Soc Netw Anal Min 12(1):165

    Article  Google Scholar 

  23. Taher Y, Moussaoui A, Moussaoui F (2022) Automatic fake news detection based on deep learning, FasTtext and news title. Int J Adv Comput Sci Appl, 13(1)

  24. Wang Y, Wang L, Yang Y et al (2022) Detecting fake news by enhanced text representation with multi-EDU-structure awareness. Expert Syst Appl 206:117781

    Article  Google Scholar 

  25. Qi P, Cao J, Yang T, et al. (2019) Exploiting multi-domain visual information for fake news detection. In: 2019 IEEE International Conference on Data Mining (ICDM). pp 518–527

  26. Singhal S, Shah RR, Chakraborty T, et al. (2019) SpotFake: a multi-modal framework for fake news detection. In: 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM). pp 39–47

  27. Singh VK, Ghosh I, Sonagara D (2021) Detecting fake news stories via multimodal analysis. J Am Soc Inf Sci 72(1):3–17

    Google Scholar 

  28. Wang Y, Ma F, Jin Z, et al. (2018) EANN: event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York, NY, USA: Association for Computing Machinery, pp 849–857

  29. Li S, Yao T, Li S et al (2022) Semantic-enhanced multimodal fusion network for fake news detection. Int J Intell Syst 37(12):12235–12251

    Article  Google Scholar 

  30. Guo Y (2023) A mutual attention based multimodal fusion for fake news detection on social network. Appl Intell 53(12):15311–15320

    Article  Google Scholar 

  31. Chen J, Wu Z, Yang Z et al (2022) Multimodal fusion network with contrary latent topic memory for rumor detection. IEEE Multimedia 29(1):104–113

    Article  Google Scholar 

  32. Jin Z, Cao J, Guo H, et al. (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM International Conference on Multimedia. New York, NY, USA: Association for Computing Machinery, pp 795–816

  33. Zhou X, Wu J, Zafarani R (2020) SAFE: Similarity-Aware Multi-modal Fake News Detection. In: Lauw HW, Wong RC-W, Ntoulas A, et al. (eds.) Advances in Knowledge Discovery and Data Mining. Cham: Springer International Publishing, pp 354–367

  34. Song C, Ning N, Zhang Y et al (2021) A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks. Inf Process Manage 58(1):102437

    Article  Google Scholar 

  35. Yadav A, Gaba S, Khan H, et al. (2023) Etma: Efficient transformer-based multilevel attention framework for multimodal fake news detection. IEEE Trans Comput Soc Syst

  36. Devlin J, Chang M W, Lee K, et al. (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  37. He K, Zhang X, Ren S, et al. (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 770–778

  38. Chen X et al (2023) Identifying Cantonese rumors with discriminative feature integration in online social networks. Expert Syst Appl 215:119347

    Article  Google Scholar 

  39. Comito C, Caroprese L, Zumpano E (2023) Multimodal fake news detection on social media: a survey of deep learning techniques. Soc Netw Anal Min 13(1):101

    Article  Google Scholar 

  40. Zhang K, Cao J, Pi D (2024) A novel fine-grained rumor detection algorithm with attention mechanism. Neurocomputing 583:127595

    Article  Google Scholar 

  41. Yao L, Mao C, Luo Y (2019) Graph convolutional networks for text classification. Proc AAAI Conf AI 33(01):7370–7377

    Google Scholar 

  42. Halliday MAK (2019) Linguistic function and literary style: an inquiry into the language of William Golding's' The Inheritors, Essays in modern stylistics. Routledge, pp 325–360

  43. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907

  44. Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: Xing EP, Jebara T (eds.) Proceedings of the 31st International Conference on Machine Learning. Bejing, China: PMLR, pp 1188–1196

  45. Vaswani A, Shazeer N, Parmar N, et al. (2017) Attention is all you need. Adv Neural Inf Process Syst, 30

  46. Rehman MZU, Nawaz R, Ullah MS et al (2025) A context-aware attention and graph neural network-based multimodal framework for misogyny detection. Inf Process Manage 62(1):103895

    Article  Google Scholar 

  47. Chen C, Han D, Chang CC (2022) CAAN: context-aware attention network for visual question answering. Pattern Recogn 132:108980

    Article  Google Scholar 

  48. Qu L, Liu M, Cao D, Nie L, Tian Q (2020) Context-aware multi-view summarization network for image-text matching. In: Proceedings of the 28th ACM International Conference on Multimedia. New York, NY, USA: Association for Computing Machinery, pp 1047–1055

  49. Loshchilov I, Hutter F (2017) Fixing weight decay regularization in Adam. arXiv preprint, arXiv:1711.05101.

  50. Boididou C, Papadopoulos S, Zampoglou M, Apostolidis L, Papadopoulou O, Kompatsiaris Y (2018) Detection and visualization of misleading content on Twitter. Int J Multimedia Inf Retr 77(1):1–86

    Google Scholar 

  51. Jin Z, Cao J, Guo H, Zhang Y, Luo J (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM International Conference on Multimedia, ACM, pp 795–816

  52. Khattar D, Goud JS, Gupta M, et al. MVAE: Multimodal Variational Autoencoder for Fake News Detection. In: The World Wide Web Conference. New York, NY, USA: Association for Computing Machinery, pp. 2915–2921.

  53. Antol S, Agrawal A, Lu J, et al. (2015) Vqa: Visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision. pp 2425–2433

  54. Vinyals O, Toshev A, Bengio S, et al. (2015) Show and tell: a neural image caption generator. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp 3156–3164

  55. Chen Y, Li D, Zhang P, et al. (2022) Cross-modal ambiguity learning for multimodal fake news detection. In: Proceedings of the ACM Web Conference 2022. New York, NY, USA: Association for Computing Machinery, pp 2897–2905

  56. Yang H, Zhang J, Zhang L et al (2024) MRAN: multimodal relationship-aware attention network for fake news detection. Comput Stand Interfaces 89:103822

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China (72174086).

Author information

Authors and Affiliations

  1. College of Economics and Management, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China

    Yaozeng Zhang & Jing Ma

  2. School of Information Management, Nanjing University, Nanjing, 211106, China

    Yuguang Jia

Authors
  1. Yaozeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jing Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuguang Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Yaozeng Zhang contributed to conceptualization, methodology, original draft preparation, and software. Jing Ma contributed to supervision. Yuguang Jia contributed to validation and visualization.

Corresponding author

Correspondence to Jing Ma.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Consent to publish

The images from social networking platforms presented in the text are all sourced from publicly available datasets and have been widely cited, thus allowing their publication.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Ma, J. & Jia, Y. MCAN: multimodal cross-aware network for fake news detection by extracting semantic-physical feature consistency. J Supercomput 81, 299 (2025). https://doi.org/10.1007/s11227-024-06815-1

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11227-024-06815-1

Keywords