Skip to main content
SpringerLink
Log in

Multimodal fake news detection on social media: a survey of deep learning techniques

  • Original Article
  • Published:
Social Network Analysis and Mining Aims and scope Submit manuscript

Abstract

The escalation of false information related to the massive use of social media has become a challenging problem, and significant is the effort of the research community in providing effective solutions to detecting it. Fake news are spreading for decades, but with the rise of social media, the nature of misinformation has evolved from text-based modality to visual modalities, such as images, audio, and video. Therefore, the identification of media-rich fake news requires an approach that exploits and effectively combines the information acquired from different multimodal categories. Multimodality is a key approach to improving fake news detection, but effective solutions supporting it are still poorly explored. More specifically, many different works exist that investigate if a text, an image, or a video is fake or not, but effective research on a real multimodal setting, ‘fusing’ the different modalities with their different structure and dimension is still an open problem. The paper is a focused survey concerning a very specific topic which is the use of deep learning (DL) methods for multimodal fake news detection on social media. The survey provides, for each work surveyed, a description of some relevant features such as the DL method used, the type of analysed data, and the fusion strategy adopted. The paper also highlights the main limitations of the current state of the art and draws some future directions to address open questions and challenges, including explainability and effective cross-domain fake news detection strategies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abdali S (2022) Multi-modal misinformation detection: approaches, challenges and opportunities

  • Alam F, Cresci S, Chakraborty T, Silvestri F, Dimitrov D, Martino GDS, Shaar S, Firooz H, Nakov P (2022) A survey on multimodal disinformation detection. In: Proceedings of the 29th international conference on computational linguistics. International Committee on Computational Linguistics, Gyeongju, Republic of Korea, pp 6625–6643

  • Alonso-Bartolome S, Segura-Bedmar I (2021) Multimodal fake news detection. arXiv. https://doi.org/10.48550/ARXIV.2112.04831

  • Amerini I, Ballan L, Caldelli R, Del Bimbo A, Serra G (2011) A sift-based forensic method for copy-move attack detection and transformation recovery. In: IEEE Transactions on information forensics and security, vol 6, pp 1099–1110. https://doi.org/10.1109/TIFS.2011.2129512

  • Amri S, Sallami D, Aïmeur E (2021) Exmulf: an explainable multimodal content-based fake news detection system. Springer, Berlin, pp 177–187. https://doi.org/10.1007/978-3-031-08147-7_12

    Book  Google Scholar 

  • Benamira A, Devillers B, Lesot E, Ray AK, Saadi M, Malliaros FD (2020) Semi-supervised learning and graph neural networks for fake news detection. In: Proceedings of the 2019 IEEE/ACM international conference on advances in social networks analysis and mining. ASONAM’19, pp 568–569

  • Boididou C, Papadopoulos S, Zampoglou M, Apostolidis L, Papadopoulou O, Kompatsiaris Y (2018) Detection and visualization of misleading content on twitter. Int J Multimed Info Retr 7:71–86. https://doi.org/10.1007/s13735-017-0143-x

    Article  Google Scholar 

  • Bovet A, Makse HA (2019) Influence of fake news in twitter during the 2016 US presidential election. Nat Commun. https://doi.org/10.1038/s41467-018-07761-2

    Article  Google Scholar 

  • Cao J, Sheng Q, Qi P, Zhong L, Wang Y, Zhang X (2019) False news detection on social media. arXiv. https://doi.org/10.48550/ARXIV.1908.10818

  • Cui L, Lee D (2020) CoAID: COVID-19 healthcare misinformation dataset

  • Cui L, Wang S, Lee D (2019) SAME: Sentiment-aware multi-modal embedding for detecting fake news. In: 2019 IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM), pp 41–48. https://doi.org/10.1145/3341161.3342894

  • da Silva FCD, Vieira R, Garcia ACB (2019) Can machines learn to detect fake news? A survey focused on social media. In: HICSS

  • Dolhansky B, Howes R, Pflaum B, Baram N, Ferrer CC (2019) The DeepFake detection challenge (DFDC) preview dataset

  • Dong M, Yao L, Wang X, Benatallah B, Sheng QZ, Huang H (2018) Dual: A deep unified attention model with latent relation representations for fake news detection. In: Hacid H, Cellary W, Wang H, Paik H-Y, Zhou R (eds) WISE, pp 199–209

  • Dou Y, Shu K, Xia C, Yu PS, Sun L (2021) User preference-aware fake news detection. arXiv

  • Eyben F, Weninger F, Groß F, Schuller B (2013) Recent developments in openSMILE, the munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM international conference on multimedia

  • Ferreira W, Vlachos A (2016) Emergent: a novel data-set for stance classification. https://doi.org/10.18653/v1/N16-1138

  • Giachanou A, Zhang G, Rosso P (2020) Multimodal multi-image fake news detection. In: 2020 IEEE 7th international conference on data science and advanced analytics (DSAA), pp 647–654. https://doi.org/10.1109/DSAA49011.2020.00091

  • Hameleers M, Powell TE, Meer TGLAVD, Bos L (2020) A picture paints a thousand lies? The effects and mechanisms of multimodal disinformation and rebuttals disseminated via social media. Polit Commun 37(2):281–301. https://doi.org/10.1080/10584609.2019.1674979

    Article  Google Scholar 

  • Hangloo S, Arora B (2022) Combating multimodal fake news on social media: methods, datasets, and future perspective. Multimedia Syst 28:2391–2422

    Article  Google Scholar 

  • Hua J, Cui X, Li X, Tang K, Zhu P (2023) Multimodal fake news detection through data augmentation-based contrastive learning. Appl Soft Comput 136:110125. https://doi.org/10.1016/j.asoc.2023.110125

    Article  Google Scholar 

  • Ji S, Xu W, Yang M, Yu K (2013) 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell 35(1):221–231. https://doi.org/10.1109/TPAMI.2012.59

    Article  Google Scholar 

  • Jiang S, Chen X, Zhang L, Chen S, Liu H (2019) User-characteristic enhanced model for fake news detection in social media. In: Tang J, Kan M, Zhao D, Li S, Zan H (eds) Natural language processing and Chinese computing—8th CCF international conference, NLPCC 2019, Dunhuang, China, October 9–14, 2019, Proceedings, Part I. Lecture notes in computer science, vol 11838. Springer, Berlin, pp 634–646

  • Jin Z, Cao J, Guo H, Zhang Y, Luo J (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM international conference on multimedia. MM’17. Association for Computing Machinery, New York, pp 795–816

  • Jing Q, Yao D, Fan X, Wang B, Tan H, Bu X, Bi J (2021) TRANSFAKE: Multi-task transformer for multimodal enhanced fake news detection. In: IJCNN, pp 1–8

  • Kaliyar RK, Kumar P, Kumar M, Narkhede M, Namboodiri S, Mishra S (2020) Deepnet: an efficient neural network for fake news detection using news-user engagements. In: 2020 5th International conference on computing, communication and security (ICCCS), pp 1–6. https://doi.org/10.1109/ICCCS49678.2020.9277353

  • Karimi H, Tang J, Li Y (2018) Toward end-to-end deception detection in videos. In: 2018 IEEE international conference on big data (Big Data), pp 1278–1283. https://doi.org/10.1109/BigData.2018.8621909

  • Khattar D, Goud JS, Gupta M, Varma V (2019) Mvae: Multimodal variational autoencoder for fake news detection. In: The world wide web conference. WWW’19, pp 2915–2921

  • Kirchknopf A, Slijepčević D, Zeppelzauer M (2021) Multimodal detection of information disorder from social media. In: CBMI Conf., pp 1–4. https://doi.org/10.1109/CBMI50038.2021.9461898

  • Korshunov P, Marcel S (2018) DeepFakes: a new threat to face recognition? Assessment and detection

  • Krishnamurthy G, Majumder N, Poria S, Cambria E (2018) A deep learning approach for multimodal deception detection. arXiv. https://doi.org/10.48550/ARXIV.1803.00344

  • Kumari R, Ekbal A (2021) AMFB: Attention based multimodal factorized bilinear pooling for multimodal fake news detection. Expert Syst Appl 184:115412

    Article  Google Scholar 

  • Kumar S, Shah N (2018) False information on web and social media: a survey. arXiv. https://doi.org/10.48550/ARXIV.1804.08559

  • Li Y, Xie Y (2020) Is a picture worth a thousand words? an empirical study of image content and social media engagement. J Mark Res 57(1):1–19. https://doi.org/10.1177/0022243719881113

    Article  Google Scholar 

  • Lu Y-J, Li C-T (2020) GCAN: Graph-aware co-attention networks for explainable fake news detection on social media. In: Proceedings of the 58th annual meeting of the association for computational linguistics, pp 505–514

  • Ma J, Gao W, Mitra P, Kwon S, Jansen BJ, Wong K-F, Cha M (2016) Detecting rumors from microblogs with recurrent neural networks. In: IJCAI’16, pp 3818–3824

  • McCrae S, Wang K, Zakhor A (2022) Multi-modal semantic inconsistency detection in social media news posts. In: Multimedia modeling. Springer, Berlin, pp 331–343

  • Mendels G, Levitan SI, Lee K-Z, Hirschberg J (2017) Hybrid acoustic-lexical deep learning approach for deception detection. In: INTERSPEECH

  • Mittal T, Bhattacharya U, Chandra R, Bera A, Manocha D (2020) Emotions Don’t Lie: an audio-visual deepfake detection method using affective cues

  • Mosallanezhad A, Karami M, Shu K, Mancenido MV, Liu H (2022) Domain adaptive fake news detection via reinforcement learning. In: Proceedings of the ACM web conference 2022. https://doi.org/10.1145/3485447.3512258

  • Mu M, Bhattacharjee SD, Yuan J (2023) Self-supervised distilled learning for multi-modal misinformation identification. In: IEEE/CVF Winter conference on applications of computer vision, WACV 2023, Waikoloa, HI, USA, January 2–7, 2023. IEEE, pp 2818–2827

  • Murayama T (2021) Dataset of fake news detection and fact verification: a survey

  • Nakamura K, Levy S, Wang WY (2019) r/Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. arXiv. https://doi.org/10.48550/ARXIV.1911.03854

  • Pérez-Rosas V, Abouelenien M, Mihalcea R, Burzo M (2015) Deception detection using real-life trial data. In: 2015 ACM on international conference on multimodal interaction, pp 59–66

  • Qian F, Gong C, Sharma K, Liu Y (2018) Neural user response generator: fake news detection with collective user intelligence. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence, IJCAI-18, pp 3834–3840

  • Qi P, Cao J, Li X, Liu H, Sheng Q, Mi X, He Q, Lv Y, Guo C, Yu Y (2021) Improving fake news detection by using an entity-enhanced framework to fuse diverse multimodal clues, pp 1212–1220

  • Raj C, Meel P (2021) Convnet frameworks for multi-modal fake news detection. Appl Intell. https://doi.org/10.1007/s10489-021-02345-y

    Article  Google Scholar 

  • Rezayi S, Soleymani S, Arabnia HR, Li S (2021) Socially aware multimodal deep neural networks for fake news classification. In: 2021 IEEE 4th international conference on multimedia information processing and retrieval (MIPR), pp 253–259. https://doi.org/10.1109/MIPR51284.2021.00048

  • Ruchansky N, Seo S, Liu Y (2017) CSI: A hybrid deep model for fake news detection. In: Proceedings of the 2017 ACM on conference on information and knowledge management. ACM. https://doi.org/10.1145/3132847.3132877

  • Sachan T, Pinnaparaju N, Gupta M, Varma V (2021) SCATE: Shared cross attention transformer encoders for multimodal fake news detection. In: Proceedings of the 2021 IEEE/ACM international conference on advances in social networks analysis and mining. ASONAM’21, pp 399–406

  • Shang L, Kou Z, Zhang Y, Wang D (2021) A multimodal misinformation detector for COVID-19 short videos on Tiktok. In: 2021 IEEE international conference on big data (big data), pp 899–908. https://doi.org/10.1109/BigData52589.2021.9671928

  • Shang L, Kou Z, Zhang Y, Wang D (2022) A duo-generative approach to explainable multimodal COVID-19 misinformation detection. In: Proceedings of the ACM web conference 2022. WWW’22, pp 3623–3631

  • Shu K, Sliva A, Wang S, Tang J, Liu H (2017) Fake news detection on social media: a data mining perspective. SIGKDD Explor Newsl 19(1):22–36

    Article  Google Scholar 

  • Shu K, Cui L, Wang S, Lee D, Liu H (2019a) dEFEND: Explainable fake news detection. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery and data mining. KDD’19, pp 395–405

  • Shu K, Mahudeswaran D, Wang S, Lee D, Liu H (2019b) FakeNewsNet: a data repository with news content, social context and spatial temporal information for studying fake news on social media

  • Shu K, Mahudeswaran D, Wang S, Liu H (2020) Hierarchical propagation networks for fake news detection: Investigation and exploitation. In: Proceedings of the international AAAI conference on web and social media, vol 14, issue 1, pp 626–637

  • Shu K, Wang S, Liu H (2019) Beyond news contents: The role of social context for fake news detection. In: Proceedings of the twelfth ACM international conference on web search and data mining. WSDM’19, pp 312–320

  • Silva A, Luo L, Karunasekera S, Leckie C (2021) Embracing domain differences in fake news: cross-domain fake news detection using multi-modal data. In: The thirty-fifth AAAI conference on artificial intelligence (AAAI-21). https://doi.org/10.48550/ARXIV.2102.06314

  • Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition

  • Singhal S, Dhawan M, Shah RR, Kumaraguru P (2021) Inter-modality discordance for multimodal fake news detection. In: MMAsia

  • Song C, Ning N, Zhang Y, Wu B (2021) A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks. Inf Process Manag 58(1):102437

    Article  Google Scholar 

  • Volkova S, Shaffer K, Jang JY, Hodas N (2017) Separating facts from fiction: linguistic models to classify suspicious and trusted news posts on Twitter. In: Proceedings of the 55th annual meeting of the association for computational linguistics (volume 2: short papers), pp 647–653

  • Vosoughi S, Roy D, Aral S (2018) The spread of true and false news online. Science 359:1146–1151. https://doi.org/10.1126/science.aap9559

    Article  Google Scholar 

  • Wang WY (2017) “Liar, Liar Pants on Fire”: a new benchmark dataset for fake news detection. arXiv. https://doi.org/10.48550/ARXIV.1705.00648

  • Wang Y, Ma F, Jin Z, Yuan Y, Xun G, Jha K, Su L, Gao J (2018) Eann: Event adversarial neural networks for multi-modal fake news detection. In: KDD, pp 849–857

  • Wang Y, Ma F, Wang H, Jha K, Gao J (2021) Multimodal emergent fake news detection via meta neural process networks. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery and data mining. KDD’21, pp 3708–3716

  • Wang K, Chan D, Zhao SZ, Canny J, Zakhor A (2022) Misinformation detection in social media video posts

  • Wu L, Rao Y (2020) Adaptive interaction fusion networks for fake news detection. In: 24th European conference on artificial intelligence—ECAI 2020

  • Xie J, Liu S, Liu R, Zhang Y, Zhu Y (2021) SERN: Stance extraction and reasoning network for fake news detection. In: ICASSP, pp 2520–2524. https://doi.org/10.1109/ICASSP39728.2021.9414787

  • Xue J, Wang Y, Tian Y, Li Y, Shi L, Wei L (2021) Detecting fake news by exploring the consistency of multimodal data. Inf Process Manag 58(5):102610

    Article  Google Scholar 

  • Yang Y, Zheng L, Zhang J, Cui Q, Li Z, Yu PS (2018) TI-CNN: convolutional neural networks for fake news detection. arXiv

  • Yang S, Shu K, Wang S, Gu R, Wu F, Liu H (2019) Unsupervised fake news detection on social media: a generative approach. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, issue 01, pp 5644–5651

  • Zannettou S, Caulfield T, Blackburn J, De Cristofaro E, Sirivianos M, Stringhini G, Suarez-Tangil G (2018) On the origins of memes by means of fringe web communities. arXiv. https://doi.org/10.48550/ARXIV.1805.12512

  • Zhang DY, Shang L, Geng B, Lai S, Li K, Zhu H, Amin MT, Wang D (2018) FauxBuster: A content-free fauxtography detector using social media comments. In: 2018 IEEE Big Data Conf., pp 891–900. https://doi.org/10.1109/BigData.2018.8622344

  • Zhang H, Fang Q, Qian S, Xu C (2019) Multi-modal knowledge-aware event memory network for social media rumor detection. In: Proceedings of the 27th ACM international conference on multimedia. MM’19, pp 1942–1951

  • Zhang T, Wang D, Chen H, Zeng Z, Guo W, Miao C, Cui L (2020) BDANN: Bert-based domain adaptation neural network for multi-modal fake news detection. In: 2020 International joint conference on neural networks (IJCNN), pp 1–8. https://doi.org/10.1109/IJCNN48605.2020.9206973

  • Zhou X, Mulay A, Ferrara E, Zafarani R (2020a) Recovery: a multimodal repository for COVID-19 news credibility research. In: CIKM’20. Association for Computing Machinery, New York, pp 3205–3212. https://doi.org/10.1145/3340531.3412880

  • Zhou X, Wu J, Zafarani R (2020b) SAFE: similarity-aware multi-modal fake news detection. arXiv. https://doi.org/10.48550/ARXIV.2003.04981

  • Zubiaga A, Liakata M, Procter R, Hoi GWS, Tolmie P (2016) Analysing how people orient to and spread rumours in social media by looking at conversational threads. PLOS ONE 11(3):0150989. https://doi.org/10.1371/journal.pone.0150989

    Article  Google Scholar 

Download references

Acknowledgements

This work was partially supported by project SERICS (PE00000014) under the MUR National Recovery and Resilience Plan funded by the European Union - NextGenerationEU.

Author information

Authors and Affiliations

  1. ICAR-CNR, Rende, Italy

    Carmela Comito

  2. DIMES, University of Calabria, Arcavacata, Italy

    Luciano Caroprese

  3. INGEO, University G. d′Annunzio of Chieti-Pescara, Pescara, Italy

    Ester Zumpano

Authors
  1. Carmela Comito
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luciano Caroprese
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ester Zumpano
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luciano Caroprese.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Comito, C., Caroprese, L. & Zumpano, E. Multimodal fake news detection on social media: a survey of deep learning techniques. Soc. Netw. Anal. Min. 13, 101 (2023). https://doi.org/10.1007/s13278-023-01104-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13278-023-01104-w

Keywords

Search

Navigation