Skip to main content
SpringerLink
Log in

A unified generalization enabled ML architecture for manipulated multi-modal social media

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

With the emergence of online social networks (OSNs) incorporating multimodal media, it is challenging to differentiate manipulated multimedia content due to unseen patterns. Social media’s lack of control and verification mechanism provides fertile ground for manipulated content. Hence, maintaining social networks’ integrity is essential to identifying manipulated content that misleads and confuses social media users. In this work, we have proposed a deep-learning driven unified-generalized architecture for multimodal datasets with generalization ability. The proposed model uses a variant of a transformer-based sequential neural network with a combination of modified pre-trained CNN models to identify manipulated content from social platforms. The proposed model is configured with state-of-the-art generalization techniques with proper parametrization, optimized for (near-) optimal performance on unseen multimodal datasets. The proposed model seamlessly works on unimodal as well as multimodal datasets. We have considered Weibo multimodal and two unimodal datasets to assess the proposed model’s performance. The critical challenge with the multimodal dataset is the concatenation of several different modalities that should work well for unseen multimodal datasets showing effective generalization. The proposed model’s generalization ability is improved compared to the state-of-art models. This study aims to assess various feasible technologies that can be used to deal with unseen manipulated social media content for trust and reliability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Algorithm 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability

All data used in this work is publicly available.

Code Availability

The program code and artefacts, through the GitHub repository, necessary to run and execute for interpreting, replicating, and building on the findings reported in the paper, will be made publicly available on publication of this paper.

Notes

  1. MediaEval-VMU https://github.com/MKLab-ITI/image-verification-corpus/

  2. FakeNewsNet https://github.com/KaiDMML/FakeNewsNet/

  3. MCG-FNews http://mcg.ict.ac.cn/wordpress/share/mcg-fnews/

  4. EMNLP19 http://gitlab.com/didizlatkova/fake-image-detection/

  5. Kaggle https://www.kaggle.com/competitions/fake-news/data/

  6. https://github.com/op5637/ManipulatedMedia/

References

  1. Agrawal C, Pandey A, Goyal S (2022) Fake news detection system based on modified bi-directional LSTM. Multimed Tools Appl 81(17):24199–24223. https://doi.org/10.1007/s11042-022-12772-9

    Article  Google Scholar 

  2. Ahmed H, Traore I, Saad S (2018) Detecting opinion spams and fake news using text classification. Secur Priv 1:9. https://doi.org/10.1002/spy2.9

    Article  Google Scholar 

  3. Alnaim NM, Almutairi ZM, Alsuwat MS, Alalawi HH, Alshobaili A, Alenezi FS (2023) Dffmd: A deepfake face mask dataset for infectious disease era with deepfake detection algorithms. IEEE Access 11:16711–16722. https://doi.org/10.1109/ACCESS.2023.3246661

    Article  Google Scholar 

  4. Baarir, NF, Djeffal A (2021) Fake news detection using ML. In: Proc. IEEE Int. Workshop on Human-Centric Smart Env. for Health and Well-being, pp. 125–130. https://doi.org/10.1109/IHSH51661.2021.9378748

  5. Bhuvaneshwari P, Rao AN, Robinson YH (2021) Spam review detection using self-attention based CNN and bi-directional LSTM. Multimed Tools Appl 80:18107–18124. https://doi.org/10.1007/s11042-021-10602-y

    Article  Google Scholar 

  6. Boididou C, Andreadou K, Papadopoulos S, Dang-Nguyen DT, Boato G, Riegler M, Kompatsiaris Y et al (2015) Verifying multimedia use at MediaEval 2015. MediaEval 3(3):7

    Google Scholar 

  7. Boididou C, Middleton SE, Jin Z, Papadopoulos S, Dang-Nguyen DT, Boato G, Kompatsiaris Y (2018) Verifying information with multimedia content on Twitter. Multimed Tools Appl 77:15545–15571. https://doi.org/10.1007/s11042-017-5132-9

    Article  Google Scholar 

  8. Boididou C, Middleton S, Papadopoulos S, Dang-Nguyen DT, Riegler M, Boato G, Petlund A, Kompatsiaris I (2016) The VMU participation@ verifying multimedia use 2016. MediaEval

  9. Choudhury D, Acharjee T (2023) A novel approach to fake news detection in social networks using GA applying ML classifiers. Multimed Tools Appl 82(6):9029–9045. https://doi.org/10.1007/s11042-022-12788-1

    Article  Google Scholar 

  10. Fang P, Zhou J, Roy SK, Ji P, Petersson L, Harandi M (2022) Attention in attention networks for person retrieval. IEEE Trans Pattern Anal Mach Intell 44(9):4626–4641. https://doi.org/10.1109/TPAMI.2021.3073512

    Article  PubMed  Google Scholar 

  11. Fu T, Xia M, Yang G (2022) Detecting GAN-generated face images via hybrid texture and sensor noise-based features. Multimed Tools Appl 81(18):26345–26359. https://doi.org/10.1007/s11042-022-12661-1

    Article  Google Scholar 

  12. Galli A, Masciari E, Moscato V, Sperlí G (2022) A comprehensive benchmark for fake news detection. J Intell Inf Syst 59(1):237–261

    Article  PubMed  PubMed Central  Google Scholar 

  13. Ghayoomi M, Mousavian M (2022) Deep transfer learning for covid-19 fake news detection in persian. Expert Syst 39(8):13008. https://doi.org/10.1111/exsy.13008

    Article  Google Scholar 

  14. Hua J, Cui X, Li X, Tang K, Zhu P (2023) Multimodal fake news detection through data augmentation-based contrastive learning. Appl Soft Comput 136:110125. https://doi.org/10.1016/j.asoc.2023.110125

    Article  Google Scholar 

  15. Jahanbakhsh-Nagadeh Z, Feizi-Derakhshi MR, Sharifi A (2021) A semi-supervised model for persian rumor verification based on content information. Multimed Tools Appl 80:35267–35295. https://doi.org/10.1007/s11042-020-10077-3

    Article  Google Scholar 

  16. Jarrahi A, Safari L (2023) Evaluating the effectiveness of publishers’ features in fake news detection on social media. Multimed Tools Appl 82(2):2913–2939. https://doi.org/10.1007/s11042-022-12668-8

    Article  PubMed  Google Scholar 

  17. Jin Z, Cao J, Zhang Y, Zhou J, Tian Q (2016) Novel visual and statistical image features for microblogs news verification. IEEE Trans Multimed 19(3):598–608. https://doi.org/10.1109/TMM.2016.2617078

    Article  Google Scholar 

  18. Jin Z, Cao J, Zhang Y, Zhou J, Tian Q (2016) Novel visual and statistical image features for microblogs news verification. IEEE Trans Multimed 19(3):598–608. https://doi.org/10.1109/TMM.2016.2617078

    Article  Google Scholar 

  19. Jin Z, Cao J, Guo H, Zhang Y, Luo J (2017) Multimodal fusion with RNNs for rumor detection on microblogs. In: Proc. 25th ACM Int. Conf. Multimedia, pp. 795–816. https://doi.org/10.1145/3123266.3123454

  20. Kaliyar RK, Goswami A, Narang P (2021) FakeBERT: Fake news detection in social media with a BERT-based deep learning approach. Multimed Tools Appl 80(8):11765–11788. https://doi.org/10.1007/s11042-020-10183-2

    Article  PubMed  PubMed Central  Google Scholar 

  21. Kalsnes B (2018) Fake news. Oxford Research Encyclopedia of Communication. https://doi.org/10.1093/acrefore/9780190228613.013.809

    Article  Google Scholar 

  22. Khattar D, Goud JS, Gupta M, Varma V (2019) Mvae: Multimodal variational autoencoder for fake news detection. In: Proc. The World Wide Web (WWW) Conf., pp. 2915–2921. https://doi.org/10.1145/3308558.3313552

  23. Kumar A, Bhatia M, Sangwan SR (2022) Rumour detection using DL and filter-wrapper feature selection in benchmark Twitter dataset. Multimed Tools Appl 81(24):34615–34632. https://doi.org/10.1007/s11042-021-11340-x

    Article  PubMed  Google Scholar 

  24. Lago F, Phan QT, Boato G (2019) Visual and textual analysis for image trustworthiness assessment within online news. Secur Commun Netw. https://doi.org/10.1155/2019/9236910

    Article  Google Scholar 

  25. Li Q, Yu G, Wang J, Liu Y (2020) A deep multimodal generative and fusion framework for class-imbalanced multimodal data. Multimed Tools Appl 79:25023–25050. https://doi.org/10.1007/s11042-020-09227-4

    Article  Google Scholar 

  26. Li X, Lu P, Hu L, Wang X, Lu L (2022) A novel self-learning semi-supervised DL network to detect fake news on social media. Multimed Tools Appl 81(14):19341–19349. https://doi.org/10.1007/s11042-021-11065-x

    Article  PubMed  Google Scholar 

  27. Li S, Xiao T, Li H, Zhou B, Yue D, Wang X (2017) Person search with natural language description. In: Proc. IEEE Conf. Compt. Vision & Pattern Recognition, pp. 1970–1979. https://doi.org/10.48550/arXiv.1702.05729

  28. Martín-Gutiérrez D, Hernández-Peñaloza G, Hernández AB, Lozano-Diez A, Álvarez F (2021) A deep learning approach for robust detection of bots in twitter using transformers. IEEE Access 9:54591–54601. https://doi.org/10.1109/ACCESS.2021.3068659

    Article  Google Scholar 

  29. Meel P, Chawla P, Jain S, Rai U (2020) Web text content credibility analysis using max voting and stacking ensemble classifiers. In: Proc. IEEE Conf. Adv. Computing & Communication Technologies for High-Performance Applications, pp. 157–161. https://doi.org/10.1109/ACCTHPA49271.2020.9213234

  30. Mohapatra A, Thota N, Prakasam P (2022) Fake news detection and classification using hybrid BiLSTM and self-attention model. Multimed Tools Appl 81(13):18503–18519. https://doi.org/10.1007/s11042-022-12764-9

    Article  Google Scholar 

  31. Ni P, Zhu J, Wang G (2023) Misinformation influence minimization by entity protection on multi-social networks. Appl Intell 53(6):6401–6420. https://doi.org/10.1007/s10489-022-03798-5

    Article  Google Scholar 

  32. Peng X, Xintong B (2022) An effective strategy for multi-modal fake news detection. Multimed Tools Appl 81(10):13799–13822. https://doi.org/10.1007/s11042-022-12290-8

    Article  Google Scholar 

  33. Preiss J (2022) Predicting the impact of online news articles-is information necessary? application to COVID-19 articles. Multimed Tools Appl 1–19. https://doi.org/10.1007/s11042-021-11621-5

  34. Qi P, Cao J, Yang T, Guo J, Li J (2019) Exploiting multi-domain visual information for fake news detection. In: Proc. IEEE Int. Conf. Data Mining (ICDM), pp. 518–527. https://doi.org/10.1109/ICDM.2019.00062

  35. Rashed KAN, Renzel D, Klamma R, Jarke M (2014) Community and trust-aware fake media detection. Multimed Tools Appl 70:1069–1098. https://doi.org/10.1007/s11042-012-1103-3

    Article  Google Scholar 

  36. Rastogi S, Bansal D (2022) Disinformation detection on social media: An integrated approach. Multimed Tools Appl 81(28):40675–40707. https://doi.org/10.1007/s11042-022-13129-y

    Article  PubMed  PubMed Central  Google Scholar 

  37. Raza S, Ding C (2022) Fake news detection based on news content and social contexts: a transformer-based approach. Int J Data Sci Anal 13(4):335–362. https://doi.org/10.1007/s41060-021-00302-z

    Article  PubMed  PubMed Central  Google Scholar 

  38. Reddy H, Raj N, Gala M, Basava A (2020) Text-mining-based fake news detection using ensemble methods. Int J Autom Comput 17(2):210–221. https://doi.org/10.1007/s11633-019-1216-5

    Article  Google Scholar 

  39. Saleh H, Alharbi A, Alsamhi SH (2021) OPCNN-FAKE: Optimized CNN for fake news detection. IEEE Access 9:129471–129489. https://doi.org/10.1109/ACCESS.2021.3112806

    Article  Google Scholar 

  40. Samadi M, Mousavian M, Momtazi S (2021) Deep contextualized text representation and learning for fake news detection. Inf Process Manage 58(6):102723

    Article  Google Scholar 

  41. Shelke S, Attar V (2022) Rumor detection in social network based on user, content, and lexical features. Multimed Tools Appl 81(12):17347–17368. https://doi.org/10.1007/s11042-022-12761-y

    Article  PubMed  PubMed Central  Google Scholar 

  42. Shrivastava G, Kumar P, Ojha RP, Srivastava PK, Mohan S, Srivastava G (2020) Defensive modeling of fake news through online social networks. IEEE Trans Comput Social Syst 7(5):1159–1167. https://doi.org/10.1109/TCSS.2020.3014135

    Article  Google Scholar 

  43. Shu K, Sliva A, Wang S, Tang J, Liu H (2017) Fake news detectionon social media: A data mining perspective. ACM SIGKDD Explor Newslett 19(1):22–36. https://doi.org/10.1145/3137597.3137600

    Article  Google Scholar 

  44. Shu K, Mahudeswaran D, Wang S, Lee D, Liu H (2020) Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. Big Data 8(3):171–188. https://doi.org/10.1089/big.2020.0062

    Article  PubMed  Google Scholar 

  45. Singh B, Sharma DK (2022) Predicting image credibility in fake news over social media using multi-modal approach. Neural Comput Appl 34(24):21503–21517. https://doi.org/10.1007/s00521-021-06086-4

    Article  PubMed  Google Scholar 

  46. Smitha N, Bharath R (2020) Performance comparison of ML classifiers for fake news detection. In: Proc. 2nd Int. IEEE Conf. Inventive Research in Computing Applications, pp. 696–700. https://doi.org/10.1109/ICIRCA48905.2020.9183072

  47. Tan KL, Poo Lee C, Lim KM (2021) Fake news detection with hybrid CNN-LSTM. In: Proc. IEEE Int. Conf. Info. & Comm. Tech. (ICoICT), pp. 606–610. https://doi.org/10.1109/ICoICT52021.2021.9527469

  48. Tan L, Wang G, Jia F, Lian X (2023) Research status of deep learning methods for rumor detection. Multimed Tools Appl 82(2):2941–2982. https://doi.org/10.1007/s11042-022-12800-8

    Article  PubMed  Google Scholar 

  49. Umer M, Imtiaz Z, Ullah S, Mehmood A, Choi GS, On BW (2020) Fake news stance detection using DL architecture CNN-LSTM. IEEE Access 8:156695–156706. https://doi.org/10.1109/ACCESS.2020.3019735

    Article  Google Scholar 

  50. Velayudhan SP, Bhanu SMS (2020) UbCadet: detection of compromised accounts in Twitter based on user behavioural profiling. Multimed Tools Appl 79:19349–19385. https://doi.org/10.1007/s11042-020-08721-z

    Article  Google Scholar 

  51. Viji D, Revathy S (2022) A hybrid approach of weighted fine-tuned BERT extraction with deep siamese bi-LSTM model for semantic text similarity identification. Multimed Tools Appl 81(5):6131–6157. https://doi.org/10.1007/s11042-021-11771-6

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  52. Xie J, Ma Z, Lei J, Zhang G, Xue JH, Tan ZH, Guo J (2022) Advanced dropout: A model-free methodology for Bayesian dropout optimization. IEEE Trans Pattern Anal Mach Intell 44(9):4605–4625. https://doi.org/10.1109/TPAMI.2021.3083089

    Article  PubMed  Google Scholar 

  53. Xu S, Liu X, Ma K, Dong F, Riskhan B, Xiang S, Bing C (2023) Rumor detection on social media using hierarchically aggregated feature via graph NNs. Appl Intell 53(3):3136–3149. https://doi.org/10.1007/s10489-022-03592-3

    Article  Google Scholar 

  54. You Q, Cao L, Jin H, Luo J (2016) Robust visual-textual sentiment analysis: When attention meets tree-structured RNNs. In: Proc. 24th ACM Int. Conf. Multimedia, pp. 1008–1017. https://doi.org/10.1145/2964284.2964288

  55. Zhang T (2022) Deepfake generation and detection, a survey. Multimed Tools Appl 81(5):6259–6276. https://doi.org/10.1007/s11042-021-11733-y

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors thank the reviewers for their insightful comments and suggestions. The authors also express their gratitude to the Editor-in-Chief, the Associate Editor, and the Editorial Office Assistant(s) of this journal for managing this manuscript.

Author information

Authors and Affiliations

  1. Data to Knowledge (D2K) Lab, School of Computer and Systems Sciences, Jawaharlal Nehru University, 110067, New Delhi, India

    Om Prakash & Rajeev Kumar

Authors
  1. Om Prakash
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rajeev Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Om Prakash.

Ethics declarations

Competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Prakash, O., Kumar, R. A unified generalization enabled ML architecture for manipulated multi-modal social media. Multimed Tools Appl 83, 22749–22771 (2024). https://doi.org/10.1007/s11042-023-16198-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16198-9

Keywords

Search

Navigation