Skip to main content
Springer Nature Link
Log in

Enhancing Few-Shot Multi-modal Fake News Detection Through Adaptive Fusion

  • Conference paper
  • First Online:
Web and Big Data (APWeb-WAIM 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14964))

  • 412 Accesses

Abstract

Currently, news content with images and texts on social media is widely spread, prompting significant interest in multi-modal fake news detection. However, existing research in this field focuses on large-scale annotated data to train models. Furthermore, data scarcity characterizes the initial stages of fake news propagation. Hence, addressing the challenge of few-shot multi-modal fake news detection becomes essential. In scenarios of limited data availability, current research inadequately utilizes the information inherent in each modality, leading to underutilization of modal information. To address the above challenges, in the paper, we propose a novel detection approach called Prompt-based Adaptive Fusion(ProAF). Specifically, to enhance the model’s comprehension of news content, we extract supplementary information from two modalities to facilitate timely guidance for model training. Then the model employs adaptive fusion to integrate the output predictions of different prompts during training, effectively enhancing the robust performance of the model. Experimental results on two datasets illustrate that our model surpasses existing methods, representing a significant advancement in few-shot multi-modal fake news detection.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. O’Connor, C., Murphy, M.: Going viral: doctors must tackle fake news in the COVID-19 pandemic. BMJ 369(10.1136) (2020)

    Google Scholar 

  2. Grinberg, N., Joseph, K., Friedland, L., Swire-Thompson, B., Lazer, D.: Fake news on twitter during the 2016 US presidential election. Science 363(6425), 374–378 (2019)

    Article  Google Scholar 

  3. Yu, C., Ma, Y., An, L., Li, G.: BCMF: a bidirectional cross-modal fusion model for fake news detection. Inf. Process. Manag. 59(5), 103063 (2022)

    Article  Google Scholar 

  4. Wang, B., Feng, Y., Xiong, X.C., Wang, Y.H., Qiang, B.H.: Multi-modal transformer using two-level visual features for fake news detection. Appl. Intell. 53(9), 10429–10443 (2023)

    Google Scholar 

  5. Bowman, S., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632–642 (2015)

    Google Scholar 

  6. Wu, Y., Zhan, P., Zhang, Y., Wang, L., Xu, Z.: Multimodal fusion with co-attention networks for fake news detection. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2560–2569 (2021)

    Google Scholar 

  7. Gu, Y., Han, X., Liu, Z., Huang, M.: PPT: pre-trained prompt tuning for few-shot learning. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8410–8423 (2022)

    Google Scholar 

  8. Wu, J., Li, S., Deng, A., Xiong, M., Hooi, B.: Prompt-and-align: prompt-based social alignment for few-shot fake news detection. In: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 2726–2736 (2023)

    Google Scholar 

  9. Gao, W., Ni, M., Deng, H., Zhu, X., Zeng, P., Hu, X.: Few-shot fake news detection via prompt-based tuning. J. Intell. Fuzzy Syst. (Preprint), 1–10 (2023)

    Google Scholar 

  10. Jiang, Y., Yu, X., Wang, Y., Xu, X., Song, X., Maynard, D.: Similarity-aware multimodal prompt learning for fake news detection. Inf. Sci. 647, 119446 (2023)

    Article  Google Scholar 

  11. Qi, P., Cao, J., Yang, T., Guo, J., Li, J.: Exploiting multi-domain visual information for fake news detection. In: 2019 IEEE International Conference on Data Mining (ICDM), pp. 518–527. IEEE (2019)

    Google Scholar 

  12. Jin, Z., Cao, J., Guo, H., Zhang, Y., Luo, J.: Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM International Conference on Multimedia, pp. 795–816 (2017)

    Google Scholar 

  13. Khattar, D., Goud, J.S., Gupta, M., Varma, V.: MVAE: multimodal variational autoencoder for fake news detection. In: The World Wide Web Conference, pp. 2915–2921 (2019)

    Google Scholar 

  14. Kumari, R., Ekbal, A.: AMFB: attention based multimodal factorized bilinear pooling for multimodal fake news detection. Expert Syst. Appl. 184, 115412 (2021)

    Article  Google Scholar 

  15. Zhang, H., Fang, Q., Qian, S., Xu, C.: Multi-modal knowledge-aware event memory network for social media rumor detection. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 1942–1951 (2019)

    Google Scholar 

  16. Floridi, L., Chiriatti, M.: GPT-3: its nature, scope, limits, and consequences. Mind. Mach. 30, 681–694 (2020)

    Article  Google Scholar 

  17. Kenton, J.D.M.W.C., Toutanova, L.K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp. 4171–4186 (2019)

    Google Scholar 

  18. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  19. Shin, T., Razeghi, Y., Logan IV, R.L., Wallace, E., Singh, S.: Autoprompt: eliciting knowledge from language models with automatically generated prompts. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222–4235 (2020)

    Google Scholar 

  20. Li, X.L., Liang, P.: Prefix-tuning: optimizing continuous prompts for generation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics (2021)

    Google Scholar 

  21. Jiang, G., Liu, S., Zhao, Y., Sun, Y., Zhang, M.: Fake news detection via knowledgeable prompt learning. Inf. Process. Manag. 59(5), 103029 (2022)

    Article  Google Scholar 

  22. Yu, Y., Zhang, D., Li, S.: Unified multi-modal pre-training for few-shot sentiment analysis with prompt-based learning. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 189–198 (2022)

    Google Scholar 

  23. Brock, A., De, S., Smith, S.L.: Characterizing signal propagation to close the performance gap in unnormalized resnets. In: International Conference on Learning Representations (2020)

    Google Scholar 

  24. Yu, Y., Zhang, D.: Few-shot multi-modal sentiment analysis with prompt-based vision-aware language modeling. In: 2022 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2022)

    Google Scholar 

  25. Mokady, R., Hertz, A., Bermano, A.H.: Clipcap: clip prefix for image captioning. arXiv preprint arXiv:2111.09734 (2021)

  26. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., Neubig, G.: Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9), 1–35 (2023)

    Article  Google Scholar 

  27. Ferragina, P., Scaiella, U.: TagMe: on-the-fly annotation of short text fragments (by Wikipedia entities). In: Proceedings of the 19th ACM International Conference on Information and Knowledge Management, pp. 1625–1628 (2010)

    Google Scholar 

  28. Vrandečić, D., Krötzsch, M.: WikiData: a free collaborative knowledgebase. Commun. ACM 57(10), 78–85 (2014)

    Article  Google Scholar 

  29. Chen, Y.: Convolutional neural network for sentence classification. Master’s thesis, University of Waterloo (2015)

    Google Scholar 

  30. Chung, J., Gulcehre, C., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014 Workshop on Deep Learning, December 2014 (2014)

    Google Scholar 

  31. Shu, K., Mahudeswaran, D., Wang, S., Lee, D., Liu, H.: FakeNewsnet: a data repository with news content, social context, and spatiotemporal information for studying fake news on social media. Big data 8(3), 171–188 (2020)

    Article  Google Scholar 

  32. Xiong, S., Zhang, G., Batra, V., Xi, L., Shi, L., Liu, L.: Trimoon: two-round inconsistency-based multi-modal fusion network for fake news detection. Inf. Fusion 93, 150–158 (2023)

    Article  Google Scholar 

  33. Liu, Z., et al.: Swin transformer v2: scaling up capacity and resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12009–12019 (2022)

    Google Scholar 

Download references

Acknowledgement

This work was supported by the National Natural Science Foundation of China (No. 62376062), the Ministry of Education of Humanities and Social Science Project (No. 23YJAZH220), the Philosophy and Social Sciences 14th Five-Year Plan Project of Guangdong Province (No. GD23CTS03), and the Guangdong Basic and Applied Basic Research Foundation of China (No. 2023A1515012718).

Author information

Authors and Affiliations

  1. School of Information Science and Technology, Guangdong University of Foreign Studies, Guangzhou, 510006, Guangdong, China

    Qiang Ouyang, Yongmei Zhou & Dong Zhou

  2. School of Computer Science and Technology, Guangdong University of Technology, Guangzhou, 510006, Guangdong, China

    Nankai Lin & Aimin Yang

  3. School of Computer Science and Intelligence Education, Lingnan Normal University, Zhanjiang, 524000, Guangdong, China

    Aimin Yang

Authors
  1. Qiang Ouyang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nankai Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yongmei Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aimin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dong Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dong Zhou .

Editor information

Editors and Affiliations

  1. University of New South Wales, Sydney, NSW, Australia

    Wenjie Zhang

  2. National University of Singapore, Queenstown, Singapore

    Anthony Tung

  3. Zhejiang Normal University, Jinhua, China

    Zhonglong Zheng

  4. University of New South Wales, Sydney, NSW, Australia

    Zhengyi Yang

  5. University of New South Wales, Sydney, NSW, Australia

    Xiaoyang Wang

  6. Zhejiang Normal University, Jinhua, China

    Hongjie Guo

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ouyang, Q., Lin, N., Zhou, Y., Yang, A., Zhou, D. (2024). Enhancing Few-Shot Multi-modal Fake News Detection Through Adaptive Fusion. In: Zhang, W., Tung, A., Zheng, Z., Yang, Z., Wang, X., Guo, H. (eds) Web and Big Data. APWeb-WAIM 2024. Lecture Notes in Computer Science, vol 14964. Springer, Singapore. https://doi.org/10.1007/978-981-97-7241-4_27

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-7241-4_27

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-7240-7

  • Online ISBN: 978-981-97-7241-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics