Skip to main content
SpringerLink
Log in

TMIF: transformer-based multi-modal interactive fusion for automatic rumor detection

  • Special Issue Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

The rapid development of social media platforms has made them one of the most important news sources. While it provides people with convenient real-time communication channels, fake news and rumors are also spread rapidly through social media platforms, misleading the public and even causing bad social impact. In view of the slow speed and poor consistency of artificial rumor detection, we propose an end-to-end automatic rumor detection model named TMIF, which is based on transformer to map multi-modal feature representations to the same data domain for fusion. It can capture the multi-level dependencies among multi-modal content while reducing the impact of multi-modal heterogeneity differences. We validated it on two multi-modal rumor detection datasets and proved the superior performance and early detection performance of the proposed model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Availability of data and materials

Not applicable.

References

  1. Kapoor, K.K., Tamilmani, K., Rana, N.P., Patil, P., Dwivedi, Y.K., Nerur, S.: Advances in social media research: past, present and future. Inf. Syst. Front. 20(3), 531–558 (2018). https://doi.org/10.1007/s10796-017-9810-y

    Article  Google Scholar 

  2. Jurgens, M., Helsloot, I.: The effect of social media on the dynamics of (self) resilience during disasters: a literature review. J. Conting. Crisis Manag. 26(1), 79–88 (2018). https://doi.org/10.1111/1468-5973.12212

    Article  Google Scholar 

  3. Ghani, N.A., Hamid, S., Hashem, I.A.T., Ahmed, E.: Social media big data analytics: a survey. Comput. Hum. Behav. 101, 417–428 (2019). https://doi.org/10.1016/j.chb.2018.08.039

    Article  Google Scholar 

  4. Monti, F., Frasca, F., Eynard, D., Mannion, D., Bronstein, M.M.: Fake news detection on social media using geometric deep learning. arXiv:1902.06673 (2019)

  5. Appel, G., Grewal, L., Hadi, R., Stephen, A.T.: The future of social media in marketing. J. Acad. Mark. Sci. 48(1), 79–95 (2020). https://doi.org/10.1007/s11747-019-00695-1

    Article  Google Scholar 

  6. Richard, K.: Predicting the future with social media. Int. J. Sci. Soc. 3(1), 33–39 (2021)

    Article  Google Scholar 

  7. Alalwan, A.A., Rana, N.P., Dwivedi, Y.K., Algharabat, R.: Social media in marketing: a review and analysis of the existing literature. Telemat. Inform. 34(7), 1177–1190 (2017). https://doi.org/10.1016/j.tele.2017.05.008

    Article  Google Scholar 

  8. Mheidly, N., Fares, J.: Leveraging media and health communication strategies to overcome the covid-19 infodemic. J. Public Health Policy 41(4), 410–420 (2020). https://doi.org/10.1057/s41271-020-00247-w

    Article  Google Scholar 

  9. Gao, J., Zheng, P., Jia, Y., Chen, H., Mao, Y., Chen, S., Wang, Y., Fu, H., Dai, J.: Mental health problems and social media exposure during covid-19 outbreak. Plos One (2020). https://doi.org/10.1371/journal.pone.0231924

    Article  Google Scholar 

  10. Wu, D., Cui, Y.: Disaster early warning and damage assessment analysis using social media data and geo-location information. Decis. Support Syst. 111, 48–59 (2018). https://doi.org/10.1016/j.dss.2018.04.005

    Article  Google Scholar 

  11. Lu, Y.-J., Li, C.-T.: GCAN: graph-aware co-attention networks for explainable fake news detection on social media. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 505–514. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.48

  12. Alkhodair, S.A., Ding, S.H., Fung, B.C., Liu, J.: Detecting breaking news rumors of emerging topics in social media. Inf. Process. Manag. 57(2), 102018 (2020). https://doi.org/10.1016/j.ipm.2019.02.016

    Article  Google Scholar 

  13. Alzanin, S.M., Azmi, A.M.: Detecting rumors in social media: a survey. Procedia Comput. Sci. 142, 294–300 (2018). https://doi.org/10.1016/j.procs.2018.10.495

    Article  Google Scholar 

  14. Choi, D., Chun, S., Oh, H., Han, J., Kwon, T., et al.: Rumor propagation is amplified by echo chambers in social media. Sci. Rep. 10(1), 1–10 (2020). https://doi.org/10.1038/s41598-019-57272-3

    Article  Google Scholar 

  15. Shahsavari, S., Holur, P., Wang, T., Tangherlini, T.R., Roychowdhury, V.: Conspiracy in the time of corona: automatic detection of emerging covid-19 conspiracy theories in social media and the news. J. Comput. Soc. Sci. 3(2), 279–317 (2020). https://doi.org/10.1007/s42001-020-00086-5

    Article  Google Scholar 

  16. Cao, J., Guo, J., Li, X., Jin, Z., Guo, H., Li, J.: Automatic rumor detection on microblogs: A survey. arXiv:1807.03505 (2018)

  17. Pathak, A.R., Mahajan, A., Singh, K., Patil, A., Nair, A.: Analysis of techniques for rumor detection in social media. Procedia Comput. Sci. 167, 2286–2296 (2020). https://doi.org/10.1016/j.procs.2020.03.281

    Article  Google Scholar 

  18. Meel, P., Vishwakarma, D.K.: Fake news, rumor, information pollution in social media and web: a contemporary survey of state-of-the-arts, challenges and opportunities. Expert Syst. Appl. 153, 112986 (2020). https://doi.org/10.1016/j.eswa.2019.112986

    Article  Google Scholar 

  19. Ma, J., Gao, W., Mitra, P., Kwon, S., Jansen, B.J., Wong, K.-F., Cha, M.: Detecting rumors from microblogs with recurrent neural networks. In: 25th International Joint Conference on Artificial Intelligence, IJCAI 2016, pp. 3818–3824 (2016)

  20. Lin, X., Liao, X., Xu, T., Pian, W., Wong, K.-F.: Rumor detection with hierarchical recurrent convolutional neural network. In: CCF International Conference on Natural Language Processing and Chinese Computing, pp. 338–348 (2019)

  21. Chen, T., Li, X., Yin, H., Zhang, J.: Call attention to rumors: deep attention based recurrent neural networks for early rumor detection. In: Ganji, M, Rashidi, L, Fung, BCM, Wang, C (eds.) Trends and Applications in Knowledge Discovery and Data Mining. Lecture Notes in Artificial Intelligence, vol. 11154, pp. 40–52. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04503-6_4

  22. Singh, J.P., Kumar, A., Rana, N.P., Dwivedi, Y.K.: Attention-based lstm network for rumor veracity estimation of tweets. Inf. Syst. Front. (2020). https://doi.org/10.1007/s10796-020-10040-5

    Article  Google Scholar 

  23. Ma, J., Gao, W., Wong, K.-F.: Rumor detection on twitter with tree-structured recursive neural networks. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 1980–1989 (2018)

  24. Guo, H., Cao, J., Zhang, Y., Guo, J., Li, J.: Rumor detection with hierarchical social attention network. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management. CIKM ’18, pp. 943–951. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3269206.3271709

  25. Li, Q., Zhang, Q., Si, L.: Rumor detection by exploiting user credibility information, attention and multi-task learning. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1173–1179. Association for Computational Linguistics, Florence (2019). https://doi.org/10.18653/v1/P19-1113

  26. Jin, Z., Cao, J., Guo, H., Zhang, Y., Luo, J.: Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM International Conference on Multimedia. MM ’17, pp. 795–816. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3123266.3123454

  27. Khattar, D., Goud, J.S., Gupta, M., Varma, V.: Mvae: Multimodal variational autoencoder for fake news detection. In: The World Wide Web Conference. WWW ’19, pp. 2915–2921. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3308558.3313552

  28. Wang, Y., Ma, F., Jin, Z., Yuan, Y., Xun, G., Jha, K., Su, L., Gao, J.: Eann: event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’18, pp. 849–857. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3219819.3219903

  29. Zhang, H., Fang, Q., Qian, S., Xu, C.: Multi-modal knowledge-aware event memory network for social media rumor detection. In: Proceedings of the 27th ACM International Conference on Multimedia. MM ’19, pp. 1942–1951. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3343031.3350850

  30. Duc Tuan, N.M., Quang Nhat Minh, P.: Multimodal fusion with bert and attention mechanism for fake news detection. In: 2021 RIVF International Conference on Computing and Communication Technologies, pp. 1–6 (2021). https://doi.org/10.1109/RIVF51545.2021.9642125

  31. Chen, J., Wu, Z., Yang, Z., Xie, H., Wang, F.L., Liu, W.: Multimodal fusion network with latent topic memory for rumor detection. In: 2021 IEEE International Conference on Multimedia and Expo, pp. 1–6 (2021). https://doi.org/10.1109/ICME51207.2021.9428404

  32. Sharma, S., Sharma, R.: Identifying possible rumor spreaders on twitter: A weak supervised learning approach. In: 2021 International Joint Conference on Neural Networks, pp. 1–8 (2021). https://doi.org/10.1109/IJCNN52387.2021.9534185

  33. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)

  34. Ilić, S., Marrese-Taylor, E., Balazs, J., Matsuo, Y.: Deep contextualized word representations for detecting sarcasm and irony. In: Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 2–7. Association for Computational Linguistics, Brussels (2018). https://doi.org/10.18653/v1/W18-6202

  35. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., Hon, H.-W.: Unified language model pre-training for natural language understanding and generation. In: Wallach, H, Larochelle, H, Beygelzimer, A, d’Alche-Buc, F, Fox, E, Garnett, R (eds.) Advances in Neural Information Processing Systems, vol. 32 (2019)

  36. Lee, J.-S., Hsiang, J.: Patent classification by fine-tuning Bert language model. World Patent Inf. 61, 101965 (2020). https://doi.org/10.1016/j.wpi.2020.101965

    Article  Google Scholar 

  37. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Guyon, I, Luxburg, UV, Bengio, S, Wallach, H, Fergus, R, Vishwanathan, S, Garnett, R (eds.) Advances in Neural Information Processing Systems, vol. 30 (2017)

  38. Dey, R., Salem, F.M.: Gate-variants of gated recurrent unit (GRU) neural networks. In: 2017 IEEE 60th International Midwest Symposium on Circuits and Systems, pp. 1597–1600 (2017). https://doi.org/10.1109/MWSCAS.2017.8053243

  39. Hara, K., Saito, D., Shouno, H.: Analysis of function of rectified linear unit used in deep learning. In: 2015 International Joint Conference on Neural Networks, pp. 1–8 (2015). https://doi.org/10.1109/IJCNN.2015.7280578

  40. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  41. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017). https://doi.org/10.1145/3065386

    Article  Google Scholar 

  42. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  43. Boididou, C., Papadopoulos, S., Kompatsiaris, Y., Schifferes, S., Newman, N.: Challenges of computational verification in social multimedia. In: Proceedings of the 23rd International Conference on World Wide Web, pp. 743–748 (2014). https://doi.org/10.1145/2567948.2579323

  44. Diederik, K., Jimmy, B., et al.: Adam: A method for stochastic optimization. arXiv:1412.6980, pp. 273–297 (2014)

Download references

Acknowledgements

The authors would like to thank the anonymous referees for their valuable comments and helpful suggestions. This work was supported by National Key R&D Program of China(No. 2019YFB1404700).

Funding

This work was supported by National Key R&D Program of China (no. 2019YFB1404700).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Qilu University of Technology (Shandong Academy of Sciences), 3501 Daxue Road, Changqing, 250300, Jinan, China

    Jiandong Lv, Xingang Wang & Cuiling Shao

Authors
  1. Jiandong Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xingang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cuiling Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JL, XW contributed to the conception of the study; JL performed the experiment; JL, XW contributed significantly to analysis and manuscript preparation; JL, XW, CS performed the data analyses and wrote the manuscript; JL, XW, CS helped perform the analysis with constructive discussions.)

Corresponding author

Correspondence to Xingang Wang.

Ethics declarations

Some journals require declarations to be submitted in a standardised format. Please check the Instructions for Authors of the journal to which you are submitting to see if you need to complete this section. If yes, your manuscript must contain the following sections under the heading ‘Declarations’.

Conflict of interest

(Check journal-specific guidelines for which heading to use): none.

Ethical approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Code availability

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lv, J., Wang, X. & Shao, C. TMIF: transformer-based multi-modal interactive fusion for automatic rumor detection. Multimedia Systems 29, 2979–2989 (2023). https://doi.org/10.1007/s00530-022-00916-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-022-00916-8

Keywords

Search

Navigation