Abstract
The growth of unverified multimodal content on microblogging sites has emerged as a challenging problem in recent times. One major roadblock to this problem is the unavailability of automated tools for rumour detection. Previous work in this field mainly involves rumour detection for textual content only. As per recent studies, the incorporation of multiple modalities (text and image) is provably useful in many tasks since it enhances the understanding of the context. This paper introduces a novel multimodal architecture for rumour detection. It consists of two attention-based BiLSTM neural networks for the generation of text and image feature representations, fused using a cross-modal fusion block and ultimately passing through the rumour detection module. To establish the efficiency of the proposed approach, we extend the existing PHEME-2016 data set by collecting available images and in case of non-availability, additionally downloading new images from the Web. Experiments show that our proposed architecture outperforms state-of-the-art results by a large margin.
R. Kumar and R. Sinha—These authors contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
Unfortunately, that dataset was not made public.
- 3.
- 4.
References
Bai, N., Meng, F., Rui, X., Wang, Z.: Rumour detection based on graph convolutional neural net. IEEE Access 1 (2021). https://doi.org/10.1109/ACCESS.2021.3050563
Bai, Y., Yi, J., Tao, J., Tian, Z., Wen, Z., Zhang, S.: Fast end-to-end speech recognition via non-autoregressive models and cross-modal knowledge transferring from bert. IEEE/ACM Trans. Audio, Speech and Lang. Proc. 29, 1897–1911 (2021). https://doi.org/10.1109/TASLP.2021.3082299
Borth, D., Ji, R., Chen, T., Breuel, T., Chang, S.F.: Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: Proceedings of the 21st ACM International Conference on Multimedia, MM 2013, pp. 223–232. Association for Computing Machinery, New York, NY, USA (2013). https://doi.org/10.1145/2502081.2502282
Chen, Y., Sui, J., Hu, L., Gong, W.: Attention-residual network with CNN for rumor detection. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, pp. 1121–1130. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3357384.3357950
Cheung, T.H., Lam, K.M.: Transformer-graph neural network with global-local attention for multimodal rumour detection with knowledge distillation (2022). https://doi.org/10.48550/ARXIV.2206.04832, https://arxiv.org/abs/2206.04832
Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1800–1807. IEEE Computer Society, Los Alamitos, CA, USA, July 2017. https://doi.org/10.1109/CVPR.2017.195, https://doi.org/ieeecomputersociety.org/10.1109/CVPR.2017.195
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota, June 2019. https://doi.org/10.18653/v1/N19-1423, https://aclanthology.org/N19-1423
Ghani, N.A., Hamid, S., Targio Hashem, I.A., Ahmed, E.: Social media big data analytics: a survey. Computers in Human Behavior 101, 417–428 (2019). https://doi.org/10.1016/j.chb.2018.08.039, https://www.sciencedirect.com/science/article/pii/S074756321830414X
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
Jin, Z., Cao, J., Guo, H., Zhang, Y., Luo, J.: Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM International Conference on Multimedia, MM 2017, pp. 795–816. Association for Computing Machinery, New York, NY, USA (2017). https://doi.org/10.1145/3123266.3123454, https://doi.org/10.1145/3123266.3123454
Kwon, S., Cha, M., Jung, K., Chen, W., Wang, Y.: Prominent features of rumor propagation in online social media. In: 2013 IEEE 13th International Conference on Data Mining, pp. 1103–1108 (2013). https://doi.org/10.1109/ICDM.2013.61
Liu, T., Lam, K., Zhao, R., Qiu, G.: Deep cross-modal representation learning and distillation for illumination-invariant pedestrian detection. IEEE Trans. Circ. Syst. Video Technol. 32(1), 315–329 (2022). https://doi.org/10.1109/TCSVT.2021.3060162
Liu, Y., et al.: Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
Ma, J., et al.: Detecting rumors from microblogs with recurrent neural networks. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, pp. 3818–3824. AAAI Press (2016)
Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., Peng, X.: Smil: multimodal learning with severely missing modality (2021). https://doi.org/10.48550/ARXIV.2103.05677, https://arxiv.org/abs/2103.05677
Mukherjee, R., et al.: MTLTS: a multi-task framework to obtain trustworthy summaries from crisis-related microblogs. In: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, ACM, February 2022. https://doi.org/10.1145/3488560.3498536
Nguyen, D.Q., Vu, T., Nguyen, A.T.: BERTweet: a pre-trained language model for English Tweets. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 9–14 (2020)
Pathak, A.R., Mahajan, A., Singh, K., Patil, A., Nair, A.: Analysis of techniques for rumor detection in social media. Procedia Comput. Sci. 167, 2286–2296 (2020). https://doi.org/10.1016/j.procs.2020.03.281, https://www.sciencedirect.com/science/article/pii/S187705092030747X, international Conference on Computational Intelligence and Data Science
Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543. Association for Computational Linguistics, Doha, Qatar, October 2014. https://doi.org/10.3115/v1/D14-1162, https://aclanthology.org/D14-1162
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (2015)
Song, C., Yang, C., Chen, H., Tu, C., Liu, Z., Sun, M.: Ced: credible early detection of social media rumors. IEEE Trans. Knowl. Data Eng. 33(8), 3035–3047 (2021). https://doi.org/10.1109/TKDE.2019.2961675
Sun, S., Cheng, Y., Gan, Z., Liu, J.: Patient knowledge distillation for BERT model compression. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4323–4332. Association for Computational Linguistics, Hong Kong, China, November 2019. https://doi.org/10.18653/v1/D19-1441, https://aclanthology.org/D19-1441
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826 (2016). https://doi.org/10.1109/CVPR.2016.308
Takahashi, T., Igata, N.: Rumor detection on twitter. In: The 6th International Conference on Soft Computing and Intelligent Systems, and The 13th International Symposium on Advanced Intelligence Systems, pp. 452–457 (2012)
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.: Hierarchical attention networks for document classification. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480–1489. Association for Computational Linguistics, San Diego, California, June 2016. https://doi.org/10.18653/v1/N16-1174, https://aclanthology.org/N16-1174
Zubiaga, A., Aker, A., Bontcheva, K., Liakata, M., Procter, R.: Detection and resolution of rumours in social media. ACM Comput. Surv. 51(2), 1–36 (2018). https://doi.org/10.1145/3161603
Zubiaga, A., Liakata, M., Procter, R.: Exploiting context for rumour detection in social media. In: Ciampaglia, G.L., Mashhadi, A., Yasseri, T. (eds.) SocInfo 2017. LNCS, vol. 10539, pp. 109–123. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67217-5_8
Acknowledgements
Raghvendra Kumar would like to express his heartfelt gratitude to the Technology Innovation Hub (TIH), Vishlesan I-Hub Foundation, IIT Patna for providing the Chanakya Fellowship, which has been instrumental in supporting his research endeavours. Dr. Sriparna Saha gratefully acknowledges the Young Faculty Research Fellowship (YFRF) Award, supported by Visvesvaraya Ph.D. Scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia) for carrying out this research.
Author information
Authors and Affiliations
Contributions
Raghvendra Kumar, Ritika Sinha : These authors contributed equally to this work.
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kumar, R., Sinha, R., Saha, S., Jatowt, A. (2023). Multimodal Rumour Detection: Catching News that Never Transpired!. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds) Document Analysis and Recognition - ICDAR 2023. ICDAR 2023. Lecture Notes in Computer Science, vol 14189. Springer, Cham. https://doi.org/10.1007/978-3-031-41682-8_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-41682-8_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-41681-1
Online ISBN: 978-3-031-41682-8
eBook Packages: Computer ScienceComputer Science (R0)
