Abstract
This work explores the potential to include visual information from images in social media campaign recognition. The diverse content shared on social media platforms, including text, photos, videos, and links, necessitates a multimodal analysis approach. With the emergence of Large Language Models (LLMs), there is now an opportunity to convert image content into textual descriptions, enabling the incorporation of previously text-based methods into a multimodal analysis. We evaluate this approach by conducting a parameter study to assess the resulting differences in image captions and a case study to examine the contribution of textualized image information to campaign recognition. The results indicate that, using image captions separate from or alongside tweet texts, connections between campaigns can be identified, and new campaigns detected.
The authors acknowledge support by the European Research Center in Information Systems (ERCIS) and by the project HybriD (FKZ: 16KIS1531K) funded through the German Federal Ministry of Education and Research.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Alayrac, J.B., et al.: Flamingo: a visual language model for few-shot learning. Adv. Neural. Inf. Process. Syst. 35, 23716–23736 (2022)
Assenmacher, D., Clever, L., Pohl, J.S., Trautmann, H., Grimme, C.: A two-phase framework for detecting manipulation campaigns in social media. In: Meiselwitz, G. (ed.) HCII 2020. LNCS, vol. 12194, pp. 201–214. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49570-1_14
Bao, H., Dong, L., Piao, S., Wei, F.: Beit: BERT pre-training of image transformers. arXiv:2106.08254 (2021)
Bellutta, D., Carley, K.M.: Investigating coordinated account creation using burst detection and network analysis. J. of Big Data 10(1), 1–17 (2023)
Chen, M., Radford, A., Child, R., Jun, H., Luan, D., Sutskever, I.: Generative pretraining from pixels. In: Proceedings of the 37th International Conference on Machine Learning, PMLR, pp. 1691–1703 (2020)
Cinelli, M., Cresci, S., Quattrociocchi, W., Zola, P.: Coordinated inauthentic behavior and information spreading on twitter. Decision Support Systems 160, 113819 (2022)
Eichenberg, C., Black, S., Weinbach, S., Parcalabescu, L., Frank, A.: MAGMA - multimodal augmentation of generative models through adapter-based finetuning. arXiv: 2112.05253 (2021)
Erhardt, K., Albassam, D.: Detecting the hidden dynamics of networked actors using temporal correlations. In: Companion Proceedings of the ACM Web Conference 2023, pp. 1214–1217. WWW 2023 Companion, ACM, Austin, TX, USA (2023)
Giachanou, A., Zhang, G., Rosso, P.: Multimodal multi-image fake news detection. In: 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pp. 647–654. IEEE, Sydney, Australia (2020)
Guo, B., Ding, Y., Yao, L., Liang, Y., Yu, Z.: The future of false information detection on social media. ACM Comput. Surv. 53(4), 1–36 (2020)
Jin, Z., Cao, J., Guo, H., Zhang, Y., Luo, J.: Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM International Conference on Multimedia, pp. 795–816. ACM, New York, NY, USA (2017)
John, J., Sherif, B.V.: Multi-model deepfake detection using deep and temporal features. In: Chen, J.IZ., Tavares, J.M.R.S., Shi, F. (eds.) Third International Conference on Image Processing and Capsule Networks. ICIPCN 2022. Lecture Notes in Networks and Systems, vol. 514, pp. 672–684. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-12413-6_53
Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: bootstrapping language-image pre-training with frozen image encoders and LLM. arXiv:2301.12597 (2023)
Li, J., Li, D., Xiong, C., Hoi, S.: BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning, pp. 12888–12900 (2022)
Pacheco, D., Hui, P.M., Torres-Lugo, C., Flammini, A., Menczer, F.: Uncovering coordinated networks on social media: methods and case studies. In: Proceedings of the 14th International Conference on Web and Social Media, pp. 455–466. AAAI Press, Online (2021)
Pohl, J.S., Markmann, S., Assenmacher, D., Grimme, C.: Invasion@Ukraine: providing and describing a twitter streaming dataset that captures the outbreak of war between Russia and Ukraine in 2022. In: Proceedings of the 17th International Conference on Web and Social Media, pp. 1093–1101. AAAI Press, Limassol, Cyprus (2023)
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763 (2021)
Rafique, R., Gantassi, R., Amin, R., Frnda, J., Mustapha, A., Alshehri, A.H.: Deep fake detection and classification using error-level analysis and deep learning. Sci. Rep. 13, 7422 (2023)
Tsimpoukelli, M., Menick, J.L., Cabi, S., Eslami, S., Vinyals, O., Hill, F.: Multimodal few-shot learning with frozen language models. Adv. Neural. Inf. Process. Syst. 34, 200–212 (2021)
Uppada, S.K., Patel, P., B., S.: An image and text-based multimodal model for detecting fake news in OSN’s. J. Intell. Inf. Syst. (2022). https://doi.org/10.1007/s10844-022-00764-y
Wang, Y., Ma, F., Jin, Z., Yuan, Y., Xun, G., et al.: EANN: event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th International Conference on Knowledge Discovery & Data Mining, pp. 849–857. ACM, NY, USA (2018)
Wang, Z., Yu, J., Yu, A.W., Dai, Z., Tsvetkov, Y., Cao, Y.: SimVLM: simple visual language model pretraining with weak supervision. arXiv:2108.10904 (2021)
Weber, D., Neumann, F.: Amplifying influence through coordinated behaviour in social networks. Soc. Netw. Anal. Min. 11, 111 (2021)
Xue, J., Wang, Y., Tian, Y., Li, Y., Shi, L.: Detecting fake news by exploring the consistency of multimodal data. Inf. Process. Manage. 58(5), 102610 (2021)
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., Wu, Y.: CoCa: contrastive captioners are image-text foundation models. arXiv:2205.01917 (2022)
Zhang, X., Dadkhah, S., Weismann, A.G., Kanaani, M.A., Ghorbani, A.A.: Multimodal fake news analysis based on image-text similarity. IEEE Trans. Comput. Soc. Syst., pp. 1–14 (2023)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Stampe, L., Pohl, J., Grimme, C. (2023). Towards Multimodal Campaign Detection: Including Image Information in Stream Clustering to Detect Social Media Campaigns. In: Ceolin, D., Caselli, T., Tulin, M. (eds) Disinformation in Open Online Media. MISDOOM 2023. Lecture Notes in Computer Science, vol 14397. Springer, Cham. https://doi.org/10.1007/978-3-031-47896-3_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-47896-3_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47895-6
Online ISBN: 978-3-031-47896-3
eBook Packages: Computer ScienceComputer Science (R0)