Skip to main content
SpringerLink
Log in

A RoBERTa based model for identifying the multi-modal informative tweets during disaster

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Detecting informative tweets is very important to the government or non-government organizations during a disaster. Most of the literature works focused on either text or image separately for getting informative tweets. A very few existing works used multi-modal information such as both image and text to identify the informative tweets. However, the existing works do not give much performance on multi-modal informative tweets. There is a chance to lose useful information in critical times. Hence, we propose a novel approach to identify the multi-modal informative tweets during a disaster. Our proposed method comprises the pre-trained RoBERTa and VGG-16 models to extract the text and image features, respectively. The outputs of these two models are combined using a multiplicative fusion technique. Experiments are conducted on diverse disaster datasets such as Hurricane Maria, Hurricane Harvey, California wildfires, Iraq-Iran earthquake, Hurricane Irma, and Mexico earthquake. Experimental results demonstrated that the proposed method outperforms the existing baseline methods on various parameters.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Aipe A, Mukuntha N, Ekbal A, Kurohashi S (2018) Deep learning approach towards multi-label classification of crisis related tweets. In: Proceedings of the 15th ISCRAM conference

  2. Alam F, Joty S, Imran M (2018) Graph based semi-supervised learning with convolution neural networks to classify crisis related tweets. In: Twelfth international AAAI conference on web and social media

  3. Alam F, Ofli F, Imran M (2018) Crisismmd: multimodal twitter datasets from natural disasters. In: AAAI conference on web and social media (ICWSM). AAAI, AAAI, Stanford, California, USA

  4. Alam F, Ofli F, Imran M (2019) Descriptive and visual summaries of disaster events using artificial intelligence techniques: case studies of hurricanes harvey, irma, and maria. Behav Inf Technol, 1–31

  5. Ashktorab Z, Brown C, Nandi M, Culotta A (2014) Tweedr: mining twitter to inform disaster response. In: ISCRAM, pp 269–272. Citeseer

  6. Avgerinakis K, Moumtzidou A, Andreadis S, Michail E, Gialampoukidis I, Vrochidis S, Kompatsiaris I (2017) Visual and textual analysis of social media and satellite images for flood detection@ multimedia satellite task mediaeval 2017. In: MediaEval

  7. Batool R, Khattak AM, Maqbool J, Lee S (2013) Precise tweet classification and sentiment analysis. In: 2013 IEEE/ACIS 12th international conference on computer and information science (ICIS), pp 461–466. IEEE

  8. Berahmand K, Mohammadi M, Faroughi A, Mohammadiani RP (2022) A novel method of spectral clustering in attributed networks by constructing parameter-free affinity matrix. Clust Comput 25(2):869–888

    Article  Google Scholar 

  9. Caragea C, Silvescu A, Tapia AH (2016) Identifying informative messages in disaster events using convolutional neural networks. In: International conference on information systems for crisis response and management, pp 137–147

  10. Caragea C, Squicciarini AC, Stehle S, Neppalli K, Tapia AH (2014) Mapping moods: geo-mapped sentiment analysis during hurricane sandy. In: ISCRAM

  11. Da Silva NF, Hruschka ER, Hruschka ER Jr (2014) Tweet sentiment analysis with classifier ensembles. Decis Support Syst 66:170–179

    Article  Google Scholar 

  12. Fan C, Wu F, Mostafavi A (2020) A hybrid machine learning pipeline for automated mapping of events and locations from social media in disasters. IEEE Access

  13. Gautam AK, Misra L, Kumar A, Misra K, Aggarwal S, Shah RR (2019) Multimodal analysis of disaster tweets. In: 2019 IEEE Fifth international conference on multimedia big data (BigMM), pp 94–103. IEEE

  14. Houston JB, Hawthorne J, Perreault MF, Park EH, Goldstein Hode M, Halliwell MR, Turner McGowen SE, Davis R, Vaid S, McElderry JA et al (2015) Social media and disasters: a functional framework for social media use in disaster planning, response, and research. Disasters 39(1):1–22

    Article  Google Scholar 

  15. Li Q, Shah S, Fang R, Nourbakhsh A, Liu X (2016) Tweet sentiment analysis by incorporating sentiment-specific word embedding and weighted text features. In: 2016 IEEE/WIC/ACM International conference on web intelligence (WI), pp 568–571. IEEE

  16. Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: a robustly optimized bert pretraining approach. arXiv:1907.11692

  17. Madichetty S, Sridevi M (2019) Detecting informative tweets during disaster using deep neural networks. In: 2019 11th international conference on communication systems & networks (COMSNETS), pp 709–713. IEEE

  18. Maiya A S (2020) ktrain: a low-code library for augmented machine learning. arXiv:2004.10703 [cs.LG]

  19. Moumtzidou A, Giannakeris P, Andreadis S, Mavropoulos A, Meditskos G, Gialampoukidis I, Avgerinakis K, Vrochidis S, Kompatsiaris I (2018) A multimodal approach in estimating road passability through a flooded area using social media and satellite images. In: MediaEval

  20. Nalluru G, Pandey R, Purohit H (2019) Classifying relevant social media posts during disasters using ensemble of domain-agnostic and domain-specific word embeddings. arXiv:1911.05165

  21. Nguyen DT, Ofli F, Imran M, Mitra P (2017) Damage assessment from social media imagery data during disasters. In: Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017, pp 569–576. ACM

  22. Nizzoli L, Avvenuti M, Cresci S, Tesconi M (2019) Extremist propaganda tweet classification with deep learning in realistic scenarios. In: Proceedings of the 10th ACM conference on web science, pp 203–204

  23. Rostami M, Berahmand K, Forouzandeh S (2020) A novel method of constrained feature selection by the measurement of pairwise constraints uncertainty. J Big Data 7(1):1–21

    Article  Google Scholar 

  24. Rostami M, Berahmand K, Nasiri E, Forouzandeh S (2021) Review of swarm intelligence-based feature selection methods. Eng Appl Artif Intel 100:104210

    Article  Google Scholar 

  25. Rudra K, Ganguly N, Goyal P, Ghosh S (2018) Extracting and summarizing situational information from the twitter social media during disasters. ACM Trans Web (TWEB) 12(3):17

    Google Scholar 

  26. Rudra K, Sharma A, Ganguly N, Ghosh S (2016) Characterizing communal microblogs during disaster events. In: 2016 IEEE/ACM International conference on advances in social networks analysis and mining (ASONAM), pp 96–99. IEEE

  27. Siddiqua UA, Chy AN, Aono M (2019) Tweet stance detection using an attention based neural ensemble model. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp 1868–1873

  28. Silva NFFD, Coletta LF, Hruschka ER (2016) A survey and comparative study of tweet sentiment analysis via semi-supervised learning. ACM Comput Surv (CSUR) 49(1):1–26

    Google Scholar 

  29. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  30. Stefan I, Rebedea T, Caragea D (2019) Classification of eyewitness tweets in emergency situations. In: RoCHI, pp 46–52

  31. Stowe K, Paul M, Palmer M, Palen L, Anderson KM (2016) Identifying and categorizing disaster-related tweets. In: Proceedings of the fourth international workshop on natural language processing for social media, pp 1–6

  32. To H, Agrawal S, Kim SH, Shahabi C (2017) On identifying disaster-related tweets: matching-based or learning-based?. In: 2017 IEEE Third international conference on multimedia big data (BigMM), pp 330–337. IEEE

Download references

Author information

Authors and Affiliations

  1. Centific, Hyderabad, India

    Sreenivasulu Madichetty

  2. National Institute of Technology, Tiruchirappalli, India

    Sridevi M

  3. Endicott College of International Studies, Woosong University, Daejeon, South Korea

    Sreekanth Madisetty

  4. Jio Platforms Limited, Hyderabad, Telangana, India

    Sreekanth Madisetty

Authors
  1. Sreenivasulu Madichetty
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sridevi M
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sreekanth Madisetty
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sreenivasulu Madichetty.

Ethics declarations

Competing interests

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Madichetty, S., M, S. & Madisetty, S. A RoBERTa based model for identifying the multi-modal informative tweets during disaster. Multimed Tools Appl 82, 37615–37633 (2023). https://doi.org/10.1007/s11042-023-14780-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14780-9

Keywords

Search

Navigation