Skip to main content
SpringerLink
Log in

AMAE: Adversarial multimodal auto-encoder for crisis-related tweet analysis

  • Regular Paper
  • Published:
Computing Aims and scope Submit manuscript

Abstract

Social media platforms have grown in importance as sources of information and as a complement to traditional media. If informative and relevant tweets on social media platforms can be effectively detected and analyzed during crisis events, it will help humanitarian organizations with situational awareness and planning relief activities. In this work, we propose an adversarial multimodal auto-encoder model for detecting and analyzing crisis-related tweets, which analyzes the complex multimodal content of tweets and integrates adversarial strategies to generate a joint representation containing information from multiple sources. Through extensive experiments on real datasets, we demonstrate the superior performance of the proposed model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data Availability Statement

Not applicable.

References

  1. Zhang C, Fan C, Yao W, Hu X, Mostafavi A (2019) Social media for intelligent public information and warning in disasters: An interdisciplinary review. Int J Inf Manage 49:190–207. https://doi.org/10.1016/j.ijinfomgt.2019.04.004

    Article  Google Scholar 

  2. Liu Y, Wu Y-F (2018) Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. Proceedings of the AAAI Conference on Artificial Intelligence 32(1)

  3. Aldwairi M, Alwahedi A (2018) Detecting fake news in social media networks. Procedia Computer Science 141:215–222. https://doi.org/10.1016/j.procs.2018.10.171

    Article  Google Scholar 

  4. Shu K, Wang S, Liu H (2018) Understanding user profiles on social media for fake news detection. In: 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 430–435 https://doi.org/10.1109/MIPR.2018.00092

  5. Takahashi B, Tandoc EC, Carmichael C (2015) Communicating on twitter during a disaster: An analysis of tweets during typhoon haiyan in the philippines. Comput Hum Behav 50:392–398. https://doi.org/10.1016/j.chb.2015.04.020

    Article  Google Scholar 

  6. Chen Q, Wang W, Huang K, De S, Coenen F (2020) Multi-modal adversarial training for crisis-related data classification on social media. In: 2020 IEEE International Conference on Smart Computing (SMARTCOMP), pp. 232–237 https://doi.org/10.1109/SMARTCOMP50058.2020.00051

  7. Meurisch C, Hamza Z, Bayrak B, Mühlhäuser M (2019) Enhanced detection of crisis-related microblogs by spatiotemporal feedback loops. In: 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), vol. 1, pp. 507–512 https://doi.org/10.1109/COMPSAC.2019.00078

  8. Nazer TH, Xue G, Ji Y, Liu H (2017) Intelligent disaster response via social media analysis a survey. SIGKDD Explor. Newsl. 19(1):46–59. https://doi.org/10.1145/3137597.3137602

    Article  Google Scholar 

  9. Reuter C, Hughes AL, Kaufhold M-A (2018) Social media in crisis management: An evaluation and analysis of crisis informatics research. International Journal of Human-Computer Interaction 34(4):280–294. https://doi.org/10.1080/10447318.2018.1427832

    Article  Google Scholar 

  10. Cresci S, Cimino A, Dell’Orletta F, Tesconi M (2015) Crisis mapping during natural disasters via text analysis of social media messages. Web Information Systems Engineering - WISE 2015. Springer, Cham, pp 250–258

  11. Liu J, Singhal T, Blessing LTM, Wood KL, Lim KH (2021) Crisisbert: A robust transformer for crisis classification and contextual crisis embedding. In: Proceedings of the 32nd ACM Conference on Hypertext and Social Media. HT ’21, pp. 133–141. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3465336.3475117

  12. Zhang M, Zhang Y, Fu G (2016) Tweet sarcasm detection using deep neural network. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 2449–2460. The COLING 2016 Organizing Committee, Osaka, Japan

  13. Ozdikis O, Senkul P, Oguztuzun H (2012) Semantic expansion of tweet contents for enhanced event detection in twitter. In: 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, pp. 20–24 https://doi.org/10.1109/ASONAM.2012.14

  14. Paul NR, Sahoo D, Balabantaray RC (2022) Classification of crisis-related data on twitter using a deep learning-based framework. Multimedia Tools and Applications 1–21. https://doi.org/10.1007/s11042-022-12183-w

  15. Nguyen DT, Alam F, Ofli F, Imran M (2017) Automatic image filtering on social networks using deep learning and perceptual hashing during crises. arXiv preprint arXiv:1704.02602

  16. Neppalli VK, Caragea C, Caragea D (2018) Deep neural networks versus naive bayes classifiers for identifying informative tweets during disasters. In: Proceedings of the 15th Annual Conference for Information Systems for Crisis Response and Management (ISCRAM)

  17. Alam F, Joty S, Imran M (2018) Graph based semi-supervised learning with convolution neural networks to classify crisis related tweets. In: Twelfth International AAAI Conference on Web and Social Media

  18. Alam F, Imran M, Ofli F (2017) Image4act: Online social media image processing for disaster response. In: Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017. ASONAM ’17, pp. 601–604. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3110025.3110164

  19. Ma J, Gao W, Mitra P, Kwon S, Jansen BJ, Wong K-F, Cha M (2016) Detecting rumors from microblogs with recurrent neural networks. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. IJCAI’16, pp. 3818–3824

  20. Ma J, Gao W, Wong K-F (2018) Rumor detection on twitter with tree-structured recursive neural networks. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1980–1989. Association for Computational Linguistics, Melbourne, Australia. https://doi.org/10.18653/v1/P18-1184

  21. Gao W, Li L, Zhu X, Wang Y (2020) Detecting disaster-related tweets via multimodal adversarial neural network. IEEE Multimedia 27(4):28–37. https://doi.org/10.1109/MMUL.2020.3012675

    Article  Google Scholar 

  22. Jin Z, Cao J, Guo H, Zhang Y, Luo J (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM International Conference on Multimedia. MM ’17, pp. 795–816. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3123266.3123454

  23. Wang Y, Ma F, Jin Z, Yuan Y, Xun G, Jha K, Su L, Gao J (2018) Eann: Event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’18, pp. 849–857. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3219819.3219903

  24. Abavisani M, Wu L, Hu S, Tetreault J, Jaimes A (2020) Multimodal categorization of crisis events in social media. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

  25. Khattar D, Goud JS, Gupta M, Varma V (2019) Mvae: Multimodal variational autoencoder for fake news detection. In: The World Wide Web Conference. WWW ’19, pp. 2915–2921. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3308558.3313552

  26. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  27. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJ, Bottou L, Weinberger KQ (eds.) Advances in Neural Information Processing Systems, vol. 25

  28. Devlin J, Chang M-W, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  29. Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078

  30. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Lu, Polosukhin I (2017) Attention is all you need. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds.) Advances in Neural Information Processing Systems, vol. 30

  31. Baldi P, Sadowski PJ (2013) Understanding dropout. In: Burges CJ, Bottou L, Welling M, Ghahramani Z, Weinberger KQ (eds.) Advances in Neural Information Processing Systems, vol. 26

  32. Agostinelli F, Hoffman M, Sadowski P, Baldi P (2014) Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830

  33. Creswell A, White T, Dumoulin V, Arulkumaran K, Sengupta B, Bharath AA (2018) Generative adversarial networks: An overview. IEEE Signal Process Mag 35(1):53–65. https://doi.org/10.1109/MSP.2017.2765202

    Article  Google Scholar 

  34. Alam F, Ofli F, Imran M (2018) Crisismmd: Multimodal twitter datasets from natural disasters. Proceedings of the International AAAI Conference on Web and Social Media 12(1)

  35. Ofli F, Alam F, Imran M (2020) Analysis of social media data using multimodal deep learning for disaster response. arXiv preprint arXiv:2004.11838

  36. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

Download references

Acknowledgements

The authors would like to thank the anonymous referees for their valuable comments and helpful suggestions. This work was supported by National Key R &D Program of China(No. 2019YFB1404700).

Funding

This work was supported by National Key R &D Program of China(No. 2019YFB1404700).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Qilu University of Technology (Shandong Academy of Sciences), 3501 Daxue Road, Changqing, 250300, Jinan, China

    Jiandong Lv, Xingang Wang & Cuiling Shao

Authors
  1. Jiandong Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xingang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cuiling Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Jiandong Lv, Xingang Wang contributed to the conception of the study; Jiandong Lv performed the experiment; Jiandong Lv, Xingang Wang contributed significantly to analysis and manuscript preparation; Jiandong Lv, Xingang Wang,Cuiling Shao performed the data analyses and wrote the manuscript; Jiandong Lv, Xingang Wang,Cuiling Shao helped perform the analysis with constructive discussions.

Corresponding author

Correspondence to Xingang Wang.

Ethics declarations

Conflict of interest

None.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Code availability

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lv, J., Wang, X. & Shao, C. AMAE: Adversarial multimodal auto-encoder for crisis-related tweet analysis. Computing 105, 13–28 (2023). https://doi.org/10.1007/s00607-022-01098-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00607-022-01098-x

Keywords

Search

Navigation