Skip to main content
SpringerLink
Log in

Image-Enhanced Multi-Modal Representation for Local Topic Detection from Social Media

  • Conference paper
  • First Online:
Book cover Database Systems for Advanced Applications (DASFAA 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12682))

Included in the following conference series:

Abstract

Detecting local topics from social media is an important task for many applications, ranging from event tracking to emergency warning. Recent years have witnessed growing interest in leveraging multi-modal social media information for local topic detection. However, existing methods suffer great limitation in capturing comprehensive semantics from social media and fall short in bridging semantic gaps among multi-modal contents, i.e., some of them overlook visual information which contains rich semantics, others neglect indirect semantic correlation among multi-modal information. To deal with above problems, we propose an effective local topic detection method with two major modules, called IEMM-LTD. The first module is an image-enhanced multi-modal embedding learner to generate embeddings for words and images, which can capture comprehensive semantics and preserve both direct and indirect semantic correlations. The second module is an embedding based topic model to detect local topics represented by both words and images, which adopts different prior distributions to model multi-modal information separately and can find the number of topics automatically. We evaluate the effectiveness of IEMM-LTD on two real-world tweet datasets, the experimental results show that IEMM-LTD has achieved the best performance compared to the existing state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://docs.tweepy.org/en/latest/.

References

  1. Anderson, P., et al.: Bottom-up and top-down attention for image captioning and visual question answering. In: CVPR, pp. 6077–6086. IEEE Computer Society (2018)

    Google Scholar 

  2. Bao, J., Duan, N., Zhou, M.: Knowledge-based question answering as machine translation. In: ACL (1). The Association for Computer Linguistics (2014)

    Google Scholar 

  3. Batmanghelich, K.N., Saeedi, A., Narasimhan, K., Gershman, S.: Nonparametric spherical topic modeling with word embeddings. In: ACL (2). The Association for Computer Linguistics (2016)

    Google Scholar 

  4. Chen, F., Xie, S., Li, X., Li, S.: What topics do images say: A neural image captioning model with topic representation. In: ICME Workshops, IEEE (2019)

    Google Scholar 

  5. Chen, J., Gao, N., Xue, C., Tu, C., Zha, D.: Perceiving topic bubbles: local topic detection in spatio-temporal tweet stream. In: Li, G., Yang, J., Gama, J., Natwichai, J., Tong, Y. (eds.) DASFAA 2019. LNCS, vol. 11447, pp. 730–747. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-18579-4_43

    Chapter  Google Scholar 

  6. Chen, J., Gao, N., Xue, C., Zhang, Y.: The application of network based embedding in local topic detection from social media. In: ICTAI, pp. 1311–1319. IEEE (2019)

    Google Scholar 

  7. Cheng, Z., Bai, F., Xu, Y., Zheng, G., Pu, S., Zhou, S.: Focusing attention: Towards accurate text recognition in natural images. In: ICCV, pp. 5086–5094. IEEE Computer Society (2017)

    Google Scholar 

  8. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (2019)

    Google Scholar 

  9. Duan, J., Ai, Y., Li, X.: LDA topic model for microblog recommendation. In: IALP, pp. 185–188. IEEE (2015)

    Google Scholar 

  10. Fang, A., Macdonald, C., Ounis, I., Habel, P.: Topics in tweets: a user study of topic coherence metrics for twitter data. In: Ferro, N. (ed.) ECIR 2016. LNCS, vol. 9626, pp. 492–504. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-30671-1_36

    Chapter  Google Scholar 

  11. Giannakopoulos, K., Chen, L.: Incremental and adaptive topic detection over social media. In: Pei, J., Manolopoulos, Y., Sadiq, S., Li, J. (eds.) DASFAA 2018. LNCS, vol. 10827, pp. 460–473. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91452-7_30

    Chapter  Google Scholar 

  12. Huang, S., Chen, H., Dai, X., Chen, J.: Non-linear learning for statistical machine translation. In: ACL (1). The Association for Computer Linguistics (2015)

    Google Scholar 

  13. Jégou, S., Drozdzal, M., Vázquez, D., Romero, A., Bengio, Y.: The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In: CVPR Workshops, pp. 1175–1183 (2017)

    Google Scholar 

  14. Li, G., Zhu, L., Liu, P., Yang, Y.: Entangled transformer for image captioning. In: ICCV, pp. 8927–8936. IEEE (2019)

    Google Scholar 

  15. Liu, H., Ge, Y., Zheng, Q., Lin, R., Li, H.: Detecting global and local topics via mining twitter data. Neurocomputing 273 120-132 (2018)

    Google Scholar 

  16. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR, pp. 3431–3440 (2015)

    Google Scholar 

  17. Morstatter, F., Liu, H.: In search of coherence and consensus: measuring the interpretability of statistical topics. J. Mach. Learn. Res. 18(169), 1–32 (2017)

    MathSciNet  MATH  Google Scholar 

  18. Muhammad, U., Wang, W., Chattha, S.P., Ali, S.: Pre-trained vggnet architecture for remote-sensing image scene classification. In: ICPR, pp. 1622–1627. IEEE Computer Society (2018)

    Google Scholar 

  19. Qian, S., Zhang, T., Xu, C., Shao, J.: Multi-modal event topic model for social event analysis. IEEE Trans. Multimedia 18(2), 233–246 (2016)

    Article  Google Scholar 

  20. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)

    Google Scholar 

  21. Singh, V.K., et al.: Classification of breast cancer molecular subtypes from their micro-texture in mammograms using a vggnet-based convolutional neural network. In: CCIA, vol. 300, pp. 76–85. IOS Press (2017)

    Google Scholar 

  22. Vaswani, A., et al.: Attention is all you need. In: NIPS, pp. 5998–6008 (2017)

    Google Scholar 

  23. Wold, H.M., Vikre, L., Gulla, J.A., Özgöbek, Ö., Su, X.: Twitter topic modeling for breaking news detection. In: WEBIST (2), pp. 211–218. SciTePress (2016)

    Google Scholar 

  24. Yan, X., Guo, J., Lan, Y., Cheng, X.: A biterm topic model for short texts. In: WWW, pp. 1445–1456. International World Wide Web Conferences Steering Committee/ACM (2013)

    Google Scholar 

  25. Zhang, C., et al.: Regions, periods, activities: Uncovering urban dynamics via cross-modal representation learning. In: WWW, pp. 361–370. ACM (2017)

    Google Scholar 

  26. Zhang, C., Lu, S., Zhang, C., Xiao, X., Wang, Q., Chen, G.: A novel hot topic detection framework with integration of image and short text information from twitter. IEEE Access 7, 9225–9231 (2019)

    Article  Google Scholar 

  27. Zhao, W.X., et al.: Comparing Twitter and traditional media using topic models. In: Clough, P. (ed.) ECIR 2011. LNCS, vol. 6611, pp. 338–349. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20161-5_34

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of Information Security, Chinese Academy of Sciences, Beijing, China

    Junsha Chen, Neng Gao, Yifei Zhang & Chenyang Tu

  2. School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

    Junsha Chen & Yifei Zhang

  3. Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China

    Junsha Chen, Neng Gao, Yifei Zhang & Chenyang Tu

Authors
  1. Junsha Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Neng Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yifei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chenyang Tu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chenyang Tu .

Editor information

Editors and Affiliations

  1. Aalborg University, Aalborg, Denmark

    Christian S. Jensen

  2. Singapore Management University, Singapore, Singapore

    Ee-Peng Lim

  3. Academia Sinica, Taipei, Taiwan

    De-Nian Yang

  4. The Pennsylvania State University, University Park, PA, USA

    Wang-Chien Lee

  5. National Chiao Tung University, Hsinchu, Taiwan

    Vincent S. Tseng

  6. Athens University of Economics and Business, Athens, Greece

    Vana Kalogeraki

  7. National Cheng Kung University, Tainan City, Taiwan

    Jen-Wei Huang

  8. National Tsing Hua University, Hsinchu, Taiwan

    Chih-Ya Shen

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, J., Gao, N., Zhang, Y., Tu, C. (2021). Image-Enhanced Multi-Modal Representation for Local Topic Detection from Social Media. In: Jensen, C.S., et al. Database Systems for Advanced Applications. DASFAA 2021. Lecture Notes in Computer Science(), vol 12682. Springer, Cham. https://doi.org/10.1007/978-3-030-73197-7_45

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-73197-7_45

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-73196-0

  • Online ISBN: 978-3-030-73197-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation