Skip to main content
Log in

Event-based cross media question answering

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

User generated content, available in massive amounts on the Internet, is receiving increased attention due to its many potential applications. One of such applications is the representation of events using multimedia data. In this paper, an event-based cross media question answering system, which retrieves and summarizes events on a given topic is proposed. In other words, we present a framework for leveraging social media data to extract and illustrate social events automatically on any given query. The system is built in three steps. First, the input query is parsed semantically to identify the topic, location, and time information related to the News of interest. Then, we use the parsed information to mine the latest and hottest related News from social news web services. Third, to identify a unique event, we model the News content by latent Dirichlet Allocation and cluster the News using the DBSCAN algorithm. In the end, for each event, we retrieve both textual and visual content of News that refer the same event. The resulting documents are shown within a vivid interface featuring both event description, tag cloud and photo collage.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. http://www.eventburn.com/

  2. http://nltk.org/

  3. http://dbpedia.org

  4. http://news.google.com

  5. http://digg.com/

References

  1. Allan J, Carbonell J, Doddington G, Yamron J, Yang Y (1998) Topic detection and tracking pilot study: final report. In Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop, Lansdowne, VA, USA, Feb. 007, pp 194–218

  2. Arthur D, Vassilvitskii S (2007) k-means++: the advantages of careful seeding. In SODA’07: Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pages 1027–1035, Philadelphia, PA, USA. Society for Industrial and Applied Mathematics

  3. Bao B-K, Min W, Sang J, Xu C (2012) Multimedia news digger on emerging topics from social streams. In Proceedings of the 20th ACM international conference on Multimedia, MM’12, pp 1357–1358

  4. Becker H, Iter D, Naaman M, Gravano L (2012) Identifying content for planned events across social media sites. In ACM conference on WSDM

  5. Chen L, Roy A (2009) Event detection from flickr data through wavelet-based spatial analysis. In ACM conference on CIKM

  6. Chengjie S, Yi G (2004) A statistical approach for content extraction from web page. J Chin Inf Process 18(5):17–22

    Google Scholar 

  7. Delgado D, Magalhães JA, Correia N (2010) Assisted News Reading with Automated Illustrations. In ACM conference on Multimedia, pp 1647–1650

  8. Firan CS, Georgescu M, Nejdl W, Paiu R (2010) Bringing order to your photos: Event-Driven Classification of Flickr Images Based on Social Knowledge. In Proceedings of the 19th ACM international conference on Information and knowledge management, New York, USA, pp 189

  9. Gao Y, Wang M, Zha Z-J, Shen J, Li X, Wu X (2013) Visual-textual joint relevance learning for tag-based social image search. IEEE Trans Image Process 22(1):363–376

    Article  MathSciNet  Google Scholar 

  10. Hong R, Wang M, Li G, Nie L, Zha Z-J, Chua T-S (2012) Multimedia question answering. MultiMedia IEEE 19(4):72–78

    Article  Google Scholar 

  11. Joshi D, Wang JZ, Li J (2006) The story picturing engine—a system for automatic text illustration. ACM Trans Multimed Comput Commun Appl 2(1):68–89

    Article  Google Scholar 

  12. Li G, Ming Z, Li H, Chua T-S (2009) Video reference: question answering on youtube. In Proceedings of the 17th ACM international conference on Multimedia, pp 773–776

  13. Li H, Tang J, Wang Y, Liu B (2012) Looking into the world on google maps with view direction estimated photos. Neurocomput 95:72–77

    Article  Google Scholar 

  14. Liu X, Huet B, Troncy R (2011) Eurecom@ mediaeval 2011 social event detection task. In MediaEval

  15. Liu X, Huet B, Troncy R (2011) Eurecom@ MediaEval 2011 social event detection task. In Proceedings of the MediaEval 2011 Workshop

  16. Liu X, Troncy R, Huet B (2011) Finding Media Illustrating Events. In ACM International Conference on ICMR, Trento, Italy

  17. Manning CD, Raghavan P, Schütze H (2008) Introduction to Information Retrieval. 1 edition, July

  18. Mei T, Yang B, Yang S-Q, Hua X-S (2009) Video collage: presenting a video sequence using a single image. Visual Comput 39–51

    Article  Google Scholar 

  19. Nie L, Wang M, Gao Y, Zha Z-J, Chua T-S (2013) Beyond text qa: multimedia answer generation by harvesting web information. IEEE Trans Multimedia 15(2):426–441

    Article  Google Scholar 

  20. Pan C-C, Mitra P (2011) Event detection with spatial latent Dirichlet allocation. In Proceeding of the 11th annual international ACM/IEEE joint conference on Digital libraries, page 349, New York, USA, June.

  21. Quack T, Leibe B, Van Gool L (2008) World-scale mining of objects and events from community photo collections. In Proceedings of the 2008 international conference on Content-based image and video retrieval, page 47, New York, USA, July

  22. Sakaki T, Okazaki M, Matsuo Y (2010) Earthquake shakes twitter users: real-time event detection by social sensors. In International conference on World Wide Web, Raleigh, North Carolina, USA

  23. Shi J, Malik J (2000) Normalized cuts and image segmentation. Pattern Anal Mach Intell IEEE Trans 22(8):888–905

    Article  Google Scholar 

  24. Wang M, Hong R, Li G, Zha Z-J, Yan S, Chua T-S (2012) Event driven web video summarization by tag localization and key-shot identification. IEEE Trans Multimedia 14(4):975–985

    Article  Google Scholar 

  25. Wang M, Li G, Lu Z, Gao Y, Chua T-S (2013) When amazon meets google: product visualization by exploring multiple web sources. ACM Trans Internet Technol 12(4):12:1–12:17

    Article  Google Scholar 

  26. Weng J, Lee F (2011) Event detection in twitter. In AAAI conference on Weblogs and Social Media, Barcelona, Spain

  27. Yahiaoui I, Mérialdo B, Huet B (2003) Comparison of multi-episode video summarization algorithms. EURASIP Journal on applied signal processing special issue on multimedia signal processing - Volume 2003 N°1, January 2003, 01

  28. Zha Z-J, Yang L, Mei T, Wang M, Wang Z, Chua T-S, Hua X-S (2010) Visual query suggestion: towards capturing user intent in internet image search. ACM Trans Multimedia Comput Commun Appl 6(3):13:1–13:19

    Article  Google Scholar 

  29. Zha Z-J, Zhang H, Wang M, Luan H, Chua T-S (2013) Detecting group activities with multi-camera context. IEEE Trans Circ Syst Video Technol 23(5):856–869

    Article  Google Scholar 

  30. Zhu X, Goldberg AB, Eldawy M, Dyer CR, Strock B (2007) A text-to-picture synthesis system for augmenting communication. In Proceedings of the 22nd national conference on Artificial intelligence, number 2, p 1590–1595

Download references

Acknowledgments

This work was supported by the 973 Program of China (No. 2013CB329604), and the European Commission under contracts FP7-287911 LinkedTV and FP7-318101 MediaMixer, as well as.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xueliang Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, X., Huet, B. Event-based cross media question answering. Multimed Tools Appl 75, 1495–1508 (2016). https://doi.org/10.1007/s11042-014-2085-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-014-2085-0

Keywords

Navigation