Skip to main content
SpringerLink
Log in

MQSS: multimodal query suggestion and searching for video search

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we propose a multimodal query suggestion method for video search which can leverage multimodal processing to improve the quality of search results. When users type general or ambiguous textual queries, our system MQSS provides keyword suggestions and representative image examples in an easy-to-use dropdown manner which can help users specify their search intent more precisely and effortlessly. It is a powerful complement to initial queries. After the queries are formulated as multimodal query (i.e., text, image), the new queries are input to individual search models, such as text-based, concept-based and visual example-based search model. Then we apply multimodal fusion method to aggregate the above-mentioned several search results. The effectiveness of MQSS is demonstrated by evaluations over a web video data set.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Amir A, Argillandery J, Campbell M et al (2005) IBM research TRECVID-2005 video retrieval system. In: TRECVID workshop, Washington DC

  2. Baeza-Yates R, Ribeiro-Neto B (1999) Modern information retrieval. Addison Wesley, Boston

    Google Scholar 

  3. Baeza-Yates R, Hurtado C, Mendoza M (2004) Query recommendation using query logs in search engines. In: EDBT, 2004

  4. Beeferman D, Berger A (2000) Agglomerative clustering of a search engine query log. In: Proceedings of ACM SIGKDD ’00, pp 407–416

  5. Campbell M, Haubold A et al (2006) IBM research TRECVID-2006 video retrieval system. In: TRECVID 2006

  6. Carpineto C, de Mori R, Romano G, Bigi B (2001) An information-theoretic approach to automatic query expansion. ACM TOIS 19(1):1–27

    Article  Google Scholar 

  7. Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas influence, and trends of the new age. In: ACM computing surveys

  8. Devijver PA, Kittler J (1982) Pattern recognition: a statistical approach. Prentice-Hall, London

    MATH  Google Scholar 

  9. Donald KM, Smeaton AF (2005) A comparison of score, rank and probability-based fusion methods for video shot retrieval. In: Proceedings of CIVR ’05

  10. Flickr APIs. http://www.flickr.com/services/api/

  11. Frey B, Dueck D (2007) Clustering by passing messages between data points. Science 315 (5814):972

    Article  MathSciNet  Google Scholar 

  12. Gao W, Niu C, Nie J-Y, Zhou M, Hu J, Wong K-F, Hon H-W (2007) Cross-lingual query suggestion using query logs of different languages. In: Proceedings of ACM SIGIR’07, pp 463–470

  13. Google Search. http://www.google.com/

  14. Hauptmann AG, Chen M-Y, Christel M, Das D, Lin W-H, Yan R, Yang J, Backfried G, Wu X (2006) Multi-lingual broadcast news retrieval. In: TRECVID 2006

  15. Hauptmann AG, Lin W-H, Yan R et al (2006) Extreme video retrieval: joint maximization of human and computer performance. In: Proceedings of ACM multimedia ’06, Santa Barbara, pp 385–394

  16. Hsu WH, Kennedy LS, Chang SF (2006) Video search reranking via information bottleneck principle. In: Proceedings of ACM multimedia ’06, Santa Barbara, pp 35–44

  17. Hua X-S, Mei T, Lai W, Wang M, Tang J, Qi G-J, Li L, Gu Z (2006) Microsoft research Asia TRECVID 2006: high-level feature extraction and rushes exploitation. In: TREC video retrieval evaluation online proceedings (TRECVID)

  18. Iyengar G, Duygulu P, Feng S et al (2005) Joint visual-text modeling for automatic retrieval of multimedia documents. In: Proceedings of ACM multimedia ’05, Singapore

  19. Kennedy L, Natsev A, Chang S-F (2005) Automatic discovery of query class dependent models for multimodal search. In: Proceedings of ACM Multimedia ’05, Singapore, pp 882–891

  20. Kennedy L, Chang S-F, Natsev A (2008) Query-adaptive fusion for multimodal search. In: Proceedings of the IEEE, vol 96, no 4, pp 567–588

  21. Lam-Adesina AM, Jones GJF (2001) Applying summarization techniques for term selection in relevance feedback. In: Proceedings of ACM SIGIR’01

  22. Liu J, Lai W, Hua X-S, Huang Y, Li S (2007) Video search re-ranking via multi-graph propagation. In: Proceedings of ACM multimedia ’07, Augsburg, pp 208–217

  23. Ribeiro-Neto B, Cristo M, Golgher P, de Moura E (2005) Impedance coupling in content-targeted advertising, In: Proceedings of ACM SIGIR ’05, Salvador, Brazil, pp 496–503

    Chapter  Google Scholar 

  24. Tian X, Yang L, Wang J et al (2008) Bayesian video search reranking. In: Proceedings of ACM multimedia ’08, Vancouver

  25. TRECVID. http://www-nlpir.nist.gov/projects/trecvid/

  26. Vapnik VN (1995) Statistical learning theory. Wiley, New York

    MATH  Google Scholar 

  27. Wang M, Hua X-S, Song Y et al (2006) Automatic video annotation by semi-supervised learning with Kernel density estimation. In: Proceedings of ACM multimedia ’06, Santa Barbara, pp 967–976

  28. Wen J-R, Nie J-Y, Zhang H-J (2003) Clustering user queries of a search engine. In: Proceedings of WWW ’03, pp 162–168

  29. Wu L, Hua X-S, Yu N, Ma W-Y, Li S (2007) Flickr distance, In: Proceedings of ACM multimedia ’07, Augsburg, pp 31–40

  30. Xu J, Croft WB (1996) Query expansion using local and global document analysis. In: Proceedings of ACM SIGIR’96

  31. Yahoo! Search. http://search.yahoo.com/

  32. Yu S, Cai D, Wen J-R, Ma W-Y (2003) Improving pseudo-relevance feedback in web information retrieval using web page segmentation. In: Proceedings of WWW ’03, pp 11–18

Download references

Acknowledgements

The authors would like to gratefully acknowledge Dr. Xian-Sheng Hua, Dr. Tao Mei, Dr. Zheng-jun Zha, Dr. Wei Lai, Dr. Meng Wang and Linjun Yang for their thoughtful brainstorming and constructive suggestions on this work. The research was supported by National 973 Program of China (Grant No.2005CB321901).

Author information

Authors and Affiliations

  1. State Key Laboratory of Software Development Environment, School of Computer Science and Engineering, Beihang University, Beijing, People’s Republic of China

    Lusong Li

  2. Institute of Advanced Computing Technology, School of Computer Science and Engineering, Beihang University, Beijing, People’s Republic of China

    Jing Li

Authors
  1. Lusong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lusong Li.

Additional information

Jing Li and Lusong Li contributed equally to this work.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Li, L., Li, J. MQSS: multimodal query suggestion and searching for video search. Multimed Tools Appl 54, 55–68 (2011). https://doi.org/10.1007/s11042-010-0540-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-010-0540-0

Keywords

Search

Navigation