Skip to main content

Advertisement

SpringerLink
Log in

Graph-based multi-space semantic correlation propagation for video retrieval

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

By introducing the concept detection results to the retrieval process, concept-based video retrieval (CBVR) has been successfully used for semantic content-based video retrieval application. However, how to select and fuse the appropriate concepts for a specific query is still an important but difficult issue. In this paper, we propose a novel and effective concept selection method, named graph-based multi-space semantic correlation propagation (GMSSCP), to explore the relationship between the user query and concepts for video retrieval application. Compared with traditional methods, GMSSCP makes use of a manifold-ranking algorithm to collectively explore the multi-layered relationships between the query and concepts, and the expansion result is more robust to noises. Parallel to this, GMSSCP has a query-adapting property, which can enhance the process of concept correlation propagation and selection with strong pertinence of query cues. Furthermore, it can dynamically update the unified propagation graph by flexibly introducing the multi-modal query cues as additional nodes, and is not only effective for automatic retrieval but also appropriate for the interactive case. Encouraging experimental results on TRECVID datasets demonstrate the effectiveness of GMSSCP over the state-of-the-art concept selection methods. Moreover, we also apply it to the interactive retrieval system—VideoMap and gain an excellent performance and user experience.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Astam, J.A., Yilmaz, E.: Inferring document relevance via average precision. In: Proc. ACM Int. Conf. on Research and Development in Information Retrieval, pp. 601–602, New York, USA (2006)

  2. Cao, J., Zhang, Y.D., Feng, B.L., Hua, X.F., Bao, L., Zhang, X.: MCG-ICT-CAS TRECVID2008 search task report. In: Proc. TRECVID Workshop, Gaithersburg, MD (2008)

  3. Cao, J., Jing, H.F., Ngo, C.W., Zhang, Y.D.: Distribution-based concept selection for concept-based video retrieval. In: Proc. ACM Int. Conf. on Multimedia, pp. 19–24, Beijing, China (2009)

  4. Cao, J., Zhang, Y.D., Guo, J.B., Bao, L., Li, J.T.: VideoMap: An interactive video retrieval system of MCG-ICT-CAS. In: Proc. ACM Int. Conf. on Image and Video Retrieval, Santorini, Greece (2009)

  5. Fellbaum, C., et al.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998)

    MATH  Google Scholar 

  6. Haubold, A., Natsev, A., Naphade, M.: Semantic multimedia retrieval using lexical query expansion and model-based reranking. In: Proc. IEEE Int. Conf. on Multimedia and Expo, pp. 1761–1764, Toronto, Canada (2006)

  7. Hauptmann, A., Rong, Y., Lin, W.H., Christel, M., Wactlar, H.: Can high-level concepts fill the semantic gap in video retrieval? A case study with broadcast news. IEEE Trans. Multimedia 9(5), 958–966 (2007)

    Article  Google Scholar 

  8. He, J.R., Li, M.J., Zhang, H.J., Tong, H.H., Zhang, C.S.: Manifold-ranking based image retrieval. In: Proc. ACM Int. Conf. on Multimedia, pp. 9–16, New York, USA (2004)

  9. Jiang, Y.G., Ngo, C.W., Yang, J.: Towards optimal bag-of-features for object categorization and semantic video retrieval. In: Proc. ACM Int. Conf. on Image and Video Retrieval, pp. 494–501, Amsterdam, the Netherlands (2007)

  10. Li, X., Wang, D., Li, J., Zhang, B.: Video search in concept subspace: A text-like paradigm. In: Proc. ACM Int. Conf. on Image and Video Retrieval, pp. 603–610, Amsterdam, the Netherlands (2007)

  11. Naphade, M.R., Kennedy, L., Kender, J.R., Chang, S.F., Smith, J.R., Over, P., Hauptmann, A.: A light scale concept ontology for multimedia understanding for TRECVID2005. IBM Research Technical Report (2005)

  12. Naphade, M.R., Smith, J.R., Tesic, J., Chang, S.F., Hsu, W., Kennedy, L., Hauptmann, A., Curtis, J.: Large-scale concept ontology for multimedia. In: Proc. ACM Int. Conf. on Multimedia, pp. 86–91, Santa Barbara, CA (2006)

  13. Natsev, A.P., Haubold, A., Tesic, J., Xie, L., Yan, R.: Semantic concept based query expansion and re-ranking for multimedia retrieval. In: Proc. ACM Int. Conf. on Multimedia, pp. 991–1000, Augsburg, Germany (2007)

  14. Neo, S.Y., Zhao, J., Kan, M.Y., Chua, T.S.: Video retrieval using high level features: Exploiting query matching and confidence-based weighting. In: Proc. ACM Int. Conf. on Image and Video Retrieval, pp. 143–152, Heidelberg, Germany (2006)

  15. Ngo, C.W., Zhang, H.J., Pone, T.C.: Recent advances in content based video analysis. J. Image Graph. 1(3), 445–468 (2001)

    Article  Google Scholar 

  16. Ngo, C.W., Jiang, Y.G., Wei, X.Y., Zhao, W.L., Wang, F., Wu, X., Tan, H.K.: Beyond semantic search: What you observe may not be what you think. In: Proc. TRECVID Workshop, Gaithersburg, MD (2008)

  17. Over, P., Awad, G., Rose, T., Fiscus, J., Kraaij, W., Semeaton, A.F.: TRECVID 2008-goals, tasks, data, evaluation mechanisms and metrics. In: Proc. TRECVID Workshop, Gaithersburg, MD (2008)

  18. Qi, G.J., Hua, X.S., Rui, Y., Tang, J.H., Mei, T., Zhang, H.J.: Correlative multi-label video annotation. In: Proc. ACM Int. Conf. on Multimedia, pp. 17–26, Augsburg, Germany (2007)

  19. Rasiwasia, N., Moreno, P.J., Vasconcelos, N.: Bridging the gap: Query by semantic example. IEEE Trans. Multimedia 9(5), 923–938 (2007)

    Article  Google Scholar 

  20. Smeaton, A.F., Over, P., Kraaij, W.: Evaluation campaigns and trecvid. In: Proc. ACM Int. Workshop on Multimedia Information Retrieval, pp. 321–330, New York, USA (2006)

  21. Snoek, C.G.M., Worring, M.: Concept-based video retrieval. Found. Trends Inf. Retr. 4(2), 215–322 (2009)

    Google Scholar 

  22. Snoek, C.G.M., Worring, M., Gemert, J.C.V., Geusebroek, J.M., Smeulders, A.W.M.: The challenge problem for automated detection of 101 semantic concepts in multimedia. In: Proc. ACM Int. Conf. on Multimedia, pp. 421–430, Santa Barbara, CA (2006)

  23. Snoek, C.G.M., Huurnink, B., Hollink, L., Rijke, M.D., Schreiber, G., Worring, M.: Adding semantics to detectors for video retrieval. IEEE Trans. Multimedia 9(5), 975–986 (2007)

    Article  Google Scholar 

  24. Snoek, C.G.M., Worring, M., Rooij, O.D., Sande, K.E.A.V., Yan, R., Hauptmann, A.: VideOlympics: Real-time evaluation of multimedia retrieval systems. IEEE Multimedia 15(1), 86–91 (2008)

    Article  Google Scholar 

  25. Tang, J.H., Hua, X.S., Qi, G.J., Wang, M., Mei, T., Wu, X.Q.: Structure-sensitive manifold ranking for video concept detection. In: Proc. ACM Int. Conf. on Multimedia, pp. 852–861, Augsburg, Germany (2007)

  26. Tang, J., Hua, X.S., Wang, M., Gu, Z., Qi, G.J., Wu, X.: Correlative linear neighborhood propagation for video annotation. IEEE Trans. Syst. Man Cybern., B Cybern. 39(2), 409–416 (2009)

    Article  Google Scholar 

  27. Tang, J.H., Yan, S.C., Hong, R.C., Qi, G.J., Chua, T.S.: Inferring semantic concepts from community-contributed images and noisy tags. In: Proc. ACM Int. Conf. on Multimedia, pp. 223–232, Beijing, China (2009)

  28. Tesic, J., Natsev, A., Smith, J.R.: Cluster-based data modeling for semantic video search. In: Proc. ACM Int. Conf. on Image and Video Retrieval, pp. 595–602, Amsterdam, the Netherlands (2007)

  29. Wan, H.L., Morshed, U., Hu, H., et al.: Texture feature and its application in CBIR. J. Comput.-Aided Des. Comput. Graph. 15(2), 195–199 (2003)

    Google Scholar 

  30. Wang, D., Li, X., Li, J., Zhang, B.: The importance of query-concept-mapping for automatic video retrieval. In: Proc. ACM Int. Conf. on Multimedia, pp. 285–288, Augsburg, Germany (2007)

  31. Wei, X.Y., Ngo, C.W.: Ontology-enriched semantic space for video search. In: Proc. ACM Int. Conf. on Multimedia, pp. 981–990, Augsburg, Germany (2007)

  32. Wei, X.Y., Ngo, C.W.: Fusing semantics, observability, reliability and diversity of concept detectors for video search. In: Proc. ACM Int. Conf. on Multimedia, pp. 26–31, Vancouver, Canada (2008)

  33. Wei, X.Y., Ngo, C.W., Jiang, Y.G.: Selection of concept detectors for video search by ontology-enriched semantic spaces. IEEE Trans. Multimedia 10(6), 1085–1096 (2008)

    Article  Google Scholar 

  34. Yuan, X., Hua, X.S., Wang, M., Wu, X.: Manifold-ranking based video concept detection on large database and feature pool. In: Proc. ACM Int. Conf. on Multimedia, pp. 623–626, Santa Barbara, CA (2006)

  35. Yanagawa, A., et al.: Columbia University’s baseline detectors for 374 LSCOM semantic visual concepts. Technical report, Columbia University (2007)

  36. Zhou, D., Weston, J., Gretton, A., Bousquet, O., Scho, B.: Ranking on data manifolds. In: Proc. Annual Conf. on Neural Information Processing Systems, pp. 169–176, Cambridge, MA (2003)

  37. Zhou, D., Bousquet, O., Lal, T.N., Weston, J., Scholkopf, B.: Learning with local and global consistency. In: Proc. Annual Conf. on Neural Information Processing Systems, pp. 321–328, Cambridge, MA (2003)

  38. Zhou, X.S., Rui, Y., Huang, T.: Water-Filling: A novel way for image structural feature extraction. In: Proc. IEEE Int. Conf. on Image Processing, pp. 570–574, Kobe, Japan (1999)

Download references

Author information

Authors and Affiliations

  1. Laboratory of Advanced Computing Research, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China

    Bailan Feng, Juan Cao, Lei Bao, Yongdong Zhang & Shouxun Lin

  2. Graduate University of the Chinese Academy of Sciences, Beijing, 100039, China

    Bailan Feng & Lei Bao

  3. National Computer Network Emergency Response Technical/Coordination Center of China, Beijing, 100029, China

    Xiuguo Bao & Xiaochun Yun

Authors
  1. Bailan Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Juan Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiuguo Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lei Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yongdong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shouxun Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Xiaochun Yun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bailan Feng.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Feng, B., Cao, J., Bao, X. et al. Graph-based multi-space semantic correlation propagation for video retrieval. Vis Comput 27, 21–34 (2011). https://doi.org/10.1007/s00371-010-0510-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-010-0510-6

Keywords

Search

Navigation