Abstract
By introducing the concept detection results to the retrieval process, concept-based video retrieval (CBVR) has been successfully used for semantic content-based video retrieval application. However, how to select and fuse the appropriate concepts for a specific query is still an important but difficult issue. In this paper, we propose a novel and effective concept selection method, named graph-based multi-space semantic correlation propagation (GMSSCP), to explore the relationship between the user query and concepts for video retrieval application. Compared with traditional methods, GMSSCP makes use of a manifold-ranking algorithm to collectively explore the multi-layered relationships between the query and concepts, and the expansion result is more robust to noises. Parallel to this, GMSSCP has a query-adapting property, which can enhance the process of concept correlation propagation and selection with strong pertinence of query cues. Furthermore, it can dynamically update the unified propagation graph by flexibly introducing the multi-modal query cues as additional nodes, and is not only effective for automatic retrieval but also appropriate for the interactive case. Encouraging experimental results on TRECVID datasets demonstrate the effectiveness of GMSSCP over the state-of-the-art concept selection methods. Moreover, we also apply it to the interactive retrieval system—VideoMap and gain an excellent performance and user experience.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Astam, J.A., Yilmaz, E.: Inferring document relevance via average precision. In: Proc. ACM Int. Conf. on Research and Development in Information Retrieval, pp. 601–602, New York, USA (2006)
Cao, J., Zhang, Y.D., Feng, B.L., Hua, X.F., Bao, L., Zhang, X.: MCG-ICT-CAS TRECVID2008 search task report. In: Proc. TRECVID Workshop, Gaithersburg, MD (2008)
Cao, J., Jing, H.F., Ngo, C.W., Zhang, Y.D.: Distribution-based concept selection for concept-based video retrieval. In: Proc. ACM Int. Conf. on Multimedia, pp. 19–24, Beijing, China (2009)
Cao, J., Zhang, Y.D., Guo, J.B., Bao, L., Li, J.T.: VideoMap: An interactive video retrieval system of MCG-ICT-CAS. In: Proc. ACM Int. Conf. on Image and Video Retrieval, Santorini, Greece (2009)
Fellbaum, C., et al.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998)
Haubold, A., Natsev, A., Naphade, M.: Semantic multimedia retrieval using lexical query expansion and model-based reranking. In: Proc. IEEE Int. Conf. on Multimedia and Expo, pp. 1761–1764, Toronto, Canada (2006)
Hauptmann, A., Rong, Y., Lin, W.H., Christel, M., Wactlar, H.: Can high-level concepts fill the semantic gap in video retrieval? A case study with broadcast news. IEEE Trans. Multimedia 9(5), 958–966 (2007)
He, J.R., Li, M.J., Zhang, H.J., Tong, H.H., Zhang, C.S.: Manifold-ranking based image retrieval. In: Proc. ACM Int. Conf. on Multimedia, pp. 9–16, New York, USA (2004)
Jiang, Y.G., Ngo, C.W., Yang, J.: Towards optimal bag-of-features for object categorization and semantic video retrieval. In: Proc. ACM Int. Conf. on Image and Video Retrieval, pp. 494–501, Amsterdam, the Netherlands (2007)
Li, X., Wang, D., Li, J., Zhang, B.: Video search in concept subspace: A text-like paradigm. In: Proc. ACM Int. Conf. on Image and Video Retrieval, pp. 603–610, Amsterdam, the Netherlands (2007)
Naphade, M.R., Kennedy, L., Kender, J.R., Chang, S.F., Smith, J.R., Over, P., Hauptmann, A.: A light scale concept ontology for multimedia understanding for TRECVID2005. IBM Research Technical Report (2005)
Naphade, M.R., Smith, J.R., Tesic, J., Chang, S.F., Hsu, W., Kennedy, L., Hauptmann, A., Curtis, J.: Large-scale concept ontology for multimedia. In: Proc. ACM Int. Conf. on Multimedia, pp. 86–91, Santa Barbara, CA (2006)
Natsev, A.P., Haubold, A., Tesic, J., Xie, L., Yan, R.: Semantic concept based query expansion and re-ranking for multimedia retrieval. In: Proc. ACM Int. Conf. on Multimedia, pp. 991–1000, Augsburg, Germany (2007)
Neo, S.Y., Zhao, J., Kan, M.Y., Chua, T.S.: Video retrieval using high level features: Exploiting query matching and confidence-based weighting. In: Proc. ACM Int. Conf. on Image and Video Retrieval, pp. 143–152, Heidelberg, Germany (2006)
Ngo, C.W., Zhang, H.J., Pone, T.C.: Recent advances in content based video analysis. J. Image Graph. 1(3), 445–468 (2001)
Ngo, C.W., Jiang, Y.G., Wei, X.Y., Zhao, W.L., Wang, F., Wu, X., Tan, H.K.: Beyond semantic search: What you observe may not be what you think. In: Proc. TRECVID Workshop, Gaithersburg, MD (2008)
Over, P., Awad, G., Rose, T., Fiscus, J., Kraaij, W., Semeaton, A.F.: TRECVID 2008-goals, tasks, data, evaluation mechanisms and metrics. In: Proc. TRECVID Workshop, Gaithersburg, MD (2008)
Qi, G.J., Hua, X.S., Rui, Y., Tang, J.H., Mei, T., Zhang, H.J.: Correlative multi-label video annotation. In: Proc. ACM Int. Conf. on Multimedia, pp. 17–26, Augsburg, Germany (2007)
Rasiwasia, N., Moreno, P.J., Vasconcelos, N.: Bridging the gap: Query by semantic example. IEEE Trans. Multimedia 9(5), 923–938 (2007)
Smeaton, A.F., Over, P., Kraaij, W.: Evaluation campaigns and trecvid. In: Proc. ACM Int. Workshop on Multimedia Information Retrieval, pp. 321–330, New York, USA (2006)
Snoek, C.G.M., Worring, M.: Concept-based video retrieval. Found. Trends Inf. Retr. 4(2), 215–322 (2009)
Snoek, C.G.M., Worring, M., Gemert, J.C.V., Geusebroek, J.M., Smeulders, A.W.M.: The challenge problem for automated detection of 101 semantic concepts in multimedia. In: Proc. ACM Int. Conf. on Multimedia, pp. 421–430, Santa Barbara, CA (2006)
Snoek, C.G.M., Huurnink, B., Hollink, L., Rijke, M.D., Schreiber, G., Worring, M.: Adding semantics to detectors for video retrieval. IEEE Trans. Multimedia 9(5), 975–986 (2007)
Snoek, C.G.M., Worring, M., Rooij, O.D., Sande, K.E.A.V., Yan, R., Hauptmann, A.: VideOlympics: Real-time evaluation of multimedia retrieval systems. IEEE Multimedia 15(1), 86–91 (2008)
Tang, J.H., Hua, X.S., Qi, G.J., Wang, M., Mei, T., Wu, X.Q.: Structure-sensitive manifold ranking for video concept detection. In: Proc. ACM Int. Conf. on Multimedia, pp. 852–861, Augsburg, Germany (2007)
Tang, J., Hua, X.S., Wang, M., Gu, Z., Qi, G.J., Wu, X.: Correlative linear neighborhood propagation for video annotation. IEEE Trans. Syst. Man Cybern., B Cybern. 39(2), 409–416 (2009)
Tang, J.H., Yan, S.C., Hong, R.C., Qi, G.J., Chua, T.S.: Inferring semantic concepts from community-contributed images and noisy tags. In: Proc. ACM Int. Conf. on Multimedia, pp. 223–232, Beijing, China (2009)
Tesic, J., Natsev, A., Smith, J.R.: Cluster-based data modeling for semantic video search. In: Proc. ACM Int. Conf. on Image and Video Retrieval, pp. 595–602, Amsterdam, the Netherlands (2007)
Wan, H.L., Morshed, U., Hu, H., et al.: Texture feature and its application in CBIR. J. Comput.-Aided Des. Comput. Graph. 15(2), 195–199 (2003)
Wang, D., Li, X., Li, J., Zhang, B.: The importance of query-concept-mapping for automatic video retrieval. In: Proc. ACM Int. Conf. on Multimedia, pp. 285–288, Augsburg, Germany (2007)
Wei, X.Y., Ngo, C.W.: Ontology-enriched semantic space for video search. In: Proc. ACM Int. Conf. on Multimedia, pp. 981–990, Augsburg, Germany (2007)
Wei, X.Y., Ngo, C.W.: Fusing semantics, observability, reliability and diversity of concept detectors for video search. In: Proc. ACM Int. Conf. on Multimedia, pp. 26–31, Vancouver, Canada (2008)
Wei, X.Y., Ngo, C.W., Jiang, Y.G.: Selection of concept detectors for video search by ontology-enriched semantic spaces. IEEE Trans. Multimedia 10(6), 1085–1096 (2008)
Yuan, X., Hua, X.S., Wang, M., Wu, X.: Manifold-ranking based video concept detection on large database and feature pool. In: Proc. ACM Int. Conf. on Multimedia, pp. 623–626, Santa Barbara, CA (2006)
Yanagawa, A., et al.: Columbia University’s baseline detectors for 374 LSCOM semantic visual concepts. Technical report, Columbia University (2007)
Zhou, D., Weston, J., Gretton, A., Bousquet, O., Scho, B.: Ranking on data manifolds. In: Proc. Annual Conf. on Neural Information Processing Systems, pp. 169–176, Cambridge, MA (2003)
Zhou, D., Bousquet, O., Lal, T.N., Weston, J., Scholkopf, B.: Learning with local and global consistency. In: Proc. Annual Conf. on Neural Information Processing Systems, pp. 321–328, Cambridge, MA (2003)
Zhou, X.S., Rui, Y., Huang, T.: Water-Filling: A novel way for image structural feature extraction. In: Proc. IEEE Int. Conf. on Image Processing, pp. 570–574, Kobe, Japan (1999)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Feng, B., Cao, J., Bao, X. et al. Graph-based multi-space semantic correlation propagation for video retrieval. Vis Comput 27, 21–34 (2011). https://doi.org/10.1007/s00371-010-0510-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-010-0510-6