Abstract
The effectiveness of image indexing and retrieval systems has been negatively impacted by both the semantic and user intention gaps. The first is related to the challenge of characterizing the visual semantic information through low-level extracted features (color, texture...) while the second highlights the difficulty for human users to convey their search intents using the traditional relevance feedback or query-by-example visual query mechanisms. We address both issues by highlighting vocabularies of visual concepts that are mapped to extracted low-level features through an automated learning paradigm. These are then instantiated within a semantic indexing and retrieval framework based on a Bayesian model considering the joint distribution of visual and semantic concepts. To address the user intention gap and enrich the expressiveness of the retrieval module, visual and semantic concepts can be coupled within text-based queries. We are therefore able to process both single-concept queries such as the state-of-the-art solutions but also topic-based queries, i.e. non-trivial queries involving multiple characterizations of the visual content. We evaluate our proposal in a precision/recall based evaluation framework on the IAPR-TC 12 benchmark dataset.




Similar content being viewed by others
References
Berlin B, Kay P (1991) Basic color terms: Their universality and evolution. UC Press
Belkhatir M, Thiem HC (2010) A multimedia retrieval framework highlighting agents and coordinating their interactions to address the semantic gap. Expert Syst Appl 37(12):8903–8909
Belkhatir M (2011) A three level framework for bridging the image semantic gap. Multimedia Systems, pp 135–148
Blei D, Jordan M (2003) Modeling annotated data. ACM SIGIR, pp 127–134
Chatfield K (2015) On-the-fly learning for visual search of large-scale image and video datasets. Int J Multimed Info Retr 2(2):75–93
Cortese JM, Dyre BP (1996) Perceptual similarity of shapes generated from Fourier descriptors. J Exp Psychol Hum Percept Perform 22(1):133–143
Escalantea H et al (2009) The segmented and annotated IAPR TC-12 benchmark. Comput Vision Image Underst 114(4):419–428
Fayyad UM, Irani KB (1993) Multi-interval discretization of continuous-valued attributes for classification learning. IJCAI, pp 1022–1029
Hoo WL, Chan CS (2015) Zero-shot object recognition system based on topic model. IEEE Trans Hum Mach Syst 45(4):518–525
Li J, Wang JZ (2008) Real-time computerized annotation of pictures. IEEE PAMI 30(6), pp 985–1002
Liu J et al (2007) Dual cross-media relevance model for image annotation. ACM MM, pp 605–614, 2007
Sarafis C, Diou C, Delopoulos A (2015) Building effective SVM concept detectors from clickthrough data for large-scale image retrieval. Int J Multimed Inf Retr 4(2):129–142
Yeh J-B, Wu C-H, Mai S-X (2013) Multiple visual concept discovery using concept-based visual word clustering. Multimed Syst 19(4):381–393
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Jarrar, R., Belkhatir, M. On the coupled use of signal and semantic concepts to bridge the semantic and user intention gaps for visual content retrieval. Int J Multimed Info Retr 5, 165–172 (2016). https://doi.org/10.1007/s13735-016-0101-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13735-016-0101-z