skip to main content
10.1145/2072298.2072365acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Contextual image search

Published: 28 November 2011 Publication History

Abstract

In this paper, we propose a novel image search scheme, contextual image search. Different from conventional image search schemes that present a separate interface (e.g., text input box) to allow users to submit a query, the new search scheme enables users to search images by only masking a few words when they are reading through Web pages or other documents. Rather than merely making use of the explicit query input that is often not sufficient to express user's search intent, our approach explores the context information to better understand the search intent with two key steps: query augmenting and search results reranking using context, and expects to obtain better search results. Beyond contextual Web search, the context in our case is much richer and includes images besides texts. In addition to this type of search scheme, called contextual image search with text input, we also present another type of scheme, called contextual image search with image input, to allow users to select an image as the search query from Web pages or other documents they are reading. The key idea is to use the search-to-annotation technique and the contextual textual query mining scheme to determine the corresponding textual query, to finally get semantically similar search results. Experiments show that the proposed schemes make image search more convenient and the search results are more relevant to user intention.

References

[1]
D. Cai, S. Yu, J.-R. Wen, and W.-Y. Ma. Vips: a vision-based page segmentation algorithm. Technical Report MSR-TR-2003--79, Microsoft, 2003.
[2]
J. Cui, F. Wen, and X. Tang. Intentsearch: interactive on-line image search re-ranking. In ACM Multimedia, pages 997--998, 2008.
[3]
R. Datta, D. Joshi, J. Li, and J. Z. Wang. Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv., 40(2), 2008.
[4]
S. K. Divvala, D. Hoiem, J. Hays, A. A. Efros, and M. Hebert. An empirical study of context in object detection. In CVPR, pages 1271--1278, 2009.
[5]
L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, and E. Ruppin. Placing search in context: the concept revisited. ACM Trans. Inf. Syst., 20(1):116--131, 2002.
[6]
J. Fogarty, D. S. Tan, A. Kapoor, and S. A. J. Winder. Cueflik: interactive concept learning in image search. In CHI, pages 29--38, 2008.
[7]
R. Jain. Multimedia information retrieval: watershed events. In Multimedia Information Retrieval, pages 229--236, 2008.
[8]
R. Kraft, C.-C. Chang, F. Maghoul, and R. Kumar. Searching with context. In WWW, pages 477--486, 2006.
[9]
M. S. Lew, N. Sebe, C. Djeraba, and R. Jain. Content-based multimedia information retrieval: State of the art and challenges. TOMCCAP, 2(1):1--19, 2006.
[10]
D. G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91--110, 2004.
[11]
Y. Luo, W. Liu, J. Liu, and X. Tang. Mqsearch: image search by multi-class query. In CHI, pages 49--52, 2008.
[12]
C. D. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.
[13]
J. Matas, O. Chum, M. Urban, and T. Pajdla. Robust wide baseline stereo from maximally stable extremal regions. In BMVC, 2002.
[14]
T. Mei, X.-S. Hua, and S. Li. Contextual in-image advertising. In ACM Multimedia, pages 439--448, 2008.
[15]
Y. Rui and T. S. Huang. A novel relevance feedback technique in image retrieval. In ACM Multimedia (2), pages 67--70, 1999.
[16]
J. Sivic and A. Zisserman. Efficient visual search of videos cast as text retrieval. IEEE Trans. Pattern Anal. Mach. Intell., 31(4):591--606, 2009.
[17]
A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell., 22(12):1349--1380, 2000.
[18]
J. Wang and X.-S. Hua. Interactive image search by color map. ACM TIST, 3(1):26, 2012.
[19]
J. Wang, W. Lu, X.-S. Hua, S. Wang, and S. Li. Contextual image search. Technical report, MSR-TR-2010--84, 2010.
[20]
X.-J. Wang, L. Zhang, X. Li, and W.-Y. Ma. Annotating images by mining image search results. IEEE Trans. Pattern Anal. Mach. Intell., 30(11):1919--1932, 2008.
[21]
R. W. White, P. Bailey, and L. Chen. Predicting user interests from contextual information. In SIGIR, pages 363--370, 2009.
[22]
X. Xing, Y. Zhang, and B. Gong. Mixture model based contextual image retrieval. In CIVR, pages 251--258, 2010.
[23]
X. Xing, Y. Zhang, and M. Han. Query difficulty prediction for contextual image retrieval. In ECIR, pages 581--585, 2010.
[24]
H. Xu, J. Wang, X.-S. Hua, and S. Li. Image search by concept map. In SIGIR, pages 275--282, 2010.
[25]
H. Xu, J. Wang, X.-S. Hua, and S. Li. Interactive image search by 2d semantic map. In WWW, pages 1321--1324, 2010.
[26]
R. Yan, A. Natsev, and M. Campbell. Multi-query interactive image and video retrieval: theory and practice. In CIVR, pages 475--484, 2008.
[27]
E. Zavesky and S.-F. Chang. Cuzero: embracing the frontier of interactive visual search for informed users. In Multimedia Information Retrieval, pages 237--244, 2008.

Cited By

View all
  • (2020)Generating Images Instead of Retrieving ThemProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401129(1329-1338)Online publication date: 25-Jul-2020
  • (2017)Multi-index structure based on SIFT and color features for large scale image retrievalMultimedia Tools and Applications10.1007/s11042-016-3788-176:12(13929-13951)Online publication date: 1-Jun-2017
  • (2016)Context aware multimedia crawler for dynamic encyclopaedia construction2016 International Conference on Computer Communication and Informatics (ICCCI)10.1109/ICCCI.2016.7479957(1-12)Online publication date: Jan-2016
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MM '11: Proceedings of the 19th ACM international conference on Multimedia
November 2011
944 pages
ISBN:9781450306164
DOI:10.1145/2072298
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 November 2011

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. contextual query augmentation
  2. contextual reranking
  3. image search
  4. textual and visual context

Qualifiers

  • Research-article

Conference

MM '11
Sponsor:
MM '11: ACM Multimedia Conference
November 28 - December 1, 2011
Arizona, Scottsdale, USA

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)7
  • Downloads (Last 6 weeks)0
Reflects downloads up to 28 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2020)Generating Images Instead of Retrieving ThemProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401129(1329-1338)Online publication date: 25-Jul-2020
  • (2017)Multi-index structure based on SIFT and color features for large scale image retrievalMultimedia Tools and Applications10.1007/s11042-016-3788-176:12(13929-13951)Online publication date: 1-Jun-2017
  • (2016)Context aware multimedia crawler for dynamic encyclopaedia construction2016 International Conference on Computer Communication and Informatics (ICCCI)10.1109/ICCCI.2016.7479957(1-12)Online publication date: Jan-2016
  • (2015)Fine-Grained Image SearchIEEE Transactions on Multimedia10.1109/TMM.2015.240856617:5(636-647)Online publication date: May-2015
  • (2015)Exploratory Product Image Search With Circle-to-Search InteractionIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2014.237227225:7(1190-1202)Online publication date: Jul-2015
  • (2014)Browse-to-SearchACM Transactions on Information Systems10.1145/263042032:4(1-27)Online publication date: 28-Oct-2014
  • (2014)Coupled Binary Embedding for Large-Scale Image RetrievalIEEE Transactions on Image Processing10.1109/TIP.2014.233076323:8(3368-3380)Online publication date: Aug-2014
  • (2014) \(\mathcal {L}_p\) -Norm IDF for Scalable Image RetrievalIEEE Transactions on Image Processing10.1109/TIP.2014.232918223:8(3604-3617)Online publication date: Aug-2014
  • (2012)Interactive metric learning system for similar image search using Linear Discriminant AnalysisThe 1st IEEE Global Conference on Consumer Electronics 201210.1109/GCCE.2012.6379581(206-209)Online publication date: Oct-2012

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media