skip to main content
research-article

Browse-to-Search: Interactive Exploratory Search with Visual Entities

Published: 28 October 2014 Publication History

Abstract

With the development of image search technology, users are no longer satisfied with searching for images using just metadata and textual descriptions. Instead, more search demands are focused on retrieving images based on similarities in their contents (textures, colors, shapes etc.). Nevertheless, one image may deliver rich or complex content and multiple interests. Sometimes users do not sufficiently define or describe their seeking demands for images even when general search interests appear, owing to a lack of specific knowledge to express their intents. A new form of information seeking activity, referred to as exploratory search, is emerging in the research community, which generally combines browsing and searching content together to help users gain additional knowledge and form accurate queries, thereby assisting the users with their seeking and investigation activities. However, there have been few attempts at addressing integrated exploratory search solutions when image browsing is incorporated into the exploring loop. In this work, we investigate the challenges of understanding users' search interests from the images being browsed and infer their actual search intentions. We develop a novel system to explore an effective and efficient way for allowing users to seamlessly switch between browse and search processes, and naturally complete visual-based exploratory search tasks. The system, called Browse-to-Search enables users to specify their visual search interests by circling any visual objects in the webpages being browsed, and then the system automatically forms the visual entities to represent users' underlying intent. One visual entity is not limited by the original image content, but also encapsulated by the textual-based browsing context and the associated heterogeneous attributes. We use large-scale image search technology to find the associated textual attributes from the repository. Users can then utilize the encapsulated visual entities to complete search tasks. The Browse-to-Search system is one of the first attempts to integrate browse and search activities for a visual-based exploratory search, which is characterized by four unique properties: (1) in session—searching is performed during browsing session and search results naturally accompany with browsing content; (2) in context—the pages being browsed provide text-based contextual cues for searching; (3) in focus—users can focus on the visual content of interest without worrying about the difficulties of query formulation, and visual entities will be automatically formed; and (4) intuitiveness—a touch and visual search-based user interface provides a natural user experience. We deploy the Browse-to-Search system on tablet devices and evaluate the system performance using millions of images. We demonstrate that it is effective and efficient in facilitating the user's exploratory search compared to the conventional image search methods and, more importantly, provides users with more robust results to satisfy their exploring experience.

References

[1]
A. Agarawala and R. Balakrishnan. 2006. Keepin' it real: Pushing the desktop metaphor with physics, piles and the pen. In Proceedings of the SIGCHI Conference on Human Factors in Computing System (CHI'06). ACM, New York, NY, 1283--1292.
[2]
D. Bainbridge, M. B. Twidale, and D. M. Nichols. 2011. A User-driven context-aware approach to erroneous metadata in digital libraries. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries. 39--48.
[3]
D. Bainbridge, M. B. Twidale, and D. M. Nichols. 2012. Interactive context-aware user-driven metadata correction in digital libraries. Int. J. Digital Lib. 13, 1, 17--32.
[4]
H. Cao, D. H. Hu, D. Shen, D. Jiang, J.-T. Sun, E. Chen, and Q. Yang. 2009. Context-aware query classification. In Proceedings of the 32nd International Conference on Research and Development in Information Retrieval (SIGIR). 3--10.
[5]
Y. Cao, H. Wang, C. Wang, Z. Li, L. Zhang, and L. Zhang. 2010. MindFinder: Interactive sketch-based image search on millions of images. In Proceedings of the International Conference on ACM Multimedia. 1605--1608.
[6]
V. Chandrasekhar, G. Takacs, D. M. Chen, S. S. Tsai, R. Grzeszczuk, and B. Girod. 2009. CHoG: Compressed histogram of gradients A low bit-rate feature descriptor. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recoginition (CVPR). 2504--2511.
[7]
E. Cheng, F. Jing, and L. Zhang. 2009. A unified relevance feedback framework for web image retrieval. IEEE Trans. Image Process. 18, 6, 1350--1357.
[8]
M. Dörk, C. Williamson, and S. Carpendale. 2012. Navigating tomorrow's Web: From searching and browsing to visual exploration. ACM Trans. Web 6, 3, 13:1--13:28.
[9]
W.-T. Fu, T. G. Kannampallil, and R. Kang. 2010. Facilitating exploratory search by model-based navigational cues. In Proceedings of the 14th International Conference on Intelligent User Interfaces (IUI'10). ACM, New York, NY, 199--208.
[10]
G. Golovchinsky, A. Dunnigan, and A. Diriye. 2012. Designing a tool for exploratory information seeking. In Proceedings of the Extended Abstracts on Human Factors in Computing Systems (CHI). 1799--1804.
[11]
Google Related. 2012. http://www.google.com/related.
[12]
R. Ji, L.-Y. Duan, J. Chen, H. Yao, Y. Rui, S.-F. Chang, and W. Gao. 2011. Towards low bit rate mobile visual search with multiple-channel coding. In Proceedings of the International Conference on Multimedia. 573--582.
[13]
A. Kerne, E. Koh, S. M. Smith, A. Webb, and B. Dworaczyk. 2008. combinFormation: Mixed-initiative composition of image and text surrogates promotes information discovery. ACM Trans. Inf. Syst. 27, 1, 5:1--5:45
[14]
B. Kules and B. Shneiderman. 2008. Users can change their web search tactics: Design guidelines for categorized overviews. Int. J. Inf. Process. Manag. 44, 2, 463--484.
[15]
S. Kullback and R. A. Leibler. 1951. On information and sufficiency. Ann. Math. Statist. 22, 1, 79--86.
[16]
X. Li. 2010. Understanding the semantic structure of noun phrase queries. In Proceedings of the 48th Annual Meeting of the ACL (ACL). 1337--1345.
[17]
Y. Liu, T. Mei, and X.-S. Hua. 2009. CrowdReranking: Exploring multiple search engines for visual search reranking. In Proceedings of SIGIR. 500--507.
[18]
F. Loumakis, S. Stumpf, and D. Grayson. 2011. This image smells good: Effects of image information scent in search engine results pages. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM). 475--484.
[19]
D. G. Lowe. 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60, 2, 91--110.
[20]
S. Lu, T. Mei, J. Wang, J. Zhang, Z. Wang, D. D. Feng, J.-T. Sun, and S. Li. 2012. Browse-to-Search. (Video demo). In Proceedings of the International Conference on Multimedia.
[21]
W. Lu, J. Wang, X.-S. Hua, S. Wang, and S. Li. 2011. Contextual image search. In Proceedings of the International Conference on Multimedia. 513--522.
[22]
G. Marchionini. 2006. Exploratory search: From finding to understanding. Commun. ACM 49, 4, 41--46.
[23]
G. Marchionini and G. Geisler. 2002. The open video digital library. D-Lib Mag. 8, 12.
[24]
T. Mei, Y. Rui, S. Li, and Q. Tian. 2014. Multimedia search reranking: A literature survey. ACM Comput. Surv. 46, 3, 38:1--38:38.
[25]
T. Mei, B. Yang, X.-S. Hua, and S. Li. 2011. Contextual video recommendation by multimodal relevance and user feedback. ACM Trans. Inf. Syst. 29, 2, 10:1--10:24.
[26]
D. Milne, D. M. Nichols, and I. H. Witten. 2008. A competitive environment for exploratory query expansion. In Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL). 197--200.
[27]
P. Pirolli. 2009. An elementary social information foraging model. In Proceedings of the SIGCHI Conference on Human Factors on Computing Systems. 605--614.
[28]
P. Pirolli, S. K. Card, and M. M. Van Der Wege. 2003. The effects of information scent on visual search in the hyperbolic tree browser. ACM Trans. Comput. Human Interact. 10, 1, 20--53.
[29]
D. Qin, S. Gammeter, L. Bossard, T. Quack, and L. J. Van Gool. 2011. Hello neighbor: Accurate object retrieval with k-reciprocal nearest neighbors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 777--784.
[30]
J Sang, T. Mei, Y.-Q. Xu, C. Zhao, C. Xu, and S. Li. 2013. Interaction design for mobile visual search. IEEE Trans. Multiamedia 15, 7, 1665--1676.
[31]
G. Schindler, M. Brown, and R. Szeliski. 2007. City-scale location recognition. In Proceedings of (CVPR).
[32]
D. Shen, J.-T. Sun, Q. Yang, and Z. Chen. 2006. Building bridges for web query classification. In Proceedings of SIGIR. 131--138.
[33]
D. A. Smith, A. Owens, M. C. Schraefel, P. Sinclair, P. André, M. Wilson, A. Russell, K. Martinez, and P. Lewis. 2007. Challenges in supporting faceted semantic browsing of multimedia collections. In Proceedings of the 2nd International Conference on Semantic and Digital Media Technologies (SAMT). Lecture Notes in Computer Science, Vol. 4816. Springer, Berlin, 280--283.
[34]
J. Wang and S. Li. 2012. Query-driven iterated neighborhood graph search for large scale indexing. In Proceedings of the International Conference on Multimedia.
[35]
R. W. White, B. Kules, S. M. Drucker, and M. M. C. Schraefel. 2006. Supporting exploratory search. Commun. ACM 49, 4.
[36]
R. W. White and R. A. Roth. 2009. Exploratory search: Beyond the query-response paradigm. Morgan & Claypool Publishers.
[37]
M. L. Wilson, P. André, and M. C. Schraefel. 2008. Backward highlighting: Enhancing faceted search. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST). 235--238.
[38]
M. L. Wilson and M. C. Schraefel. 2010. Evaluating collaborative information-seeking interfaces with a search-oriented inspection method and re-framed information seeking theory. Int. J. Inf. Process. Manag. 46, 6, 718--732.
[39]
H. Xu, J. Wang, X.-S. Hua, and S. Li. 2010. Image search by concept map. In Proceedings of the SIGIR. 275--282.
[40]
H. Xu, J. Wang, Z. Li, G. Zeng, S. Li, and N. Yu. 2011. Complementary hashing for approximate nearest neighbor search. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 1631--1638.
[41]
J. Xu and W. B. Croft. 1996. Query expansion using local and global document analysis. In Proceedings of the SIGIR. 4--11.
[42]
F. X. Yu, R. Ji, and S.-F. Chang. 2011. Active query sensing for mobile location search. In Proceedings of the International Conference on Multimedia. 3--12.
[43]
Z.-J. Zha, L. Yang, T. Mei, M. Wang, and Z. Wang. 2009. Visual query suggestion. In Proceedings of the International Conference on Multimedia. 15--24.
[44]
W. Zhou, H. Li, Y. Lu, and Q. Tian. 2013. SIFT match verification by geometric coding for large-scale partial-duplicate web image search. ACM Trans. Multimedia Comput. Commun. Appl. 9, 1, 4:1--4:18.

Cited By

View all
  • (2025)UFTDRDHJournal of Information Science10.1177/0165551522113352951:1(111-134)Online publication date: 1-Feb-2025
  • (2024)Sample, Nudge and Rank: Exploiting Interpretable GAN Controls for Exploratory SearchProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645156(582-596)Online publication date: 18-Mar-2024
  • (2023)Enhancing Recommendation with Search Data in a Causal Learning MannerACM Transactions on Information Systems10.1145/358242541:4(1-31)Online publication date: 1-Feb-2023
  • Show More Cited By

Index Terms

  1. Browse-to-Search: Interactive Exploratory Search with Visual Entities

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Information Systems
      ACM Transactions on Information Systems  Volume 32, Issue 4
      October 2014
      198 pages
      ISSN:1046-8188
      EISSN:1558-2868
      DOI:10.1145/2684820
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 28 October 2014
      Accepted: 01 May 2014
      Revised: 01 January 2014
      Received: 01 February 2013
      Published in TOIS Volume 32, Issue 4

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Multimedia information systems
      2. exploratory search
      3. gesture
      4. interactive visual search
      5. multimedia browsing
      6. user interaction

      Qualifiers

      • Research-article
      • Research
      • Refereed

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)25
      • Downloads (Last 6 weeks)2
      Reflects downloads up to 28 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)UFTDRDHJournal of Information Science10.1177/0165551522113352951:1(111-134)Online publication date: 1-Feb-2025
      • (2024)Sample, Nudge and Rank: Exploiting Interpretable GAN Controls for Exploratory SearchProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645156(582-596)Online publication date: 18-Mar-2024
      • (2023)Enhancing Recommendation with Search Data in a Causal Learning MannerACM Transactions on Information Systems10.1145/358242541:4(1-31)Online publication date: 1-Feb-2023
      • (2023)“Webcomics Archive? Now I'm Interested”: Comics Readers Seeking Information in Web ArchivesProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578325(412-416)Online publication date: 19-Mar-2023
      • (2022)ISRE-Framework: nonlinear and multimodal exploration of image search result spacesMultimedia Tools and Applications10.1007/s11042-022-12561-481:19(27275-27308)Online publication date: 1-Aug-2022
      • (2021)A Comparative Studies of Automatic Query Formulation in Full-Text Database Search of Chinese Digital HumanitiesDiversity, Divergence, Dialogue10.1007/978-3-030-71292-1_35(457-468)Online publication date: 17-Mar-2021
      • (2019)Information search by applying VDL-based iconic tags: an experimental studyJournal of Documentation10.1108/JD-08-2018-0127Online publication date: 8-May-2019
      • (2018)Swipe and TellACM Transactions on Information Systems10.1145/318515336:4(1-36)Online publication date: 13-Jun-2018
      • (2017)Information fusion in content based image retrievalInformation Fusion10.1016/j.inffus.2017.01.00337:C(50-60)Online publication date: 1-Sep-2017
      • (2016) The Visual Vocabulary: Skos:example and the Illustrated Artists’ Books Thesaurus Journal of Library Metadata10.1080/19386389.2015.110308615:3-4(241-251)Online publication date: 25-Jan-2016
      • Show More Cited By

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media