skip to main content
10.1145/3176349.3176384acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
research-article

Here and Now: Reality-Based Information Retrieval: Perspective Paper

Published: 01 March 2018 Publication History

Abstract

Today, the widespread use of mobile devices allows users to search information "on the go»», whenever and wherever they want, no longer confining Information Retrieval to classic desktop interfaces. We believe that technical advances in Augmented Reality will allow Information Retrieval to go even further, making use of both the users» surroundings and their abilities to interact with the physical world. In this paper, we present the fundamental concept of Reality-Based Information Retrieval, which combines the classic Information Retrieval process with Augmented Reality technologies to provide context-dependent search cues and situated visualizations of the query and the results. With information needs often stemming from real-world experiences, this novel combination has the potential to better support both Just-in-time Information Retrieval and serendipity. Based on extensive literature research, we propose a conceptual framework for Reality-Based Information Retrieval. We illustrate and discuss this framework and present two prototypical implementations, which we tested in small user studies. They demonstrate the feasibility of our concepts and inspired our discussion of notable challenges for further research in this novel and promising area.

References

[1]
Antti Ajanki, Mark Billinghurst, Toni Järvenpää, Melih Kandemir, Samuel Kaski, Markus Koskela, Mikko Kurimo, Jorma Laaksonen, Kai Puolamäki, Teemu Ruokolainen, et al. 2010. Contextual information access with augmented reality. In Proc. 2010 IEEE Int. Workshop MLSP. IEEE, 95--100.
[2]
James Allan, Bruce Croft, Alistair Moffat, and Mark Sanderson. 2012. Frontiers, Challenges, and Opportunities for Information Retrieval: Report from SWIRL 2012 the Second Strategic Workshop on Information Retrieval in Lorne. SIGIR Forum 46, 1 (May 2012), 2--32.
[3]
Jürgen Assfalg, Alberto Del Bimbo, and Pietro Pala. 2002. Three-Dimensional Interfaces for Querying by Example in Content-Based Image Retrieval. IEEE Trans. Vis. Comput. Graphics 8, 4 (Oct. 2002), 305--318.
[4]
Juhee Bae, Vidya Setlur, and Benjamin Watson. 2015. GraphTiles: A Visual Interface Supporting Browsing and Imprecise Mobile Search. In Proc. MobileHCI '15. ACM, New York, NY, USA, 63--70.
[5]
Robert Ball, Chris North, and Doug A. Bowman. 2007. Move to Improve: Promoting Physical Navigation to Increase User Performance with Large Displays. In Proc. CHI '07. ACM, New York, NY, USA, 191--200.
[6]
Daniel Belcher, Mark Billinghurst, SE Hayes, and Randy Stiles. 2003. Using Augmented Reality for Visualizing Complex Graphs in Three Dimensions. In Proc. ISMAR '03. IEEE CS, Washington, DC, USA, 84--93.
[7]
Mark Billinghurst, Adrian Clark, and Gun Lee. 2015. A Survey of Augmented Reality. Found. Trends Hum.-Comput. Interact. 8, 2--3 (March 2015), 73--272.
[8]
Ourdia Bouidghaghen, Lynda Tamine-Lechani, and Mohand Boughanem. 2009. Dynamically Personalizing Search Results for Mobile Users. In Proc. FQAS '09. Springer-Verlag, Berlin, Heidelberg, 99--110.
[9]
John Brooke. 1996. SUS: a "quick and dirty" usability scale. In Usability Evaluation in Industry. Taylor and Francis.
[10]
Michael K. Buckland and Christian Plaunt. 2000. On the Construction of Selection Systems. Library Hi Tech 12 (06 2000).
[11]
Wolfgang Büschel, Patrick Reipschläger, Ricardo Langner, and Raimund Dachselt. 2017. Investigating the Use of Spatial Interaction for 3D Data Visualization on Mobile Devices. In Proc. ISS '17. ACM, New York, NY, USA, 62--71.
[12]
Juan Pablo Carrascal and Karen Church. 2015. An In-Situ Study of Mobile App&Mobile Search Interactions. In Proc. CHI '15. ACM, New York, NY, USA, 2739--2748.
[13]
Patrick Chiu, Andreas Girgensohn, Surapong Lertsithichai, Wolf Polak, and Frank Shipman. 2005. MediaMetro: Browsing Multimedia Document Collections with a 3D City Metaphor. In Proc. MULTIMEDIA '05. ACM, New York, NY, USA, 213--214.
[14]
Karen Church, Mauro Cherubini, and Nuria Oliver. 2014. A Large-scale Study of Daily Information Needs Captured in Situ. ACM Trans. Comput.-Hum. Interact. 21, 2, Article 10 (Feb. 2014), 46 pages.
[15]
Karen Church and Barry Smyth. 2009. Understanding the Intent Behind Mobile Information Needs. In Proc. IUI '09. ACM, New York, NY, USA, 247--256.
[16]
John Eakins, Margaret Graham, and Tom Franklin. 1999. Content-based Image Retrieval. In Library and Information Briefings.
[17]
Neven ElSayed, Bruce Thomas, Kim Marriott, Julia Piantadosi, and Ross Smith. 2015. Situated Analytics. In 2015 Big Data Visual Analytics (BDVA). 1--8. https: //doi.org/10.1109/BDVA.2015.7314302{18} Barrett Ens, Ahmad Byagowi, Teng Han, Juan David Hincapié-Ramos, and Pourang Irani. 2016. Combining Ring Input with Hand Tracking for Precise, Natural Interaction with Spatial Analytic Interfaces. In Proc. SUI '16. ACM, New York, NY, USA, 99--102.
[18]
Barrett Ens, Juan David Hincapié-Ramos, and Pourang Irani. 2014. Ethereal Planes: A Design Framework for 2D Information Space in 3D Mixed Reality Environments. In Proc. SUI '14. ACM, New York, NY, USA, 2--12.
[19]
Barrett Ens and Pourang Irani. 2017. Spatial Analytic Interfaces: Spatial User Interfaces for In Situ Visual Analytics. IEEE Comp. Graph. Appl. 37, 2 (2017), 66--79.
[20]
Xin Fan, Xing Xie, Zhiwei Li, Mingjing Li, and Wei-Ying Ma. 2005. Photo-tosearch: Using Multimodal Queries to Search the Web from Mobile Devices. In Proc. MIR '05. ACM, New York, NY, USA, 143--150.
[21]
Bernd Girod, Vijay Chandrasekhar, David M Chen, Ngai-Man Cheung, Radek Grzeszczuk, Yuriy Reznik, Gabriel Takacs, Sam S Tsai, and Ramakrishna Vedantham. 2011. Mobile visual search. IEEE Signal Process. Mag. 28, 4 (2011), 61--76.
[22]
Raphael Grasset, Tobias Langlotz, Denis Kalkofen, Markus Tatzgern, and Dieter Schmalstieg. 2012. Image-driven View Management for Augmented Reality Browsers. In Proc. ISMAR '12. IEEE Computer Society, Washington, DC, USA, 177--186.
[23]
Yulan Guo, Mohammed Bennamoun, Ferdous Sohel, Min Lu, and Jianwei Wan. 2014. 3D Object Recognition in Cluttered Scenes with Local Surface Features: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 36, 11 (Nov 2014), 2270--2287.
[24]
Djoerd Hiemstra. 2009. Information Retrieval Models. In Information Retrieval. John Wiley&Sons, Ltd, 1--19.
[25]
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. 2016. Natural language object retrieval. In Proc. CVPR '16. 4555-- 4564.
[26]
Edwin L. Hutchins, James D. Hollan, and Donald A. Norman. 1985. Direct Manipulation Interfaces. Hum.-Comput. Interact. 1, 4 (Dec. 1985), 311--338.
[27]
Peter Ingwersen and Kalervo Järvelin. 2005. The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series). Springer-Verlag New York, Inc., Secaucus, NJ, USA.
[28]
Daisuke Iwai, Tatsunori Yabiki, and Kosuke Sato. 2013. View management of projected labels on nonplanar and textured surfaces. IEEE Trans. Vis. Comput. Graphics 19, 8 (2013), 1415--1424.
[29]
Masakazu Iwamura, Kai Kunze, Yuya Kato, Yuzuko Utsumi, and Koichi Kise. 2014. Haven'T We Met Before?: A Realistic Memory Assistance System to Remind You of the Person in Front of You. In Proc. AH '14. ACM, New York, NY, USA, Article 32, 4 pages.
[30]
Matt Jones, George Buchanan, Richard Harper, and Pierre-Louis Xech. 2007. Questions Not Answers: A Novel Mobile Search Technique. In Proc. CHI '07. ACM, New York, NY, USA, 155--158.
[31]
Amy K. Karlson, George G. Robertson, Daniel C. Robbins, Mary P. Czerwinski, and Greg R. Smith. 2006. FaThumb: A Facet-based Interface for Mobile Search. In Proc. CHI '06. ACM, New York, NY, USA, 711--720.
[32]
Jens Keil, Michael Zoellner, Timo Engelke, Folker Wientapper, and Michael Schmitt. 2013. Controlling and Filtering Information Density with Spatial Interaction Techniques via Handheld Augmented Reality. Springer, Berlin, Heidelberg, 49--57.
[33]
Konstantin Klamka and Raimund Dachselt. 2015. Elasticcon: Elastic Controllers for Casual Interaction. In Proc. MobileHCI '15. ACM, NY, USA, 410--419.
[34]
Jarrod Knibbe, Sue Ann Seah, and Mike Fraser. 2014. VideoHandles: Replicating Gestures to Search Through Action-camera Video. In Proc. SUI '14. ACM, New York, NY, USA, 50--53.
[35]
ByoungChul Ko and Hyeran Byun. 2002. Query-by-Gesture: An Alternative Content-Based Image Retrieval Query Scheme. J. Vis. Lang. Comput. 13 (2002), 375--390.
[36]
Jihoon Ko, Sangjin Shin, Sungkwang Eom, Minjae Song, Dong-Hoon Shin, KyongHo Lee, and Yongil Jang. 2014. Semantically Enhanced Keyword Search for Smartphones. In Proc. WWW '14 Companion. ACM, New York, NY, USA, 327--328.
[37]
Ricardo Langner, Ulrich von Zadow, Tom Horak, Annett Mitschick, and Raimund Dachselt. 2016. Content Sharing Between Spatially-Aware Mobile Phones and Large Vertical Displays Supporting Collaborative Work. Springer International Publishing, 75--96.
[38]
Gang Li, Yue Liu, and Yongtian Wang. 2017. Evaluation of Labelling Layout Methods in Augmented Reality. In Proc. VR '17. 351--352. 00000
[39]
Jacob Boesen Madsen, Markus Tatzqern, Claus B Madsen, Dieter Schmalstieg, and Denis Kalkofen. 2016. Temporal Coherence Strategies for Augmented Reality Labeling. IEEE Trans. Vis. Comput. Graphics 22, 4 (April 2016), 1415--1423.
[40]
Stephann Makri, Ann Blandford, Mel Woods, Sarah Sharples, and Deborah Maxwell. 2014. "Making my own luck": Serendipity strategies and how to support them in digital information environments. J. Assn. Inf. Sci. Tec. 65, 11 (2014), 2179--2194.
[41]
Tamás Matuszka. 2015. The Design and Implementation of Semantic WebBased Architecture for Augmented Reality Browser. In The Semantic Web. Latest Advances and New Domains. Springer, 731--739.
[42]
Lori McCay-Peet and Elaine G. Toms. 2015. Investigating serendipity: How it unfolds and what may influence it. J. Assn. Inf. Sci. Tec. 66, 7 (2015), 1463--1476.
[43]
Lori McCay-Peet, Elaine G. Toms, and E. Kevin Kelloway. 2015. Examination of relationships among serendipity, the environment, and individual differences. Information Processing&Management 51, 4 (2015), 391 -- 412.
[44]
Taniya Mishra and Srinivas Bangalore. 2010. Qme!: A Speech-based Questionanswering System on Mobile Devices. In Proc. HLT '10. Association for Computational Linguistics, Stroudsburg, PA, USA, 55--63. http://dl.acm.org/citation.cfm' id=1857999.1858006
[45]
Munehiro Nakazato and Thomas S Huang. 2001. 3D MARS: immersive virtual reality for content-based image retrieval. In Proc. ICME '01. 44--47.
[46]
Jason Orlosky, Kiyoshi Kiyokawa, Takumi Toyama, and Daniel Sonntag. 2015. Halo Content: Context-aware Viewspace Management for Non-invasive Augmented Reality. In Proc. IUI '15. ACM, New York, NY, USA, 369--373.
[47]
Bradley J Rhodes and Pattie Maes. 2000. Just-in-time Information Retrieval Agents. IBM Syst. J. 39, 3--4 (July 2000), 685--704.
[48]
Cornelis J. Van Rijsbergen. 1979. Information Retrieval (2nd ed.). ButterworthHeinemann, Newton, MA, USA.
[49]
Dieter Schmalstieg and Tobias Hollerer. 2016. Augmented reality: principles and practice. Addison-Wesley Professional.
[50]
Kimiaki Shirahama and Kuniaki Uehara. 2011. Query by Virtual Example: Video Retrieval Using Example Shots Created by Virtual Reality Techniques. In Int. Conf. on Image and Graphics '11. 829--834.
[51]
Sophie Stellmach, Sebastian Stober, Andreas Nürnberger, and Raimund Dachselt. 2011. Designing Gaze-supported Multimodal Interactions for the Exploration of Large Image Collections. In Proc. NGCA '11. ACM, New York, NY, USA, Article 1, 8 pages.
[52]
Zsolt Szalavári and Michael Gervautz. 1997. The Personal Interaction Panel -- a Two-Handed Interface for Augmented Reality. Computer Graphics Forum 16, 3 (1997), C335--C346.
[53]
Norimichi Ukita, Tomohisa Ono, and Masatsugu Kidode. 2005. Region Extraction of a Gaze Object Using the Gaze Point and View Image Sequences. In Proc. ICMI '05. ACM, NY, USA, 129--136. 00005
[54]
Nayuko Watanabe, Masayuki Okamoto, Masaaki Kikuchi, Takayuki Iida, Kenta Sasaki, Kensuke Horiuchi, and Masanori Hattori. 2011. Designing Mobile Search Interface with Query Term Extraction. Springer, Berlin, Heidelberg, 67--76.
[55]
Sean White and Steven Feiner. 2009. SiteLens: Situated Visualization Techniques for Urban Site Visits. In Proc. CHI '09. ACM, New York, NY, USA, 1117--1120.
[56]
Wesley Willett, Yvonne Jansen, and Pierre Dragicevic. 2017. Embedded Data Representations. IEEE Trans. Vis. Comput. Graphics 23, 1 (Jan 2017), 461--470.
[57]
Xing Xie, Lie Lu, Menglei Jia, Hua Li, Frank Seide, and Wei-Ying Ma. 2008. Mobile Search With Multimodal Queries. Proc. of the IEEE 96, 4 (April 2008), 589--601.
[58]
Tom Yeh, John J. Lee, and Trevor Darrell. 2008. Photo-based Question Answering. In Proc. MM '08. ACM, New York, NY, USA, 389--398.
[59]
Degi Young and Ben Shneiderman. 1993. A graphical filter/flow representation of Boolean queries: A prototype implementation and evaluation. J. Am. Soc. Inf. Sci. 44, 6 (1993), 327--339.
[60]
Stefan Zander, Chris Chiu, and Gerhard Sageder. 2012. A Computational Model for the Integration of Linked Data in Mobile Augmented Reality Applications. In Proc. I-SEMANTICS '12. ACM, New York, USA, 133--140.
[61]
Qi Zhang, Simon Zaaijer, Song Wu, and Michael S. Lew. 2014. 3D Image Browsing: The Planets. In Proc. ICMR '14. ACM, New York, USA, Article 511, 3 pages.
[62]
Yun Zhang, Hong Fu, Zhen Liang, Zheru Chi, and Dagan Feng. 2010. Eye Movement As an Interaction Mechanism for Relevance Feedback in a Content-based Image Retrieval System. In Proc. ETRA '10. ACM, New York, NY, USA, 37--40.

Cited By

View all
  • (2024)G-VOILA: Gaze-Facilitated Information Querying in Daily ScenariosProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596238:2(1-33)Online publication date: 15-May-2024
  • (2024)Mixed Reality Interaction Enhanced by Whiteboard for Product SearchProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638346(396-400)Online publication date: 10-Mar-2024
  • (2024)pARam: Leveraging Parametric Design in Extended Reality to Support the Personalization of Artifacts for Personal FabricationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642083(1-22)Online publication date: 11-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHIIR '18: Proceedings of the 2018 Conference on Human Information Interaction & Retrieval
March 2018
402 pages
ISBN:9781450349253
DOI:10.1145/3176349
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 March 2018

Permissions

Request permissions for this article.

Check for updates

Badges

  • Best Paper

Author Tags

  1. augmented reality
  2. immersive visualization
  3. in situ visual analytics
  4. reality-based information retrieval
  5. spatial user interface

Qualifiers

  • Research-article

Funding Sources

  • Bundesministerium für Bildung und Forschung

Conference

CHIIR '18
Sponsor:

Acceptance Rates

CHIIR '18 Paper Acceptance Rate 22 of 57 submissions, 39%;
Overall Acceptance Rate 55 of 163 submissions, 34%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)40
  • Downloads (Last 6 weeks)6
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)G-VOILA: Gaze-Facilitated Information Querying in Daily ScenariosProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596238:2(1-33)Online publication date: 15-May-2024
  • (2024)Mixed Reality Interaction Enhanced by Whiteboard for Product SearchProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638346(396-400)Online publication date: 10-Mar-2024
  • (2024)pARam: Leveraging Parametric Design in Extended Reality to Support the Personalization of Artifacts for Personal FabricationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642083(1-22)Online publication date: 11-May-2024
  • (2023)Immersive Sampling: Exploring Sampling for Future Creative Practices in Media-Rich, Immersive SpacesProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596131(212-229)Online publication date: 10-Jul-2023
  • (2023)Spatiality and Semantics - Towards Understanding Content Placement in Mixed RealityExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585853(1-8)Online publication date: 19-Apr-2023
  • (2023)Labeling Out-of-View Objects in Immersive Analytics to Support Situated Visual SearchingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.313351129:3(1831-1844)Online publication date: 1-Mar-2023
  • (2023)Implications and New Directions for IR Research and PracticesA Behavioral Economics Approach to Interactive Information Retrieval10.1007/978-3-031-23229-9_7(181-201)Online publication date: 18-Feb-2023
  • (2023)Formally Modeling Users in Information RetrievalA Behavioral Economics Approach to Interactive Information Retrieval10.1007/978-3-031-23229-9_2(23-64)Online publication date: 18-Feb-2023
  • (2022)What's the Situation with Situated Visualization? A Survey and Perspectives on SituatednessIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.311483528:1(107-117)Online publication date: Jan-2022
  • (2022)ABOVE & BELOW: Investigating Ceiling and Floor for Augmented Reality Content Placement2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR55827.2022.00068(518-527)Online publication date: Oct-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media