Skip to main content
SpringerLink
Log in

Context-Aware Querying for Multimodal Search Engines

  • Conference paper
Advances in Multimedia Modeling (MMM 2012)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 7131))

Included in the following conference series:

Abstract

Multimodal interaction provides the user with multiple modes of interacting with a system, such as gestures, speech, text, video, audio, etc. A multimodal system allows for several distinct means for input and output of data. In this paper, we present our work in the context of the I-SEARCH project, which aims at enabling context-aware querying of a multimodal search framework including real-world data such as user location or temperature. We introduce the concepts of MuSeBag for multimodal query interfaces, UIIFace for multimodal interaction handling, and CoFind for collaborative search as the core components behind the I-SEARCH multimodal user interface, which we evaluate via a user study.

This work is partly funded by the EU FP7 I-SEARCH project under project reference 248296. We would like to thank all of the partners in the I-SEARCH project for their support.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Adobe Flash Platform, http://www.adobe.com/flashplatform/

  2. Amershi, S., Morris, M.R.: Co-located Collaborative Web Search: Understanding Status Quo Practices. In: Proceedings of the 27th International Conference on Human Factors in Computing Systems, CHIEA 2009, pp. 3637–3642. ACM, New York (2009)

    Google Scholar 

  3. Barnett, J., Group, M.I.W.: Multimodal Architecture and Interfaces (July 2011) http://www.w3.org/TR/mmi-arch

  4. Block, S., Popescu, A.: DeviceOrientation Event Specification, Editor’s Draft (2011), http://dev.w3.org/geo/api/spec-source-orientation.html

  5. Chang, J., Bourguet, M.-L.: Usability Framework for the Design and Evaluation of Multimodal Interaction. In: Proceedings of the 22nd British HCI Group Annual Conference on People and Computers: Culture, Creativity, Interaction, BCS-HCI 2008, vol. 2, pp. 123–126. British Computer Society (2008)

    Google Scholar 

  6. Etemad, E.J.: Cascading Style Sheets (CSS) Snapshot (2010), W3C Working Group Note (May 12, 2011), http://www.w3.org/TR/CSS/

  7. EtherPad, Collaborative Text Editor, https://github.com/ether/pad

  8. FlashCanvas, JavaScript Library, http://flashcanvas.net/

  9. Google Docs, Collaborative Document Editing, https://docs.google.com/

  10. Google Search by image, Blog Post, http://googleblog.blogspot.com/2011/06/knocking-down-barriers-to-knowledge.html

  11. Hearst, M.A.: Search User Interfaces, 1st edn. Cambridge University Press, New York (2009)

    Book  Google Scholar 

  12. Hearst, M.A.: Emerging Trends in Search User Interfaces. In: Proceedings of the 22nd ACM Conference on Hypertext and Hypermedia, HT 2011, pp. 5–6. ACM, New York (2011)

    Google Scholar 

  13. Hickson, I.: W3C – The WebSocket API Editor’s Draft (2011), http://dev.w3.org/html5/websockets

  14. Huang, S.-T., Tsai, T.-H., Chang, H.-T.: The UI issues for the search engine. In: 11th IEEE International Conference on ComputerAided Design and Computer Graphics, pp. 330–335 (2009)

    Google Scholar 

  15. Hickson, I.: HTML5, A Vocabulary and Associated APIs for HTML and XHTML, W3C Working Draft (May 25, 2011), http://www.w3.org/TR/html5/

  16. Morris, M.R.: A Survey of Collaborative Web Search Practices. In: Proceedings of the 26th Annual SIGCHI Conference on Human Factors in Computing Systems, CHI 2008, pp. 1657–1660. ACM, New York (2008)

    Google Scholar 

  17. Morris, M.R., Horvitz, E.: SearchTogether: an interface for collaborative web search. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, UIST 2007, pp. 3–12. ACM, New York (2007)

    Google Scholar 

  18. Nigay, L.: Design Space for Multimodal Interaction. In: IFIP Congress Topical Sessions, pp. 403–408 (2004)

    Google Scholar 

  19. Daras, P., Axenopoulos, A., Darlagiannis, V., Tzovaras, D., Le Bourdon, X., Joyeux, L., Verroust-Blondet, A., Croce, V., Steiner, T., Massari, A., et al.: Introducing a Unified Framework for Content Object Description. International Journal of Multimedia Intelligence and Security, Special Issue on “Challenges in Scalable Context Aware Multimedia Computing (2010), http://www.inderscience.com/browse/index.php?journalID=359

  20. Pickens, J., Golovchinsky, G., Shah, C., Qvarfordt, P., Back, M.: Algorithmic Mediation for Collaborative Exploratory Search. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2008, pp. 315–322. ACM, New York (2008)

    Google Scholar 

  21. Quesenbery, W., Jarrett, C., Roddis, I., Allen, S., Stirling, V.: Designing for Search: Making Information Easy to Find. Technical report, Whitney Interactive Design (June 2008)

    Google Scholar 

  22. Quesenbery, W., Jarrett, C., Roddis, I., Stirling, V., Allen, S.: The Many Faces of User Experience (Presentation), Baltimore, Maryland, USA (June 16-20, 2008) http://www.usabilityprofessionals.org/

  23. Rigas, D., Ciuffreda, A.: An Empirical Investigation of Multimodal Interfaces for Browsing Internet Search Results. In: Proceedings of the 7th International Conference on Applied Informatics and Communications, pp. 194–199. World Scientific and Engineering Academy and Society, Stevens Point, Wisconsin, USA (2007)

    Google Scholar 

  24. Roscher, D., Blumendorf, M., Albayrak, S.: A Meta User Interface to Control Multimodal Interaction in Smart Environments. In: Proceedings of the 14th International Conference on Intelligent User Interfaces, IUI 2009, pp. 481–482. ACM, New York (2009)

    Google Scholar 

  25. Serrano, M., Nigay, L., Lawson, J.-Y.L., Ramsay, A., Murray-Smith, R., Denef, S.: The Open Interface Framework: A Tool for Multimodal Interaction. In: CHI 2008 Extended Abstracts on Human Factors in Computing Systems, pp. 3501–3506. ACM, New York (2008)

    Google Scholar 

  26. Sreekanth, N.S., Pal, S.N., Thomas, M., Haassan, A., Narayanan, N.K.: Multimodal Interface: Fusion of Various Modalities. International Journal of Information Studies 1(2) (2009)

    Google Scholar 

  27. TinEye Image Search Engine, http://www.tineye.com/

  28. W3C. W3C – Device APIs and Policy Working Group (2011), http://www.w3.org/2009/dap

  29. Zagoris, K., Arampatzis, A., Chatzichristofis, S.A.: www.MMRetrieval.net: A Multimodal Search Engine. In: Proceedings of the 3rd International Conference on SImilarity Search and Applications, SISAP 2010, pp. 117–118. ACM, New York (2010)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Erfurt University of Applied Sciences, Germany

    Jonas Etzold & Paul Grimm

  2. Google Germany GmbH, ABC-Str. 19, 20354, Hamburg, Germany

    Arnaud Brousseau & Thomas Steiner

Authors
  1. Jonas Etzold
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arnaud Brousseau
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul Grimm
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thomas Steiner
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institute of Information Technology, Alpen-Adria-Universität Klagenfurt, Universitätsstr. 65-67, 9020, Klagenfurt, Austria

    Klaus Schoeffmann

  2. EURECOM, 2229 Rout des Crêtes, BP 193, 06904, Sophia Antipolis Cedex, France

    Bernard Merialdo

  3. School of Computer Science, Carnegie Mellon University, 5000 Forbes Ave, 15213-3890, Pittsburgh, PA, USA

    Alexander G. Hauptmann

  4. Department of Computer Science, City University of Hong Kong, Tat Chee Ave, Kowloon, Hong Kong

    Chong-Wah Ngo

  5. Department of Electronic and Electrical Engineering, University College London, Roberts Building, Torrington Place, WC1E 7JE, London, UK

    Yiannis Andreopoulos

  6. Institute of Software Technology and Interactive Systems, Vienna University of Technology, Favoritenstrasse 9-11 188/2, 1040, Vienna, Austria

    Christian Breiteneder

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Etzold, J., Brousseau, A., Grimm, P., Steiner, T. (2012). Context-Aware Querying for Multimodal Search Engines. In: Schoeffmann, K., Merialdo, B., Hauptmann, A.G., Ngo, CW., Andreopoulos, Y., Breiteneder, C. (eds) Advances in Multimedia Modeling. MMM 2012. Lecture Notes in Computer Science, vol 7131. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-27355-1_77

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-27355-1_77

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-27354-4

  • Online ISBN: 978-3-642-27355-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation