skip to main content
10.1145/1180995.1181012acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
Article

GSI demo: multiuser gesture/speech interaction over digital tables by wrapping single user applications

Published: 02 November 2006 Publication History

Abstract

Most commercial software applications are designed for a single user using a keyboard/mouse over an upright monitor. Our interest is exploiting these systems so they work over a digital table. Mirroring what people do when working over traditional tables, we want to allow multiple people to interact naturally with the tabletop application and with each other via rich speech and hand gestures. In previous papers, we illustrated multi-user gesture and speech interaction on a digital table for geospatial applications -- Google Earth, Warcraft III and The Sims. In this paper, we describe our underlying architecture: GSI Demo. First, GSI Demo creates a run-time wrapper around existing single user applications: it accepts and translates speech and gestures from multiple people into a single stream of keyboard and mouse inputs recognized by the application. Second, it lets people use multimodal demonstration -- instead of programming -- to quickly map their own speech and gestures to these keyboard/mouse inputs. For example, continuous gestures are trained by saying "Computer, when I do [one finger gesture], you do [mouse drag]". Similarly, discrete speech commands can be trained by saying "Computer, when I say [layer bars], you do [keyboard and mouse macro]". The end result is that end users can rapidly transform single user commercial applications into a multi-user, multimodal digital tabletop system.

References

[1]
Boyle, M. and Greenberg, S. Rapidly Prototyping Multimedia Groupware. Proc Distributed Multimedia Systems (DMS'05), Knowledge Systems Institute, 2005.
[2]
Cao, X. and Balakrishnan, R. Evaluation of an online adaptive gesture interface with command prediction. Proc Graphics Interface, 2005. 187--194.
[3]
Cohen, P. R., Coulston, R. and Krout, K., Multimodal interaction during multiparty dialogues: Initial results. Proc IEEE Int'l Conf. Multimodal Interfaces, 2002, 448--452.
[4]
Cohen, P. R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L. and Clow, J., QuickSet: Multimodal interaction for distributed applications. Proc. ACM Multimedia, 1997, 31--40.
[5]
Cypher, A. Watch What I Do: Programming by Demonstration. MIT Press, 1993.
[6]
Dietz, P. H., Leigh, D. L., DiamondTouch: A Multi-User Touch Technology, Proc ACM UIST, 2001. 219--226.
[7]
Greenberg, S. and Boyle, M. Customizable physical interfaces for interacting with conventional applications. Proc. ACM UIST Conference, 2002, 31--40.
[8]
Gutwin, C. and Greenberg, S. Design for individuals, design for groups: Tradeoffs between power and workspace awareness. Proc ACM CSCW, 1998, 207--216.
[9]
Lunsford, R., Oviatt, S., and Coulston, R., Audio-visual cues distinguishing self- from system-directed speech in younger and older adults. Proc. ICMI, 2005, 167--174.
[10]
McGee, D. R. and Cohen, P. R., Creating tangible interfaces by augmenting physical objects with multimodal language. Proc ACM Conf. Intelligent User Interfaces, 2001, 113--119.
[11]
Oviatt, S. L. Ten myths of multimodal interaction, Comm. ACM, 42(11), 1999, 74--81.
[12]
Tse, E., Shen, C., Greenberg, S. and Forlines, C. Enabling Interaction with Single User Applications through Speech and Gestures on a Multi-User Tabletop. Proc. AVI 2006.
[13]
Tse, E., Greenberg, S., Shen, C. and Forlines, C. (2006) Multimodal Multiplayer Tabletop Gaming. Proc. Workshop on Pervasive Games 2006.
[14]
Wu, M., Shen, C., Ryall, K., Forlines, C., Balakrishnan, R., Gesture Registration, Relaxation, and Reuse for Multi-Point Direct-Touch Surfaces, Proc. TableTop2006. 183--190.

Cited By

View all
  • (2021)“You, Move There!”: Investigating the Impact of Feedback on Voice Control in Virtual EnvironmentsProceedings of the 3rd Conference on Conversational User Interfaces10.1145/3469595.3469609(1-9)Online publication date: 27-Jul-2021
  • (2017)A Comparative Study of Map Exploration Interfaces for Multi-Touch TabletopsInternational Journal of Human–Computer Interaction10.1080/10447318.2017.127889333:8(677-691)Online publication date: 11-Jan-2017
  • (2014)Studying Natural Interaction in Multimodal, Multi-Surface, Multiuser ScenariosEmerging Research and Trends in Interactivity and the Human-Computer Interface10.4018/978-1-4666-4623-0.ch008(160-181)Online publication date: 2014
  • Show More Cited By

Index Terms

  1. GSI demo: multiuser gesture/speech interaction over digital tables by wrapping single user applications

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces
    November 2006
    404 pages
    ISBN:159593541X
    DOI:10.1145/1180995
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 November 2006

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. digital tables
    2. multimodal input
    3. programming by demonstration

    Qualifiers

    • Article

    Conference

    ICMI06
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 453 of 1,080 submissions, 42%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)8
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2021)“You, Move There!”: Investigating the Impact of Feedback on Voice Control in Virtual EnvironmentsProceedings of the 3rd Conference on Conversational User Interfaces10.1145/3469595.3469609(1-9)Online publication date: 27-Jul-2021
    • (2017)A Comparative Study of Map Exploration Interfaces for Multi-Touch TabletopsInternational Journal of Human–Computer Interaction10.1080/10447318.2017.127889333:8(677-691)Online publication date: 11-Jan-2017
    • (2014)Studying Natural Interaction in Multimodal, Multi-Surface, Multiuser ScenariosEmerging Research and Trends in Interactivity and the Human-Computer Interface10.4018/978-1-4666-4623-0.ch008(160-181)Online publication date: 2014
    • (2014)Gaze enhanced speech recognition for truly hands-free and efficient text input during HCIProceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: the Future of Design10.1145/2686612.2686679(426-429)Online publication date: 2-Dec-2014
    • (2014)Multimodal Input for Perceptual User InterfacesInteractive Displays10.1002/9781118706237.ch9(285-312)Online publication date: 12-Jul-2014
    • (2013)Compilation of ReferencesEmerging Research and Trends in Interactivity and the Human-Computer Interface10.4018/978-1-4666-4623-0.chcrf(0-0)Online publication date: 31-Oct-2013
    • (2013)A usability study of multimodal input in an augmented reality environmentVirtual Reality10.1007/s10055-013-0230-017:4(293-305)Online publication date: 1-Nov-2013
    • (2012)Personal assistant agents and multi-agent middleware for CSCWProceedings of the 2012 IEEE 16th International Conference on Computer Supported Cooperative Work in Design (CSCWD)10.1109/CSCWD.2012.6221800(72-79)Online publication date: May-2012
    • (2012)Multimodal Interfaces for Augmented RealityExpanding the Frontiers of Visual Analytics and Visualization10.1007/978-1-4471-2804-5_25(449-465)Online publication date: 2012
    • (2011)Speak up your mindProceedings of the 10th Brazilian Symposium on Human Factors in Computing Systems and the 5th Latin American Conference on Human-Computer Interaction10.5555/2254436.2254472(202-211)Online publication date: 25-Oct-2011
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media