skip to main content
10.1145/1452392.1452419acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
research-article

A high-performance dual-wizard infrastructure for designing speech, pen, and multimodal interfaces

Published: 20 October 2008 Publication History

Abstract

The present paper reports on the design and performance of a novel dual-Wizard simulation infrastructure that has been used effectively to prototype next-generation adaptive and implicit multimodal interfaces for collaborative groupwork. This high-fidelity simulation infrastructure builds on past development of single-wizard simulation tools for multiparty multimodal interactions involving speech, pen, and visual input [1]. In the new infrastructure, a dual-wizard simulation environment was developed that supports (1) real-time tracking, analysis, and system adaptivity to a user's speech and pen paralinguistic signal features (e.g., speech amplitude, pen pressure), as well as the semantic content of their input. This simulation also supports (2) transparent user training to adapt their speech and pen signal features in a manner that enhances the reliability of system functioning, i.e., the design of mutually-adaptive interfaces. To accomplish these objectives, this new environment also is capable of handling (3) dynamic streaming digital pen input. We illustrate the performance of the simulation infrastructure during longitudinal empirical research in which a user-adaptive interface was designed for implicit system engagement based exclusively on users' speech amplitude and pen pressure [2]. While using this dual-wizard simulation method, the wizards responded successfully to over 3,000 user inputs with 95-98% accuracy and a joint wizard response time of less than 1.0 second during speech interactions and 1.65 seconds during pen interactions. Furthermore, the interactions they handled involved naturalistic multiparty meeting data in which high school students were engaged in peer tutoring, and all participants believed they were interacting with a fully functional system. This type of simulation capability enables a new level of flexibility and sophistication in multimodal interface design, including the development of implicit multimodal interfaces that place minimal cognitive load on users during mobile, educational, and other applications.

References

[1]
Arthur, A., Lunsford, R., Wesson, M., and Oviatt, S. L. Prototyping novel collaborative multimodal systems: Simulation, data collection and analysis tools for the next decade, Proc. ICMI, 2006.
[2]
Oviatt, S. L., Swindells, C., and Arthur, A. Implicit user-adaptive system engagement in speech and pen interfaces, Conference on Human Factors in Computing Systems (CHI '08), CHI Letters, ACM: New York, N.Y., 2008, 969--978.
[3]
Cohen, P. R. and McGee, D. R. Tangible Multimodal Interfaces for Safety-Critical Applications. CACM 47(1), 2004, 41--46.
[4]
Dahlback, N., Jonsson, A., & Ahrenberg, L., Wizard-of-Oz Studies - Why and How, in Proc. of the Int'l Workshop on Intelligent User Interfaces, 1993.
[5]
Lunsford, R., and Oviatt, S. Human perception of intended addressee during computer-assisted meetings, Proc. of Int'l Conf. on Multimodal Interfaces, 2006, 20--27.
[6]
Martin, D., Cheyer, A. & Moran, D. The Open Agent Architecture: A framework for building distributed software systems. Applied Artificial Intelligence: An International Journal. 13(1-2), 1999.
[7]
Norrie, M. C., Signer, B. and Weibel, N., General Framework for the Rapid Development of Interactive Paper Applications, CoPADD 2006, Workshop on Collaborating over Paper and Digital Documents 2006
[8]
Oviatt, S. L., Cohen, P. R., Fong, M. W., and Frank, M. P. A rapid semi-automatic simulation technique for investigating interactive speech and handwriting. In Ohala, J., et al., (Eds.), Proc. of the Int'l Conference on Spoken Language Processing, 2 Univ. of Alberta, 1992, 1351--1354.
[9]
Oviatt, S. L., Coulston R., Tomko S., Xiao, B., Lunsford, R. Wesson, M. & Carmichael L., Toward a Theory of Organized Multimodal Integration Patterns during Human-Computer Interaction, Proc. of the Int'l Conf. on Multimodal Interfaces, ACM Press, 2003, 44--51.
[10]
Salber, D. & Coutaz, J., Applying the Wizard-of-Oz technique to the study of multimodal systems, Proc. of the European Workshop on HCI, 1993.
[11]
Yeh, R. B., Liao, C. Klemmer, S. Guimbretière F., Lee, B., Kakaradov, B., Stamberger, J., and Paepcke. A., ButterflyNet: A Mobile Capture and Access System for Field Biology Research. Proc. of CHI'06, pp. 571--580.

Cited By

View all
  • (2023)Wizundry: A Cooperative Wizard of Oz Platform for Simulating Future Speech-based Interfaces with Multiple WizardsProceedings of the ACM on Human-Computer Interaction10.1145/35795917:CSCW1(1-34)Online publication date: 16-Apr-2023
  • (2022)Informing Future Gesture Elicitation Studies for Interactive Applications that Use Radar SensingProceedings of the 2022 International Conference on Advanced Visual Interfaces10.1145/3531073.3534475(1-3)Online publication date: 6-Jun-2022
  • (2022)The Impacts of Referent Display on Gesture and Speech ElicitationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.320309028:11(3885-3895)Online publication date: Nov-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '08: Proceedings of the 10th international conference on Multimodal interfaces
October 2008
322 pages
ISBN:9781605581989
DOI:10.1145/1452392
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2008

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. collaborative meetings
  2. dual-wizard protocol
  3. high-fidelity simulation
  4. implicit system engagement
  5. multi-stream multimodal data
  6. pen pressure
  7. speech amplitude
  8. streaming digital pen and paper
  9. wizard-of-oz

Qualifiers

  • Research-article

Conference

ICMI '08
Sponsor:
ICMI '08: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERFACES
October 20 - 22, 2008
Crete, Chania, Greece

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)17
  • Downloads (Last 6 weeks)3
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Wizundry: A Cooperative Wizard of Oz Platform for Simulating Future Speech-based Interfaces with Multiple WizardsProceedings of the ACM on Human-Computer Interaction10.1145/35795917:CSCW1(1-34)Online publication date: 16-Apr-2023
  • (2022)Informing Future Gesture Elicitation Studies for Interactive Applications that Use Radar SensingProceedings of the 2022 International Conference on Advanced Visual Interfaces10.1145/3531073.3534475(1-3)Online publication date: 6-Jun-2022
  • (2022)The Impacts of Referent Display on Gesture and Speech ElicitationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.320309028:11(3885-3895)Online publication date: Nov-2022
  • (2022)User-Defined Foot Gestures for Eyes-Free Interaction in Smart Shower RoomsInternational Journal of Human–Computer Interaction10.1080/10447318.2022.210926039:20(4139-4161)Online publication date: 18-Aug-2022
  • (2017)Multimodal speech and pen interfacesThe Handbook of Multimodal-Multisensor Interfaces10.1145/3015783.3015795(403-447)Online publication date: 24-Apr-2017
  • (2015)The Paradigm Shift to Multimodality in Contemporary Computer InterfacesSynthesis Lectures on Human-Centered Informatics10.2200/S00636ED1V01Y201503HCI0308:3(1-243)Online publication date: 13-Apr-2015
  • (2015)The WOZ RecognizerACM Transactions on Interactive Intelligent Systems10.1145/27430295:3(1-38)Online publication date: 16-Oct-2015

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media