skip to main content
10.1145/2047196.2047227acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

Monte carlo methods for managing interactive state, action and feedback under uncertainty

Published: 16 October 2011 Publication History

Abstract

Current input handling systems provide effective techniques for modeling, tracking, interpreting, and acting on user input. However, new interaction technologies violate the standard assumption that input is certain. Touch, speech recognition, gestural input, and sensors for context often produce uncertain estimates of user inputs. Current systems tend to remove uncertainty early on. However, information available in the user interface and application can help to resolve uncertainty more appropriately for the end user. This paper presents a set of techniques for tracking the state of interactive objects in the presence of uncertain inputs. These techniques use a Monte Carlo approach to maintain a probabilistically accurate description of the user interface that can be used to make informed choices about actions. Samples are used to approximate the distribution of possible inputs, possible interactor states that result from inputs, and possible actions (callbacks and feedback) interactors may execute. Because each sample is certain, the developer can specify most of the behavior of interactors in a familiar, non-probabilistic fashion. This approach retains all the advantages of maintaining information about uncertainty while minimizing the need for the developer to work in probabilistic terms. We present a working implementation of our framework and illustrate the power of these techniques within a paint program that includes three different kinds of uncertain input.

Supplementary Material

JPG File (fp157.jpg)
WMV File (fp157.wmv)

References

[1]
Arulampalam, M., Maskell, S., Gordon, N., Clapp, T. "A tutorial on particle filters for on-line nonlinear/non-Gaussian Bayesian tracking," IEEE Trans. on Signal Proc. 50(2), 16--17, 2002.
[2]
Bau, O., Mackay, W. "OctoPocus: a dynamic guide for learning gesture-based command sets." in Proc. UIST 2008, 37--46.
[3]
Bourguet, M. L. "A Toolkit for Creating and Testing Multimodal Interface Designs." Posters and Demos from UIST'02, Paris, Oct. 2002, 29--30.
[4]
Chang, B.-W., Mackinlay, J.D., Zellweger,P.T., and Igarashi, T. A negotiation architecture for fluid documents. in Proc. UIST '98, 123--132.
[5]
Dumas, B., Lalannel, D., Guinard, D., Koenig, R., and Ingold, R. "Strengths and weaknesses of software architectures for the rapid creation of tangible and multimodal interfaces." in Proc. TEI 2008, 47--54.
[6]
Dumas, B., Lalannel, D., Oviatt, S. "Multimodal Interfaces: A Survey of Principles, Models and Frameworks." Human Machine Interaction. vol. 5440, 3--26, 2009.
[7]
Gordon, N. J., Salmond, D. J., Smith, A. F. M. "Novel approach to nonlinear/non-Gaussian Bayesian state estimation". IEEE Proc. F on Radar and Signal Proc. 140(2):107--113.
[8]
Grover, D., King, M., Kushler, C., "Reduced keyboard disambiguating computer." U.S. Patent 5818437, Oct. 6, 1998.
[9]
Hudson, S. E., Newell, G. L. "Probabilistic state machines: dialog management for inputs with uncertainty." in Proc. UIST 1992, 199--208.
[10]
Jacob, R., Deligiannidis, L, Morrison, S. "A software model and specification language for non-WIMP user interfaces." TOCHI 6(1):1--46, 1999.
[11]
1Mankoff, J., Hudson, S. E., Abowd, G. D. "Interaction techniques for ambiguity resolution in recognition-based interfaces." in Proc. UIST 2000, 11 -- 20.
[12]
Mankoff, J., Hudson, S. E., Abowd, G. D. "Providing integrated toolkit-level support for ambiguity in recognition-based interfaces." in Proc. CHI 2000, 368--375.
[13]
Metropolis, N., Ulam, S. "The Monte Carlo Method". J. of the American Statistical Association 44(247):335--341, 1949.
[14]
Myers, B., and Kosbie, and Kosbie, D. "Reusable hierarchical command objects." in Proc. CHI 1996, 260--267.
[15]
Odell, J. "The use of context in large vocabulary speech recognition." PhD thesis, Univ. of Cambridge, England, 1995.
[16]
Olsen, D. "Propositional production systems for dialog description." in Proc. CHI 1990, 57--64.
[17]
Oviatt, S. "Ten myths of multimodal interaction." CACM 42(11):74--81, 1999.
[18]
Schwarz, J., Hudson, S., Mankoff, J. "A Framework for Robust and Flexible Handling of Inputs with Uncertainty." in Proc. UIST 2010, 47--56.
[19]
Wasserman, A.I. "Extending State Transition Diagrams for the Specification of Human-Computer Interaction." IEEE Trans. Software Engineering. 11(8):699--713, 1985.

Cited By

View all
  • (2024)TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676330(1-16)Online publication date: 13-Oct-2024
  • (2022)Select or Suggest? Reinforcement Learning-based Method for High-Accuracy Target Selection on TouchscreensProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517472(1-15)Online publication date: 29-Apr-2022
  • (2021)Building Adaptive Touch Interfaces—Case Study 6Intelligent Computing for Interactive System Design10.1145/3447404.3447426(379-406)Online publication date: 23-Feb-2021
  • Show More Cited By

Index Terms

  1. Monte carlo methods for managing interactive state, action and feedback under uncertainty

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UIST '11: Proceedings of the 24th annual ACM symposium on User interface software and technology
    October 2011
    654 pages
    ISBN:9781450307161
    DOI:10.1145/2047196
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 16 October 2011

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. dialog specification
    2. finite state machines
    3. probabilistic modeling
    4. uncertain input

    Qualifiers

    • Research-article

    Conference

    UIST '11

    Acceptance Rates

    UIST '11 Paper Acceptance Rate 67 of 262 submissions, 26%;
    Overall Acceptance Rate 561 of 2,567 submissions, 22%

    Upcoming Conference

    UIST '25
    The 38th Annual ACM Symposium on User Interface Software and Technology
    September 28 - October 1, 2025
    Busan , Republic of Korea

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)25
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676330(1-16)Online publication date: 13-Oct-2024
    • (2022)Select or Suggest? Reinforcement Learning-based Method for High-Accuracy Target Selection on TouchscreensProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517472(1-15)Online publication date: 29-Apr-2022
    • (2021)Building Adaptive Touch Interfaces—Case Study 6Intelligent Computing for Interactive System Design10.1145/3447404.3447426(379-406)Online publication date: 23-Feb-2021
    • (2021)Model-based Engineering of Feedforward Usability Function for GUI WidgetsInteracting with Computers10.1093/iwcomp/iwab01433:1(73-91)Online publication date: 18-May-2021
    • (2020)FORTNIoTProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/34322254:4(1-24)Online publication date: 18-Dec-2020
    • (2020)Using Bayes' Theorem for Command Input: Principle, Models, and ApplicationsProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376771(1-15)Online publication date: 21-Apr-2020
    • (2020)Disambiguation Techniques for Freehand Object Manipulations in Virtual Reality2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)10.1109/VR46266.2020.1581293094740(285-292)Online publication date: Mar-2020
    • (2020)Disambiguation Techniques for Freehand Object Manipulations in Virtual Reality2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)10.1109/VR46266.2020.00048(285-292)Online publication date: Mar-2020
    • (2019)FortunettesProceedings of the ACM on Human-Computer Interaction10.1145/33311623:EICS(1-20)Online publication date: 13-Jun-2019
    • (2019)Cluster TouchProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300257(1-14)Online publication date: 2-May-2019
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media