skip to main content
10.1145/604045.604058acmconferencesArticle/Chapter ViewAbstractPublication PagesiuiConference Proceedingsconference-collections
Article

Multimodal event parsing for intelligent user interfaces

Published:12 January 2003Publication History

ABSTRACT

Many intelligent interfaces must recognize patterns of user activity that cross a variety of different input channels. These multimodal interfaces offer significant challenges to both the designer and the software engineer. The designer needs a method of expressing interaction patterns that has the power to capture real use cases and a clear semantics. The software engineer needs a processing model that can identify the described interaction patterns efficiently while maintaining meaningful intermediate state to aid in debugging and system maintenanceIn this paper, we describe an input model, a general recognition model, and a series of important classes of recognition parsers with useful computational characteristics; that is, we can say with some certainty how efficient the recognizers will be, and the kind of patterns the recognizers will accept. Examples illustrate the ability of these recognizers to integrate information from multiple channels across varying time intervals.

References

  1. Allen, J., Maintaining knowledge about temporal intervals. Communications of the ACM, 1983. 26(11): 832--843. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Eclipse.org, Eclipse Integrated Development Environment. 2002, http://www.eclipse.org.Google ScholarGoogle Scholar
  3. Firby, R.J., et al. An architecture for vision and action. Proceedings of International Joint Conference on Artificial Intelligence. 1995. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Fitzgerald, W., Building Embedded Conceptual Parsers. Unpublished Ph.D. Thesis, Northwestern University, 1994. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Fitzgerald, W. and Firby, R.J. The Dynamic Predictive Memory Architecture: Integrating language with task execution. Proceedings of IEEE Symposia on Intelligence and Systems. 1998. Washington, DC. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Flachsbart, J., Franklin, D., and Hammond, K. Improving human computer interaction in a classroom environment using computer vision. Proceedings of Intelligent User Interfaces. 2000. New Orleans, LA: ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Horswill, I., Specialization of Perceptual Processes. Unpublished Ph.D. Thesis, Massachusetts Institute of Technology, 1993. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Johnson, M. Unification-based multimodal parsing. Proceedings of COLING-ACL 98. 1998. Montreal, Quebec: ACL Publications. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Johnson, M. and Bagalore, S. Finite-state multimodal understanding and parsing. Proceedings of COLING-2000. 2002. Saarbrücken, Germany: ACL Publications. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Martin, C.E., Case-based parsing and Micro-DMAP, in Inside Case-Based Reasoning, C.K. Riesbeck and R.C. Schank, Editors. 1989, Lawrence Erlbaum Associates: Hillsdale, NJ.Google ScholarGoogle Scholar
  11. Martin, D.L., Cheyer, A.J., and Moran, D.B., The Open Agent Architecture: A framework for building distributed software systems. Applied Artificial Intelligence, 1999. 13: 91--128. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Moran, D.B., et al. Multimodal user interfaces in the Open Agent Architecture. Proceedings of Intelligent User Interfaces. 1997. Orlando, FL: ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Oviatt, S., Ten myths of multimodal interaction. Communications of the ACM, 1999. 42(11): 74--81. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Schreckenghost, D.C., et al., Intelligent control of life support for space missions. IEEE Intelligent Systems, 2002. 17(5): 24--31. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Multimodal event parsing for intelligent user interfaces

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Conferences
      IUI '03: Proceedings of the 8th international conference on Intelligent user interfaces
      January 2003
      344 pages
      ISBN:1581135866
      DOI:10.1145/604045

      Copyright © 2003 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 12 January 2003

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • Article

      Acceptance Rates

      Overall Acceptance Rate683of2,684submissions,25%

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader