Skip to main content
Springer Nature Link
Log in

Multimodal maps: An agent-based approach

  • Conference paper
  • First Online:
Multimodal Human-Computer Communication (CMC 1995)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 1374))

Included in the following conference series:

  • 364 Accesses

Abstract

In this paper, we discuss how multiple input modalities may be combined to produce more natural user interfaces. To illustrate this technique, we present a prototype map-based application for a travel planning domain. The application is distinguished by a synergistic combination of handwriting, gesture and speech modalities; access to existing data sources including the World Wide Web; and a mobile handheld interface. To implement the described application, a hierarchical distributed network of heterogeneous software agents was augmented by appropriate functionality for developing synergistic multimodal applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Allegayer, J., Jansen-Winkeln, R., Reddig, C. and Reithinger, N. (1989) Bidirectional use of knowledge in the multi-modal NL access system XTRA. In Proceedings of IJCAI-89, Detroit, pp. 1492–1497.

    Google Scholar 

  • Bolt, R. (1980) Put that there: Voice and Gesture at the Graphic Interface, Computer Graphics, 14(3), pp. 262–270.

    Article  MathSciNet  Google Scholar 

  • Cohen, M., Murveit, H., Bernstein, J., Price, P., and Weintraub, M. (1990) The DE-CIPHER Speech Recognition System. In 1990 IEEE ICASSP, pp. 77–80.

    Google Scholar 

  • Cohen, P. (1992) The role of natural language in a multimodal interface. In Proceedings of UIST'92, pp. 143–149.

    Google Scholar 

  • Cohen, P.R., Cheyer, A., Wang, M. and Baeg, S.C. (1994) An Open Agent Architecture. In Proceedings AAAI'94-SA, Stanford, pp. 1–8.

    Google Scholar 

  • Dauphin DTR-1 User's Manual, Dauphin Technology, Inc., Lombard, Ill 60148.

    Google Scholar 

  • Faure, C. and Julia, L. (1994) An Agent-Based Architecture for a Multimodal Interface. In Proceedings AAAI'94 — IM4S, Stanford, pp. 82–86.

    Google Scholar 

  • Genesereth, M. and Singh, N.P. (1994) A knowledge sharing approach to software interoperation, unpublished manuscript, Computer Science Department, Stanford University.

    Google Scholar 

  • Telescript Product Documentation (1995), General Magic Inc.

    Google Scholar 

  • Koons, D.B., Sparrell, C.J., and Thorisson, K.R. (1993) Integrating Simultaneous Input from Speech, Gaze and Hand Gestures. In Intelligent Multimedia Interfaces, Maybury, M.T. (ed.), Menlo Park: AAAI Press/MIT Press.

    Google Scholar 

  • Maybury, M.T. (ed.) (1993) Intelligent Multimedia Interfaces, Menlo Park: AAAI Press/MIT Press.

    Google Scholar 

  • Neal, J.G., and Shapiro, S.C. (1991) Intelligent Multi-media Interface Technology. In Intelligent User Interfaces, Sullivan, J.W. and Tyler, S.W. (eds.), Reading: Addison-Wesley Pub. Co., pp. 11–43.

    Google Scholar 

  • Nigay, L. and Coutaz, J. (1993) A Design Space for Multimodal Systems: Concurrent Processing and Data Fusion. In Proceedings InterCHI'93, Amsterdam, ACM Press, pp. 172–178.

    Google Scholar 

  • Object Management Group (1991) The Common Object Request Broker: Architecture and Specification, OMG Document Number 91.12.1.

    Google Scholar 

  • Oviatt, S. (1994) Toward Empirically-Based Design of Multimodal Dialogue Systems. In Proceedings of AAAI'94 — IM4S, Stanford, pp. 30–36.

    Google Scholar 

  • Oviatt, S. and Olsen, E. (1994) Integration Themes in Multimodal Human-Computer Interaction. In Proceedings of ICSLP'94, Yokohama, pp. 551–554.

    Google Scholar 

  • Park, S.K., Choi J.M., Myeong-Wuk J., Lee G.L., and Lim Y.H. (submitted for publication), MASCOS: A Multi-Agent System as the Computer Secretary.

    Google Scholar 

  • Rhyne J. (1987) Dialogue Management for Gestural Interfaces, Computer Graphics, 21(2), pp. 137–142.

    Article  MathSciNet  Google Scholar 

  • Schwartz, D.G. (1993) Cooperating heterogeneous systems: A blackboard-based meta approach, Technical Report 93-112, Center for Automation and Intelligent Systems Research, Case Western Reserve University, Cleveland Ohio, (unpublished PhD. thesis).

    Google Scholar 

  • Sullivan, J. and Tyler, S. (eds.) (1991) Intelligent User Interfaces, Reading: Addison-Wesley Pub. Co.

    MATH  Google Scholar 

  • Warren, D. and Pereira, F. (1982) An Efficient Easily Adaptable System for Interpreting Natural Language Queries, American Journal of Computational Linguistics, 8(3), pp. 110–123.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. SRI International, 333 Ravenswood Ave, 94025, Menlo Park, CA, USA

    Adam Cheyer & Luc Julia

Authors
  1. Adam Cheyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luc Julia
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Harry Bunt Robbert-Jan Beun Tijn Borghuis

Rights and permissions

Reprints and permissions

Copyright information

© 1998 Springer-Verlag

About this paper

Cite this paper

Cheyer, A., Julia, L. (1998). Multimodal maps: An agent-based approach. In: Bunt, H., Beun, RJ., Borghuis, T. (eds) Multimodal Human-Computer Communication. CMC 1995. Lecture Notes in Computer Science, vol 1374. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0052316

Download citation

  • DOI: https://doi.org/10.1007/BFb0052316

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-64380-7

  • Online ISBN: 978-3-540-69764-0

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics