Skip to main content
SpringerLink
Log in

Towards the design of usable multimodal interaction languages

  • Special issue on multimodality: a step towards universal access
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

This paper presents novel recommendations for the design of usable multimodal command or query languages. These recommendations have been inferred from the results of three empirical studies focused on the use of spontaneous speech (first study) and the synergic use of spontaneous versus controlled speech and gestures for interacting with current application software (second and third studies). In particular, we propose a method for designing multimodal languages that can be considered as an appropriate substitute for direct manipulation in all contexts precluding the use of mouse and keyboard, and for all standard categories of users, especially the general public.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Amalberti R, Carbonell N, Falzon P (1993) User representations of computer systems in human-computer speech interaction. Int J Man-Mach Stud 38:547–566

    Google Scholar 

  2. Bernsen N-O (1994) Foundations of multimodal representations, a taxonomy of representational modalities. Interact Comput 6:347–371

    Google Scholar 

  3. Borenstein N (1986) Is English a natural language. In: Hopper KT, Newman IA (eds) Foundations for Human-Computer Communication. North Holland, Amsterdam, pp 58–72

  4. Carbonell N, Mignot C (1994) Natural multimodal HCI: experimental results on the use of spontaneous speech and hand gestures. Proc 2nd ERCIM Workshop on Multimodal HCI. ERCIM, Paris, pp 97–112

  5. Carbonell N (1999) Multimodality: a primary requisite for achieving an information society for all. In: Bullinger H-J, Ziegler J (eds) Proc HCI International ’99, Munich, August 22–26 1999. Lawrence Erlbaum Associates, Mahwah (NJ), London, vol 2, pp 898–902

  6. Carbonell N, Dauchy P (1999) Empirical data on the use of speech and gestures in a multimodal human-computer environment. In: Bullinger H-J, Ziegler J (eds) Proc HCI International’99, Munich, August 22–26 1999. Lawrence Erlbaum Associates, Mahwah (NJ), London, vol 1, pp 446–450

  7. Carroll JM, Smith-Kerker PL, Ford JR, Mazur-Rimetz SA (1987) The minimal manual. Hum-Comput Interact 3(2):123–153

  8. Cohen PR, Johnston M, McGee D, Oviatt SL, Clow J, Smith I (1998) The efficiency of multimodal interaction: a case study. In: Mannell RH, Robert-Ribes J (eds) Proc 5th Int Conf Spoken Language Processing (ICSLP’98), Sydney, Australia, 30 November – 4 December, 1998. ASSTA, Sydney, Australia, vol 2, pp 249–252

  9. Coutaz J, Caelen J (1991) A taxonomy for multimedia and multimodal user interfaces. Proc 1st ERCIM Workshop on Multimodal HCI. INESC, Lisbon, pp 143–148

  10. Coutaz J, Nigay L, Salber D, Blanford A, May J, Young RM (1995) Four easy pieces for assessing the usability of multimodal interaction: the CARE properties. In: Nordby K, Helmersen P, Gilmore D, Amesen S (eds) Proc 5th IFIP Int Conf Human-Computer Interaction – Bringing People Together (INTERACT’95), Lillehammer, Norway, 25–29 June, 1995. Chapman & Hall, London, pp 115–120

  11. Falzon P (1990) Human-computer interactions: lessons from human-human communication. In: Falzon P (ed) Cognitive Ergonomics: Understanding, Learning and Designing Human-Computer Interaction. Academic Press (EACE Publications), London San Diego, pp 51–65

  12. Falzon P (1991) Studying functional dialogues and operative languages. In: Queinnec Y, Daniellou F (eds) Proc XI Congress of the International Ergonomics Association (IEA), Paris, July 1991. Taylor and Francis, London, pp 543–545

  13. Harris G, Begg I, Upfold D (1980) On the role of the speaker’s expectations in interpersonal communication. J Verb Learn Verb Be 19:597–607

    Google Scholar 

  14. Johnston M (1998) Unification-based mmultimodal parsing. In: Proc 27th Int Conf Computational Linguistics and 36th Annual Meeting of the Association for Computational Linguistics (COLING’98), Montreal, Canada, 10–14 August, 1998. Morgan Kaufmann Publishers, Orlando, pp 624–630

  15. Kennedy A, Wilks A, Elder L, Murray WS (1988) Dialogues with machines. Cognition 30:73–105

    Google Scholar 

  16. Maybury MT (ed) (1993) Intelligent Multimedia Interfaces. AAAI/MIT Press, Menlo Park

  17. Morin P, Junqua J-C, Pierrel J-M (1992) A flexible multimodal dialogue architecture independent of the application. In: Ohala J, Neary TM, Derwing BL, Hodge MM, Wiebe GE (eds) Proc 2nd Int Conf Spoken Language Processing (ICSLP’92), Banff, AB, Canada, 12–16 October, 1992. University of Alberta, Edmonton, AB, Canada, vol 1, pp 939–942

  18. Nielsen J (1993) Usability engineering. Academic Press, Boston

  19. Nigay L, Coutaz J (1993) A design space for multimodal systems: concurrent processing and data fusion. In: Ashlund S, Mullet K, Henderson A, Hollnagel E, White T (eds) Proc Joint Int Conf Human Factors in Computing Systems (CHI’93 & INTERACT’93): INTERCHI’93, Amsterdam, 24–29 April, 1993. ACM Press, New York, pp 172–178

  20. Oviatt SL (2000) Multimodal system processing in mobile environments. Proc 13th Annual ACM Symposium on User Interface Software and Technology (UIST’2000), San Diego, CA, 5–8 November, 2000. ACM Press, New York, pp 21–30

  21. Oviatt S, Cohen P, Fong M, Frank M (1992) A rapid semi-automatic simulation technique for investigating interactive speech and handwriting. In: Ohala J, Neary TM, Derwing BL, Hodge MM, Wiebe GE (eds) Proc 2nd Int Conf Spoken Language Processing (ICSLP’92), Banff, AB, Canada, 12–16 October, 1992. University of Alberta, Edmonton, AB, Canada, vol 2, pp 1351–1354

  22. Robbe S, Carbonell N, Dauchy P (1996) How do users manipulate graphical icons? An empirical study. In: Harling P, Edwards A (eds) Progress in gestural interaction. Springer-Verlag, London, pp 173–186

  23. Robbe S, Carbonell N, Dauchy P (1997) Constrained versus spontaneous speech and gestures for interacting with computers: a comparative empirical study. In: Howard S, Hammond J, Lindgaard G (eds) Proc 6th IFIP Int Conf Human-Computer Interaction (INTERACT’97), Sydney, Australia, 14–18 July, 1997. Chapman and Hall, London, pp 445–452

  24. Robbe S, Carbonell N, Dauchy P (2000) Expression constraints in multimodal human-computer interaction. In: Lieberman H (ed) Proc2000 Int Conf Intelligent User Interfaces (IUI’2000), New Orleans, 9–12 January, 2000. ACM Press, New York, pp 225–229

  25. Robbe S, Carbonell N, Valot C (1997) Towards usable multimodal command languages: definition and ergonomic assessment of constraints on users’ spontaneous speech and gestures. In: Kokkinakis G, Fakotakis N, Dermatas E (eds) Proc European Conf Speech Communication and Technology (EUROSPEECH’97), Rhodes, Greece, September 1997. ESCA, Grenoble, pp 1655–1658

  26. Sears A, Plaisant C, Shneiderman B (1992) A new era for high-precision touchscreens. In: Hartson R, Hix D (eds) Advances in human-computer interaction. Ablex Publishers, Norwood, NJ, pp 1–33

  27. Shriver S, Rosenfeld R, Zhu X, Toth A, Rudnicki A, Flueckiger M (2001) Universalizing speech: Notes from the USI project. In: Dalsgaard P, Lindberg B, Benner H (eds) Proc 7th European Conf Speech Communication and Technology (EUROSPEECH’01), Aalborg, Denmark, 3–7 September, 2001. ISCA, Bonn, vol 1, pp 1563–1566

  28. Sutcliffe AG, Old AC (1987) Do users know they have user models? Some experiences in the practice of user modelling. In: Bullinger H-J, Shackel B (eds) Proc 2nd IFIP Int Conf Human-Computer Interaction (INTERACT’87), Stuttgart, Germany, 1–4 September. North Holland, Amsterdam, pp 36–41

Download references

Author information

Authors and Affiliations

  1. LORIA, UMR 7503, (CNRS, INRIA, Universités de Nancy), Campus Scientifique, BP 239, F54506, Vandoeuvre-lès-Nancy Cedex, France

    Noëlle Carbonell

Authors
  1. Noëlle Carbonell
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Noëlle Carbonell.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Carbonell, N. Towards the design of usable multimodal interaction languages. UAIS 2, 143–159 (2003). https://doi.org/10.1007/s10209-003-0051-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-003-0051-0

Keywords

Search

Navigation