Skip to main content
SpringerLink
Log in

A Pattern-Based Methodology for Multimodal Interaction Design

  • Conference paper
Text, Speech and Dialogue (TSD 2006)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 4188))

Included in the following conference series:

Abstract

This paper describes a design methodology for multimodal interactive systems. The method suggested is meant to serve as a foundation for the application of robust software engineering techniques in the field of multimodal systems. Starting from a short review of current design approaches we present a high level view of the design process for multimodal systems, highlighting design issues related to context of use factors. Our proposal is discussed in the context of a multimodal organizer which serves as our showcase application. The design of multimodal systems brings together a broad variety of analysis methods (task, context, data, user). The combination of modalities as well as the different interaction devices imply a high degree of freedom as far as design decisions are concerned. Therefore, a (simple) unification of existing approaches towards interface design like GOMS (task analysis) or Buergy’s interaction constraint model for context analysis is not sufficient. We employ the design pattern approach as a means of guiding the analysis and design process. Design patterns are discussed as a general modeling tool as well as a possible approach towards designing multimodal systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alexander, C., Ishikawa, S., Silverstein, M., Jacobson, M., Fiksdahl-King, I., Angel, S.: A Pattern Language. Oxford University Press, Oxford (1977)

    Google Scholar 

  2. Benoît, et al.: Audio-visual and Multimodal Speech Systems. Audio-visual and Multimodal Speech Systems. In: Handbook of Standards and Resources for Spoken Language Systems – Supplement Volume (2000)

    Google Scholar 

  3. Bernsen, N.O.: Modality Theory: Supporting Multimodal Interface Design. In: Proc. ERCIM (1993)

    Google Scholar 

  4. Bernsen, N.O.: A toolbox of output modalities. Representing output information in multimodal interfaces. WPCS-95-10. Centre for Cognitive Science, Roskilde University (1995)

    Google Scholar 

  5. Bernsen, N.O.: Towards a tool for predicting speech functionality. Speech Communication 23, 181–210 (1997)

    Article  Google Scholar 

  6. Bernsen, N.O.: Multimodality in language and speech systems – from theory to design support tool. In: Granström, B., House, D., Karlsson, I. (eds.) Multimodality in Language and Speech Systems, pp. 93–148. Kluwer Academic Publishers, Dordrecht (2002)

    Google Scholar 

  7. Borchers, J.O.: A Pattern Approach to Interaction Design. AI & Society Journal of Human-Centred Systems and Machine Intelligence 15(4), 359–376 (2001)

    MathSciNet  Google Scholar 

  8. Bürgy, C.: An Interaction Constraints Model for Mobile and Wearable Computer-Aided Engineering Systems in Industrial Applications. Doctoral Dissertation, University of Pittsburgh, Pennsylvania, USA (2002)

    Google Scholar 

  9. Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Bouillon, L., Vanderdonckt, J.: A unifying reference framework for multi-target user interfaces. Interacting with Computers 15(3), 289–308 (2003)

    Article  Google Scholar 

  10. Cenek, P., Melichar, M., Rajman, M.: A Framework for Rapid Multimodal Application Design. In: Matoušek, V., Mautner, P., Pavelka, T. (eds.) TSD 2005. LNCS (LNAI), vol. 3658, pp. 393–403. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  11. Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., Young, R.M.: Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The CARE Properties. In: Proc. Interact 1995, pp. 115–120. Chapman and Hall, London (1995)

    Google Scholar 

  12. Dittmar, A.: More Precise Descriptions of Temporal Relations within Task Models. In: Palanque, P., Paternó, F. (eds.) DSV-IS 2000. LNCS, vol. 1946, pp. 151–168. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  13. Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, Reading (1995)

    Google Scholar 

  14. Grasso, M.A., Ebert, D.S., Finin, T.W.: The integrality of speech in multimodal interfaces. ACM Trans. Comput.-Hum. Interact. 5(4), 303–325 (1998)

    Article  Google Scholar 

  15. Horrocks, I.: Constructing the User Interface with Statecharts. Addison-Wesley, Reading (1999)

    Google Scholar 

  16. Kiger, J.I.: The depth/breadth trade-off in the design of menu-driven user interfaces. International Journal of Man-Machine Studies 20, 201–213 (1984)

    Article  Google Scholar 

  17. Landauer, T.K., Nachbar, D.W.: Selection from alphabetic and numeric menu trees using a touch screen: Breath, depth, and width. In: Proc. CHI 1985, Human Factors in Computing Systems, pp. 73–78. ACM, New York (1985)

    Chapter  Google Scholar 

  18. Larsson, S.: Using a type hierarchy to characterize reliability of coding schemas for dialogue moves. Gothenburg Papers in Computational Linguistics (1998)

    Google Scholar 

  19. Martin, J.-C.: Towards “intelligent” cooperation between modalities. The example of a system enabling multimodal interaction with a map. In: Proc. IJCAI 1997 Workshop on Intelligent Multimodal Systems, Nagoya, Japan (1997)

    Google Scholar 

  20. Nigay, L., Coutaz, J.: A Design Space For Multimodal Systems: Concurrent Processing and Date Fusion. In: Proc. INTERCHI 1993. ACM Press, New York (1993)

    Google Scholar 

  21. Oviatt, S., Cohen, P., Wu, L., Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J., Ferro, D.: Designing the User Interface for Multimodal Speech and Pen-based Gesture Applications: State-of-the-Art Systems and Future Research Directions. Human Computer Interaction 15(4), 263–322 (2000)

    Article  Google Scholar 

  22. Pribeanu, C., Limbourg, Q., Vanderdonckt, J.: Task Modelling for Context-Sensitive User Interfaces. In: Johnson, C. (ed.) DSV-IS 2001. LNCS, vol. 2220, pp. 49–68. Springer, Heidelberg (2001)

    Google Scholar 

  23. Rajman, M., Bui, T.H., Rajman, A., Seydoux, F., Trutnev, A., Qurteroni, S.: Assessing the usability of a dialogue management system designed in the framework of a rapid dialogue prototyping methodology. Acta Acustica united with Acustica, the journal of the European Acoustics Association (EAA) 90, 1096–1111 (2004)

    Google Scholar 

  24. Ratzka, A.: Combining Modality Theory and Context Models. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Weber, M. (eds.) PIT 2006. LNCS, vol. 4021, pp. 141–151. Springer, Heidelberg (to appear, 2006)

    Chapter  Google Scholar 

  25. Reeves, L.M., Lai, J., Larson, J.A., Oviatt, S., Balaji, T.S., Buisine, S., Collings, P., Cohen, P., Kraal, B., Martin, J.-C., McTear, M., Raman, T., Stanney, K.M., Su, H., Wang, Q.Y.: Guidelines for multimodal user interface design. Communications ACM 47(1), 57–59 (2004)

    Article  Google Scholar 

  26. Shneiderman, B.: Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison-Wesley, Reading (1997)

    Google Scholar 

  27. da Silva, P.P.: User Interface Declarative Models and Development Environments: A Survey. In: Palanque, P., Paternó, F. (eds.) DSV-IS 2000. LNCS, vol. 1946, pp. 207–226. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  28. Souchon, N., Limbourg, Q., Vanderdonckt, J.: Task Modelling in Multiple Contexts of Use. In: Forbrig, P., Limbourg, Q., Urban, B., Vanderdonckt, J. (eds.) DSV-IS 2002. LNCS, vol. 2545, pp. 59–73. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  29. Trætteberg, H.: Model-based User Interface Design. Ph.D. Thesis, Department of Computer and Information Science, Norwegian University of Science and Technology (2002)

    Google Scholar 

  30. Trowbridge, D., Cunningham, W., et al.: Describing the Enterprise Architectural Space. Microsoft Corporation (June 2004)

    Google Scholar 

  31. Vernier, F., Nigay, L.: A Framework for the Combination and Characterization of Output Modalities. In: Palanque, P., Paternó, F. (eds.) DSV-IS 2000. LNCS, vol. 1946, pp. 35–50. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  32. Van Welie, M., van der Weer, G.C.: Pattern Languages in Interaction Design: Structure and Organization. In: Proc. Interact 2003 (2003)

    Google Scholar 

  33. Wilson, S., Johnson, P., Kelly, C., Cunningham, J., Markopoulos, P.: Beyond Hacking: a Model Based Approach to User Interface Design. In: Proc. HCI 1993 (1993)

    Google Scholar 

  34. Wolff, C.: Media Design Patterns. In: Eibl, M., Womser-Hacker, C., Wolff, C. (eds.) Designing Information Systems. pp. 209–217. UVK, Constance (2005)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute for Media, Information and Cultural Studies, University of Regensburg, D-93040, Regensburg, Germany

    Andreas Ratzka & Christian Wolff

Authors
  1. Andreas Ratzka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christian Wolff
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Faculty of Informatics, Masaryk University, Brno, Czech Republic

    Petr Sojka

  2. Faculty of Informatics, Masaryk University, Botanická 68a, CZ-602 00, Brno, Czech Republic

    Ivan Kopeček

  3. Faculty of Informatics, Department of Computer Graphics and Design, Masaryk University, Botanická 68a, 60200, Brno, Czech Republic

    Karel Pala

Rights and permissions

Reprints and permissions

Copyright information

© 2006 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Ratzka, A., Wolff, C. (2006). A Pattern-Based Methodology for Multimodal Interaction Design. In: Sojka, P., Kopeček, I., Pala, K. (eds) Text, Speech and Dialogue. TSD 2006. Lecture Notes in Computer Science(), vol 4188. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11846406_85

Download citation

  • DOI: https://doi.org/10.1007/11846406_85

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-39090-9

  • Online ISBN: 978-3-540-39091-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation