skip to main content
10.1145/1452392.1452416acmconferencesArticle/Chapter ViewAbstractPublication Pagesicmi-mlmiConference Proceedingsconference-collections
poster

Explorative studies on multimodal interaction in a PDA- and desktop-based scenario

Published: 20 October 2008 Publication History

Abstract

This paper presents two explorative case studies on multimodal interaction. Goal of this work is to find and underpin design recommendations to provide well proven decision support across all phases of the usability engineering lifecycle [1]. During this work, user interface patterns for multimodal interaction were identified [2, 3]. These patterns are closely related to other user interface patterns [4, 5, 6]. Two empirical case studies, one using a Wizard of Oz setting and another one using a stand-alone prototype linked to a speech recognition engine [7] were conducted to assess the acceptance of resulting interaction styles. Although the prototypes applied as well those interface patterns that increase usability by means of traditional interaction techniques and thus compete with multimodal interaction styles, multimodal interaction was preferred by most of the users.

References

[1]
Mayhew, D.J.: The Usability Engineering Lifecycle. Morgan Kaufmann, San Francisco (1999)
[2]
Ratzka, A., Wolff, C.: A pattern-based methodology for multimiodal interaction design. In Sojka, P., KopeÇcek, I., Pala, K., eds.: Proc. of Text, Speech, and Dialogue, TSD'06. LNAI 4188, Berlin, Heidelberg, Springer (2006) 677--686
[3]
Ratzka, A.: Design patterns in the context of multi-modal interaction. In: To appear in: Proceedings of the 6th Nordic Conference on Pattern Languages of Programs 2007 VikingPLoP 2007. (2008)
[4]
Tidwell, J.: Common ground: A pattern language for human-computer interface design. (1999)
[5]
Tidwell, J.: Designing Interfaces: Patterns for Effective Interaction Design. O'Reilly (2005)
[6]
van Welie, M., Trætteberg, H.: Interaction patterns in user interfaces. In: Proceedings of the Seventh Pattern Languages of Programs Conference. (2000)
[7]
Ratzka, A.: A wizard-of-oz setting for multimodal interaction. an approach to user-based elicitation of design patterns. In Osswald, A., Stempfhuber, M., Wolff, C., eds.: Open Innovation. Proc. 10th International Symposium for Information Science, Konstanz, Universit¨atsverlag Konstanz (2007) 159--170
[8]
Niedermaier, F.B.: Entwicklung und Bewertung eines Rapid-Prototyping Ansatzes zur multimodalen Mensch-Maschine-Interaktion im Kraftfahrzeug. PhD thesis, Fakultat fur Elektrotechnik und Informationstechnik der Technischen Universitat Munchen (2003)
[9]
DragiÇcevic, P.: Un modele d'interaction en entree pour des systemes interactifs multi-dispositifs hautement configurables. PhD thesis, Universite de Nantes, ecole doctorale sciences et technologies de l'information et des materieaux (Mars 2004)
[10]
Bernsen, N.O.: A toolbox of output modalities: Representing output information in multimodal interfaces. (1997)
[11]
Bernsen, N.O.: Multimodality in language and speech systems -- from theory to design support tool. In Granstrom, B., ed.: Multimodality in Language and Speech Systems, Dordrecht, Kluwer (2001)
[12]
Seifert, K.: Evaluation multimodaler Computer-Systeme in fruhen Entwicklungsphasen. PhD thesis, Fakultat V -- Verkehrs- und Maschinensysteme, Technische Universitat Berlin (2002)
[13]
Alexander, C.: The Timeless Way of Building. Oxford University Press (1979)
[14]
Gabriel, D.: A pattern definition. http://hillside.net/patterns/definition.html (checked on 2007-05-09)
[15]
Alexander, C., Ishikawa, S., Silverstein, M., Jacobson, M., Fiksdahl-King, I., Angel, S.: A Pattern Language. Oxford University Press (1977)
[16]
Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley (1995)
[17]
Buschmann, F., Meunier, R., Rohnert, H., Sommerlad, P., Stal, M.: Pattern-orientierte Softwarearchitektur. Addison--Wesley, Bonn (1998)
[18]
Borchers, J.O.: A pattern approach to interaction design. AI & Society Journal of Human-Centered Systems and Machine Intelligence 15(4) (2001) 359--376
[19]
Duyne, D.K.V., Landay, J., Hong, J.I.: The Design of Sites: Patterns, Principles, and Processes for Crafting a Customer-Centered Web Experience. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA (2002)
[20]
van Welie, M.: Gui design patterns. In: www.welie.com -- ..patterns in Interaction Design. (2003)
[21]
Hauptmann, A.G.: Speech and gestures for graphic image manipulation. In: CHI '89: Proceedings of the SIGCHI conference on Human factors in computing systems, New York, NY, USA, ACM (1989) 241--245
[22]
Oviatt, S., Olsen, E.: Integration themes in multimodal human-computer interaction. In Shirai, K., Furui, S., Kakehi, K., eds.: Proceedings of the International Conference on Spoken Language Processing. Volume 2., Yokohama, Japan, Acoustical Society of Japan (1994)
[23]
Oviatt, S., Cohen, P.R., Wang, M.Q.: Toward interface design for human language technology: modality and structure as determinants of linguistic complexity. Speech Commun. 15(3-4) (1994) 283--300
[24]
Angeli, A.D., Gerbino, W., Cassano, G., Petrelli, D.: Visual display, pointing, and natural language: the power of multimodal interaction. In: AVI '98: Proceedings of the working conference on Advanced visual interfaces, New York, NY, USA, ACM Press (1998) 164--173
[25]
Oviatt, S.: Multimodal interfaces for dynamic interactive maps. In: CHI '96: Proceedings of the SIGCHI conference on Human factors in computing systems, New York, NY, USA, ACM (1996) 95--102
[26]
Cohen, P., Johnston, M., McGee, D., Oviatt, S.L., Clow, J., Smith, I.: The efficiency of multimodal interaction: A case study. In: ICSLP'98. (1998)
[27]
Cohen, P., McGee, D., Clow, J.: The efficiency of multimodal interaction for a map-based task. In: Proceedings of the sixth conference on Applied natural language processing, San Francisco, CA, USA, Morgan Kaufmann Publishers Inc. (2000) 331--338
[28]
Oviatt, S., Kuhn, K.: Referential features and linguistic indirection in multimodal language. In: Proceedings of the International Conference on Spoken Language Processing. Volume 6., ASSTA (1998) 2339--2342
[29]
Oviatt, S.: Ten myths of multimodal interaction. Commun. ACM 42(11) (1999) 74--81
[30]
Oviatt, S.: Mutual disambiguation of recognition errors in a multimodal architecture. In: CHI '99: Proceedings of the SIGCHI conference on Human factors in computing systems, New York, NY, USA, ACM (1999) 576--583
[31]
Oviatt, S.: Multimodal system processing in mobile environments. In: UIST '00: Proceedings of the 13th annual ACM symposium on User interface software and technology, New York, NY, USA, ACM Press (2000) 21--30
[32]
Oviatt, S.L.: Multimodal signal processing in naturalistic noisy environments. In Yuan, B., Huang, T., Tang, X., eds.: Proceedings of the 6th International Conference on SPoken Language Processing (ICSLP). Volume 2., Peking, Chinese Friendship Publishers (2000) 696--699
[33]
Oviatt, S.: Taming recognition errors with a multimodal interface. Commun. ACM 43(9) (2000) 45--51
[34]
Kumar, S., Cohen, P.R., Coulston, R.: Multimodal interaction under exerted conditions in a natural field setting. In: ICMI '04: Proceedings of the 6th international conference on Multimodal interfaces, New York, NY, USA, ACM Press (2004) 227--234
[35]
Nishimoto, T., Shida, N., Kobayashi, T., Shirai, K.: Improving human interface in drawing tool using speech. In: Proceedings of 4th IEEE International Workshop on Robot and Human Communication, ROMAN'95. (1995) 107--112
[36]
Huang, X., Acero, A., Chelba, C., Deng, L., Duchene, D., Goodman, J., Hon, H., Jacoby, D., Jiang, L., Loynd, R., Mahajan, M., Mau, P., Meredith, S., Mughal, S., Neto, S., Plumpe, M., Wang, K., Wang, Y.: Mipad: A next generation pda prototype. In: ICSLP, Peking (2000)
[37]
Lai, J.: Facilitating mobile communication with multimodal access to email messages on a cell phone. In: CHI '04: CHI '04 extended abstracts on Human factors in computing systems, New York, NY, USA, ACM Press (2004) 1259--1262
[38]
Jost, M., Haußler, J., Merdes, M., Malaka, R.: Multimodal interaction for pedestrians: an evaluation study. In: IUI '05: Proceedings of the 10th international conference on Intelligent user interfaces, New York, NY, USA, ACM Press (2005) 59--66

Cited By

View all
  • (2017)Interaction History in Adaptive Multimodal InteractionCompanion Technology10.1007/978-3-319-43665-4_12(231-251)Online publication date: 5-Dec-2017
  • (2016)Measuring effects of user-specific behaviour on selection tasks in HCI2016 SAI Computing Conference (SAI)10.1109/SAI.2016.7556010(380-387)Online publication date: Jul-2016
  • (2013)Are Mobile Spreadsheet Applications Usable?Journal of Organizational and End User Computing10.4018/joeuc.201301010325:1(43-63)Online publication date: Jan-2013
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICMI '08: Proceedings of the 10th international conference on Multimodal interfaces
October 2008
322 pages
ISBN:9781605581989
DOI:10.1145/1452392
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2008

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. mobile computing
  2. multimodality
  3. user interface patterns

Qualifiers

  • Poster

Conference

ICMI '08
Sponsor:
ICMI '08: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERFACES
October 20 - 22, 2008
Crete, Chania, Greece

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)4
  • Downloads (Last 6 weeks)0
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2017)Interaction History in Adaptive Multimodal InteractionCompanion Technology10.1007/978-3-319-43665-4_12(231-251)Online publication date: 5-Dec-2017
  • (2016)Measuring effects of user-specific behaviour on selection tasks in HCI2016 SAI Computing Conference (SAI)10.1109/SAI.2016.7556010(380-387)Online publication date: Jul-2016
  • (2013)Are Mobile Spreadsheet Applications Usable?Journal of Organizational and End User Computing10.4018/joeuc.201301010325:1(43-63)Online publication date: Jan-2013
  • (2013)User Interface Patterns for Multimodal InteractionTransactions on Pattern Languages of Programming III10.1007/978-3-642-38676-3_4(111-167)Online publication date: 2013
  • (2012)Influencing factors on multimodal interaction during selection tasksJournal on Multimodal User Interfaces10.1007/s12193-012-0117-57:4(299-310)Online publication date: 25-Nov-2012

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media