skip to main content
10.1145/2493190.2493227acmconferencesArticle/Chapter ViewAbstractPublication PagesmobilehciConference Proceedingsconference-collections
research-article

Design guidelines for adaptive multimodal mobile input solutions

Published: 27 August 2013 Publication History

Abstract

The advent of advanced mobile devices in combination with new interaction modalities and methods for the tracking of contextual information, opens new possibilities in the field of context-aware user interface adaptation. One particular research direction is the automatic context-aware adaptation of input modalities in multimodal mobile interfaces. We present existing adaptive multimodal mobile input solutions and position them within closely related research fields. Based on a detailed analysis of the state of the art, we propose eight design guidelines for adaptive multimodal mobile input solutions. The use of these guidelines is further illustrated through the design and development of an adaptive multimodal calendar application.

References

[1]
Arhippainen, L., Rantakokko, T., and Tähti, M. Navigation with an Adaptive Mobile Map-Application: User Experiences of Gesture- and Context-Sensitiveness. In Proc. of UCS 2004 (Tokyo, Japan, November 2004), 62--73.
[2]
Bernsen, N. O., and Dybkjær, L. Multimodal Usability. Springer, 2009.
[3]
Bühler, D., Minker, W., Häussler, J., and Krüger, S. The SmartKom Mobile Multi-Modal Dialogue System. In Proc. of ITRW 2002 (Irsee, Germany, June 2002).
[4]
Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Bouillon, L., and Vanderdonckt, J. A Unifying Reference Framework for Multi-Target User Interfaces. Interacting with Computers 15, 3 (2003), 289--308.
[5]
Cesar, P., Vaishnavi, I., Kernchen, R., Meissner, S., Hesselman, C., Boussard, M., Spedalieri, A., Bulterman, D. C., and Gao, B. Multimedia Adaptation in Ubiquitous Environments: Benefits of Structured Multimedia Documents. In Proc. of DocEng 2008 (São Paulo, Brazil, September 2008), 275--284.
[6]
Coutaz, J., Nigay, L., Salber, D., Blandford, A., May, J., and Young, R. M. Four Easy Pieces for Assessing the Usability of Multimodal Interaction: The CARE Properties. In Proc. of INTERACT 1995 (Lillehammer, Norway, June 1995), 115--120.
[7]
David, L., Endler, M., Barbosa, S. D. J., and Filho, J. V. Middleware Support for Context-Aware Mobile Applications with Adaptive Multimodal User Interfaces. In Proc. of U-Media 2011 (São Paulo, Brazil, July 2011), 106--111.
[8]
de Sá, M., and Carricç, L. Lessons From Early Stages Design of Mobile Applications. In Proc. of MobileHCI 2008 (Amsterdam, The Netherlands, 2008), 127--136.
[9]
Dey, A. K., and Häkkilä, J. Context-Awareness and Mobile Devices. User Interface Design and Evaluation for Mobile Technology 1 (2008), 205--217.
[10]
Doyle, J., Bertolotto, M., and Wilson, D. A Survey of Multimodal Interfaces for Mobile Mapping Applications. In Map-based Mobile Services, Lecture Notes in Geoinformation and Cartography. Springer Berlin Heidelberg, 2008, 146--167.
[11]
Dumas, B., Lalanne, D., and Oviatt, S. Multimodal Interfaces: A Survey of Principles, Models and Frameworks. Human Machine Interaction: Research Results of the MMI Program (March 2009), 3--26.
[12]
Gentile, A., Santangelo, A., Sorce, S., Augello, A., Pilato, G., Genco, A., and Gaglio, S. Exploiting Multimodality for Intelligent Mobile Access to Pervasive Services in Cultural Heritage Sites. Multimodality in Mobile Computing and Mobile Devices: Methods for Adaptable Usability (2009), 217--240.
[13]
Kong, J., Zhang, W. Y., Yu, N., and Xia, X. J. Design of Human-Centric Adaptive Multimodal Interfaces. International Journal of Human-Computer Studies 69, 12 (December 2011), 854--869.
[14]
Lemmelä, S., Vetek, A., Mäkelä, K., and Trendafilov, D. Designing and Evaluating Multimodal Interaction for Mobile Contexts. In Proc. of ICMI 2008 (Chania, Greece, October 2008), 265--272.
[15]
López-Jaquero, V., Vanderdonckt, J., Montero, F., and González, P. Towards an Extended Model of User Interface Adaptation: The Isatine Framework. In Engineering Interactive Systems, vol. 4940 of LNCS. Springer, 2008, 374--392.
[16]
Malinowski, U., Thomas, K., Dieterich, H., and Schneider-Hufschmidt, M. A Taxonomy of Adaptive User Interfaces. In Proc. of HCI 1992 (York, UK, September 1992), 391--414.
[17]
Oviatt, S. Multimodal Interactive Maps: Designing for Human Performance. Human-Computer Interaction 12, 1 (March 1997), 93--129.
[18]
Oviatt, S., and Lunsford, R. Multimodal Interfaces for Cell Phones and Mobile Technology. International Journal of Speech Technology 8 (2005), 127--132.
[19]
Porta, D., Sonntag, D., and Nesselrath, R. New Business to Business Interaction: Shake Your iPhone and Speak to It. In Proc. of MobileHCI 2009 (Bonn, Germany, September 2009).
[20]
Ramsay, A., McGee-Lennon, M., Wilson, G., Gray, S., Gray, P., and De Turenne, F. Tilt and Go: Exploring Multimodal Mobile Maps in the Field. Journal on Multimodal User Interfaces 3 (2010).
[21]
Reeves, L. M., Lai, J., Larson, J. A., Oviatt, S., Balaji, T. S., Buisine, S., Collings, P., Cohen, P., Kraal, B., Martin, J.-C., McTear, M., Raman, T., Stanney, K. M., Su, H., and Wang, Q. Y. Guidelines for Multimodal User Interface Design. Communications of the ACM 47, 1 (January 2004), 57--59.
[22]
Reis, T., de Sá, M., and Carriço, L. Multimodal Interaction: Real Context Studies on Mobile Digital Artefacts. In Haptic and Audio Interaction Design, vol. 5270. Springer, 2008, 60--69.
[23]
Reithinger, N., Alexandersson, J., Becker, T., Blocher, A., Engel, R., Löckelt, M., Müller, J., Pfleger, N., Poller, P., Streit, M., and Tschernomas, V. SmartKom: Adaptive and Flexible Multimodal Access to Multiple Applications. In Proc. of ICMI 2003 (Vancouver, Canada, 2003), 101--108.
[24]
Ronkainen, S., Koskinen, E., Liu, Y., and Korhonen, P. Environment Analysis as a Basis for Designing Multimodal and Multidevice User Interfaces. Human-Computer Interaction 25, 2 (2010), 148--193.
[25]
Rothrock, L., Koubek, R., Fuchs, F., Haas, M., and Salvendy, G. Review and Reappraisal of Adaptive Interfaces: Toward Biologically Inspired Paradigms. Theoretical Issues in Ergonomics Science 3, 1 (2002), 47--84.
[26]
Salvaneschi, G., Ghezzi, C., and Pradella, M. Context-Oriented Programming: A Software Engineering Perspective. Journal of Systems and Software 85, 8 (August 2012), 1801--1817.
[27]
Schiefer, G., and Decker, M. Taxonomy for Mobile Terminals - A Selective Classification Scheme. In Proc. of ICE-B 2008 (Porto, Portugal, July 2008), 255--258.
[28]
Sonntag, D., Engel, R., Herzog, G., Pfalzgraf, A., Pfleger, N., Romanelli, M., and Reithinger, N. Smart Web Handheld - Multimodal Interaction with Ontological Knowledge Bases and Semantic Web Services. In Artifical Intelligence for Human Computing, vol. 4451 of Lecture Notes in Computer Science. Springer, 2007, 272--295.
[29]
Turunen, M., Kallinen, A., Sànchez, I., Riekki, J., Hella, J., Olsson, T., Melto, A., Rajaniemi, J.-P., Hakulinen, J., Mäkinen, E., Valkama, P., Miettinen, T., Pyykkönen, M., Saloranta, T., Gilman, E., and Raisamo, R. Multimodal Interaction with Speech and Physical Touch Interface in a Media Center Application. In Proc. of ACE 2009 (Athens, Greece, October 2009), 19--26.
[30]
Wasinger, R., Krüger, A., and Jacobs, O. Integrating Intra and Extra Gestures into a Mobile and Multimodal Shopping Assistant. In Pervasive Computing, vol. 3468 of Lecture Notes in Computer Science. Springer, 2005, 323--328.
[31]
Zaguia, A., Hina, M., Tadj, C., and Ramdane-Cherif, A. Interaction Context-aware Modalities and Multimodal Fusion for Accessing Web Services. Ubiquitous Computing and Communication Journal 5, 4 (December 2010).

Cited By

View all
  • (2024)Envisioning Support-Centered Technologies for Language Practice and Use: Needs and Design Opportunities for Immigrant English Language Learners (ELLs)Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642236(1-15)Online publication date: 11-May-2024
  • (2022)Assessing the Influence of Multimodal Feedback in Mobile-Based Musical Task PerformanceMultimodal Technologies and Interaction10.3390/mti60800686:8(68)Online publication date: 8-Aug-2022
  • (2020)Integrating Mobile Multimodal Interactions based on Programming By Demonstration​International Journal of Human–Computer Interaction10.1080/10447318.2020.182368837:5(418-433)Online publication date: 30-Oct-2020
  • Show More Cited By

Index Terms

  1. Design guidelines for adaptive multimodal mobile input solutions

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MobileHCI '13: Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services
    August 2013
    662 pages
    ISBN:9781450322737
    DOI:10.1145/2493190
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 27 August 2013

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. context-aware systems
    2. design guidelines
    3. mobile multimodal interaction
    4. user interface adaptation

    Qualifiers

    • Research-article

    Conference

    MobileHCI '13
    Sponsor:

    Acceptance Rates

    MobileHCI '13 Paper Acceptance Rate 53 of 238 submissions, 22%;
    Overall Acceptance Rate 202 of 906 submissions, 22%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)7
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Envisioning Support-Centered Technologies for Language Practice and Use: Needs and Design Opportunities for Immigrant English Language Learners (ELLs)Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642236(1-15)Online publication date: 11-May-2024
    • (2022)Assessing the Influence of Multimodal Feedback in Mobile-Based Musical Task PerformanceMultimodal Technologies and Interaction10.3390/mti60800686:8(68)Online publication date: 8-Aug-2022
    • (2020)Integrating Mobile Multimodal Interactions based on Programming By Demonstration​International Journal of Human–Computer Interaction10.1080/10447318.2020.182368837:5(418-433)Online publication date: 30-Oct-2020
    • (2020)Trade-offs in the design of multimodal interaction for older adultsBehaviour & Information Technology10.1080/0144929X.2020.185176841:5(1035-1051)Online publication date: 3-Dec-2020
    • (2019)Exploring a Design Space of Graphical Adaptive MenusACM Transactions on Interactive Intelligent Systems10.1145/323719010:1(1-40)Online publication date: 29-Jul-2019
    • (2016)A comparative exploratory study on mobile Web adaptation in front-end frameworks from the perspective of end-usersProceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems10.1145/3033701.3033731(1-10)Online publication date: 4-Oct-2016
    • (2015)The Paradigm Shift to Multimodality in Contemporary Computer InterfacesSynthesis Lectures on Human-Centered Informatics10.2200/S00636ED1V01Y201503HCI0308:3(1-243)Online publication date: 13-Apr-2015
    • (2015)Lightweight Adaptation of Classifiers to Users and Contexts: Trends of the Emerging DomainThe Scientific World Journal10.1155/2015/4348262015:1Online publication date: 10-Sep-2015
    • (2015)Authoring context-dependent cross-device user interfaces based on trigger/action rulesProceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia10.1145/2836041.2836073(313-322)Online publication date: 30-Nov-2015
    • (2015)Supporting user interaction of social network mobile application with multimodal interaction2015 10th Asia-Pacific Symposium on Information and Telecommunication Technologies (APSITT)10.1109/APSITT.2015.7217086(1-3)Online publication date: Aug-2015
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media