Skip to main content
SpringerLink
Log in

Influencing factors on multimodal interaction during selection tasks

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

When developing multimodal interactive systems it is not clear which importance should be given to which modality. In order to study influencing factors on multimodal interaction, we conducted a Wizard of Oz study on a basic recurrent task: 53 subjects performed diverse selections of objects on a screen. The way and modality of interaction was not specified nor predefined by the system, and the users were free in how and what to select. Natural input modalities like speech, gestures, touch, and arbitrary multimodal combinations of these were recorded as dependent variables. As independent variables, subjects’ gender, personality traits, and affinity towards technical devices were surveyed, as well as the system’s varying presentation styles of the selection. Our statistical analyses reveal gender as a momentous influencing factor and point out the role of individuality for the way of interaction, while the influence of the system output seems to be quite limited. This knowledge about the prevalent task of selection will be useful for designing effective and efficient multimodal interactive systems across a wide range of applications and domains.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Bellik Y, Rebaï I, Machrouh E, Barzaj Y, Jacquet C, Pruvost G, Sansonnet JP (2009) Multimodal interaction within ambient environments: an exploratory study. In: Proceedings of the 12th IFIP TC 13 international conference on human-computer interaction: part II, INTERACT ’09, Springer, Berlin, pp 89–92. doi: 10.1007/978-3-642-03658-3-13

  2. Chelazzi L (1999) Serial attention mechanisms in visual search: a critical look at the evidence. Psychol Res 62:195–219

    Google Scholar 

  3. Costa PT, McCrae RR (1992) Professional manual: revised NEO personality inventory (NEO-PI-R) and NEO five-factor inventory (NEO-FFI). Psychological Assessment Resources

  4. De Angeli A, Gerbino W, Cassano G, Petrelli D (1998) Visual display, pointing, and natural language: the power of multimodal interaction. In: Proceedings of the working conference on advanced visual interfaces, AVI ’98, ACM, New York, NY, USA, pp 164–173. doi: 10.1145/948496.948519

  5. Frank AU (1998) Formal models for cognition—taxonomy of spatial location description and frames of reference. In: Freksa C, Habel C, Wender KF (eds) Spatial cognition. Springer, Berlin, pp 293–312

  6. Gerhard U, Borkenau P, Ostendorf F (1993) NEO-Fünf-Faktoren inventar (NEO-FFI) nach Costa und McCrae. Zeitschrift für Klinische Psychologie und Psychotherapie 28(2):145–146. doi: 10.1026/0084-5345.28.2.145

  7. Huang X, Oviatt SL, Lunsford R (2006) Combining user modeling and machine learning to predict users’ multimodal integration patterns. In: Renals S, Bengio S, Fiscus JG (eds) Machine learning for multimodal interaction, 3rd international workshop, MLMI 2006, Bethesda, MD, USA, 1–4 May 2006, revised selected papers, lecture notes in computer science, vol 4299/2006, Springer, Berlin, pp 50–62. doi:10.1007/11965152-5

  8. Jöst M, Häußler J, Merdes M, Malaka R (2005) Multimodal interaction for pedestrians: an evaluation study. In: Proceedings of the 10th international conference on intelligent user interfaces, IUI ’05, ACM, New York, NY, USA, pp 59–66. doi:10.1145/1040830.1040852

  9. Karrer K, Glaser C, Clemens C, Bruder C (2009) Technikaffinität erfassen-der Fragebogen TA-EG. Der Mensch im Mittelpunkt technischer Systeme 8. Berliner Werkstatt Mensch-Maschine-Systeme 7. bis 9. Oktober 2009 (ZMMS Spektrum) 22(29):196–201

  10. Kieffer S, Carbonell N (2007) How really effective are multimodal hints in enhancing visual target spotting? some evidence from a usability study. CoRR abs/0708.3575. http://dblp.uni-trier.de/db/journals/corr/corr0708.html

  11. Kieffer S, Carbonell N (2007) Oral messages improve visual search. CoRR abs/0709.0428

  12. Kipp M (2001) Anvil—a generic annotation tool for multimodal dialogue. In: Proceedings of EUROSPEECH 2001, Aalborg, Denmark, pp 1367–1370

  13. Körner A, Drapeau M, Albani C, Geyer M, Schmutzer G, Brähler E (2008) Deutsche Normierung des NEO-Fünf-Faktoren-Inventars (NEO-FFI). Zeitschrift für Medizinische Psychologie 17(2–3):133–144

    Google Scholar 

  14. Mairesse F, Walker MA, Mehl MR, Moore RK (2007) Using linguistic cues for the automatic recognition of personality in conversation and text. J Artif Intell Res 30:457–501

    Google Scholar 

  15. Mignot C, Valot C, Carbonell N (1993) An experimental study of future “natural” multimodal human-computer interaction. In: INTERACT ’93 and CHI ’93 conference companion on Human factors in computing systems, CHI ’93, ACM, New York, NY, USA, pp 67–68. doi: 10.1145/259964.260075

  16. Oviatt S (1999) Ten myths of multimodal interaction. Commun ACM 42(11):74–81

    Article  Google Scholar 

  17. Oviatt S, Coulston R, Lunsford R (2004) When do we interact multimodally?: cognitive load and multimodal communication patterns. In: ICMI ’04, proceedings of the 6th international conference on multimodal interfaces, ACM, New York, NY, USA, pp 129–136. doi: 10.1145/1027933.1027957

  18. Ratzka A (2008) Explorative studies on multimodal interaction in a pda- and desktop-based scenario. In: Proceedings of the 10th international conference on multimodal interfaces, ICMI ’08, ACM, New York, NY, USA, pp 121–128. http://doi.acm.org/10.1145/1452392.1452416

  19. Reis T, de SM, Carrio L (2008) Multimodal interaction: real context studies on mobile digital artefacts. In: Pirhonen A, Brewster S (eds) Haptic and audio interaction design, lecture notes in computer science, vol 5270, Springer, Berlin, pp 60–69. doi: 10.1007/978-3-540-87883-4-7

  20. Ren X, Zhang G, Dai G (2000) An experimental study of input modes for multimodal human-computer interaction. In: Tan T, Shi Y, Gao W (eds) Advances in multimodal interfaces ICMI 2000, lecture notes in computer science, vol 1948, Springer, Berlin, pp 49–56. doi:10.1007/3-540-40063-X-7

  21. Wasinger R, Krüger A (2006) Modality preferences in mobile and instrumented environments. In: Proceedings of the 11th international conference on Intelligent user interfaces, IUI ’06, ACM, New York, NY, USA, pp 336–338. doi: 10.1145/1111449.1111529

Download references

Acknowledgments

The authors would like to thank Miriam Schmidt for fruitful support during the setup of this study, Steffen Walter for assistance with data collection and scoring, as well as Thilo Hörnle and Peter Kurzok for technical assistance. This work is originated in the Transregional Collaborative Research Centre SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems” funded by the German Research Foundation (DFG).

Author information

Authors and Affiliations

  1. Institute of Media Informatics, Ulm University, Ulm, Germany

    Felix Schüssel, Frank Honold & Michael Weber

Authors
  1. Felix Schüssel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frank Honold
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Weber
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Felix Schüssel.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Schüssel, F., Honold, F. & Weber, M. Influencing factors on multimodal interaction during selection tasks. J Multimodal User Interfaces 7, 299–310 (2013). https://doi.org/10.1007/s12193-012-0117-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-012-0117-5

Keywords

Search

Navigation