Abstract
Usability evaluation is an indispensable issue during the development of new interfaces and interaction paradigms [1]. Although a wide range of reliable usability evaluation methods exists for graphical user interfaces, mature methods are rarely available for speech-based interfaces [2]. When it comes to multimodal interfaces, no standardized approach has so far been established. In previous studies [3], it was shown that usability questionnaires initially developed for unimodal systems may lead to unreliable results when applied to multimodal systems. In the current study, we therefore used several data sources (direct and indirect measurements) to evaluate two unimodal versions and one multimodal version of an information system. We investigated, to which extent the different data showed concordance for the three system versions. The aim was to examine, if, and under which conditions, common and widely used methods originally developed for graphical user interfaces are also appropriate for speech-based and multimodal intelligent interfaces.
Chapter PDF
Similar content being viewed by others
References
Sturm, J.: On the usability of multimodal interaction for mobile access to information services. PhD thesis, Radboud University Nijmegen, Nijmegen, The Netherlands (2005)
Larsen, L.B.: Assessment of spoken dialogue system usability - what are we really measuring? In: Eurospeech 2003, pp. 1945–1948 (2003)
Naumann, A., Wechsung, I.: Developing Usability Methods for Multimodal Systems: The Use of Subjective and Objective Measures. In: Proceedings of the International Workshop on Meaningful Measures: Valid Useful User Experience Measurement (VUUM), pp. 8–12 (2008)
Bolt, R.A.: “Put-that-there”: Voice and gesture at the graphics interface. In: Proceedings of the 7th Annual Conference on Computer Graphics and interactive Techniques (1980)
Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: QuickSet: multimodal interaction for distributed applications. In: Proceedings of the Fifth ACM international Conference on Multimedia (1997)
Martin, J., Buisine, S., Pitel, G., Bernsen, N.O.: Fusion of children’s speech and 2D gestures when conversing with 3D characters. Signal Process 86, 12 (2006)
Perzanowski, D., Schultz, A.C., Adams, W., Marsh, E., Bugajska, M.: Building a Multimodal Human-Robot Interface. IEEE Intelligent Systems 16(1), 16–21 (2001)
Thalmann, D.: The virtual human as a multimodal interface. In: Proceedings of the Working Conference on Advanced Visual interfaces (2000)
Möller, S., Engelbrecht, K.-P., Kühnel, C., Wechsung, I., Weiss, B.: Evaluation of Multimodal Interfaces for Ambient Intelligence. In: Aghajan, H., López-Cózar Delgado, R., Augusto, J.C. (eds.) Human-Centric Interfaces for Ambient Intelligence. Elsevier, Amsterdam (2009)
Wechsung, I., Naumann, A.B.: Evaluation Methods for Multimodal Systems: A Comparison of Standardized Usability Questionnaires. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Pieraccini, R., Weber, M. (eds.) PIT 2008. LNCS (LNAI), vol. 5078, pp. 276–284. Springer, Heidelberg (2008)
Nielsen, J., Levy, J.: Measuring usability: Preference vs. performance. Communications of the ACM 37, 4 (1994)
Sauro, J., Kindlund, E.: A method to standardize usability metrics into a single score. In: Proc. CHI 2005. ACM Press, New York (2005)
Krämer, N.C., Nitschke, J.: Ausgabemodalitäten im Vergleich: Verändern sie das Eingabeverhalten der Benutzer? [Output modalities in comparison: Do they change user’s input behaviour?]. In: Marzi, R., Karavezyris, V., Erbe, H.-H., Timpe, K.-P. (Hrsg.) Bedienen und Verstehen. 4. Berliner Werkstatt Mensch-Maschine-Systeme, VDI-Verlag, Düsseldorf (2002)
Möller, S.: Messung und Vorhersage der Effizienz bei der Interaktion mit Sprachdialogdiensten [Measuring and predicting efficiency for the interaction with speech dialogue systems]. In: Langer, S., Scholl, W. (eds.) Fortschritte der Akustik - DAGA 2006. DEGA, Berlin (2006)
Frøkjær, E., Hertzum, M., Hornbæk, K.: Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? In: Proc. CHI 2000. ACM Press, New York (2000)
Hornbæk, K., Law, E.L.: Meta-analysis of correlations among usability measures. In: Proc. CHI 2007. ACM Press, New York (2007)
Hassenzahl, M., Burmester, M., Koller, F.: AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität [A questionnaire for measuring perceived hedonic and pragmatic quality]. In: Ziegler, J., Szwillus, G. (eds.) Mensch & Computer 2003. Interaktion in Bewegung. B.G. Teubner, Stuttgart (2003)
Hone, K., Graham, R.: Subjective assessment of speech-system interface usability. In: Proceedings of Eurospeech 2001, vol. 3 (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Metze, F., Wechsung, I., Schaffer, S., Seebode, J., Möller, S. (2009). Reliable Evaluation of Multimodal Dialogue Systems. In: Jacko, J.A. (eds) Human-Computer Interaction. Novel Interaction Methods and Techniques. HCI 2009. Lecture Notes in Computer Science, vol 5611. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02577-8_9
Download citation
DOI: https://doi.org/10.1007/978-3-642-02577-8_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-02576-1
Online ISBN: 978-3-642-02577-8
eBook Packages: Computer ScienceComputer Science (R0)