Skip to main content
SpringerLink
Log in

Measuring the Quality of Service and Quality of Experience of multimodal human–machine interaction

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Quality of Service (QoS) and Quality of Experience (QoE) have to be considered when designing, building and maintaining services involving multimodal human–machine interaction. In order to guide the assessment and evaluation of such services, we first develop a taxonomy of the most relevant QoS and QoE aspects which result from multimodal human–machine interactions. It consists of three layers: (1) The quality factors influencing QoS and QoE related to the user, the system, and the context of use; (2) the QoS interaction performance aspects describing user and system behavior and performance; and (3) the QoE aspects related to the quality perception and judgment processes taking place within the user. For each of these layers, we then provide metrics which are able to capture the QoS and QoE aspects in a quantitative way, either via questionnaires or performance measures. The metrics are meant to guide system evaluation and make it more systematic and comparable.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Similar content being viewed by others

References

  1. Lamel L, Bennacef S, Gauvain JL, Dartigues H, Temem JN (2002) User evaluation of the MASK kiosk. Speech Commun 38:131–139

    Article  MATH  Google Scholar 

  2. Bernsen NO, Dybkjær L, Kiilerich S (2004) Evaluating conversation with Hans Christian Andersen. In: Proc LREC, vol 3, Lisbon, pp 1011–1014

    Google Scholar 

  3. Wahlster W (2006) SmartKom: foundations of multimodal dialogue systems. Springer, Berlin

    Book  Google Scholar 

  4. Turunen M, Hakulinen J, Ståhl O, Gambäck B, Hansen P, Rodriguez Gancedo M, Santos de la Camara R, Smith C, Charlton D, Cavazza M (2011) Multimodal and mobile conversational health and fitness companions. Comput Speech Lang 25:192–209

    Article  Google Scholar 

  5. Cassell J, Kopp S, Tepper P, Ferriman K, Striegnitz K (2007) Trading spaces: how humans and humanoids use speech and gesture to give directions. In: Nishida T (ed) Conversational informatics. Wiley, New York, pp 133–160

    Google Scholar 

  6. Gibbon D, Mertins I, Moore RK (eds) (2000) Handbook of multimodal and spoken dialogue systems. Kluwer Academic, Boston

    Google Scholar 

  7. Coutaz J, Nigay L, Salber D, Blandford AE, May J, Young RM (1995) Four easy pieces for assessing the usability of multimodal interaction: the CARE properties. In: Nordby K, Helmersen PH, Gilmore DJ, Arnesen SA (eds) Human–computer interaction, proc interact 1995. Chapman & Hall, London, pp 115–120

    Google Scholar 

  8. Wickens CD (2008) Multiple resources and mental workload. Hum Factors 50:449–455

    Article  Google Scholar 

  9. Bernsen NO, Dybkjær L (2010) Multimodal usability. Human–computer interaction series. Springer, Berlin

    Book  Google Scholar 

  10. ISO 25000 (2005) Software engineering—software product quality requirements and evaluation (SQuaRE)—guide to SQuaRE. International Organization for Standardization, Geneva

    Google Scholar 

  11. Gray WD, Salzman MC (1998) Damaged merchandise? A review of experiments that compare usability evaluation methods. Hum-Comput Interact 13(3):203–261

    Article  Google Scholar 

  12. Olson GM, Moran TP (1998) Commentary on Damaged Merchandise? Hum-Comput Interact 13(3):263–323

    Article  Google Scholar 

  13. Hornbæk K (2006) Current practice in measuring usability: challenges to usability studies and research. Int J Man-Mach Stud 64(2):79–102

    Google Scholar 

  14. Hartson HR, Andre TS, Williges RC (2003) Criteria for evaluating usability evaluation methods. Int J Hum-Comput Interact 15(1):145–181

    Article  Google Scholar 

  15. ITU-T Rec. E.800 (1994) Terms and definitions related to Quality of Service and network performance including dependability. International Telecommunication Union, Geneva

    Google Scholar 

  16. ETSI TR 102 643 (2010) Human factors (HF); Quality of Experience (QoE) requirements for real-time communication services. European Telecommunications Standards Institute, Sophia Antipolis

    Google Scholar 

  17. Brooks P, Hestnes B (2010) User measures of quality of experience: why being objective and quantitative is important. IEEE Netw 24:8–13

    Article  Google Scholar 

  18. ITU-T Rec. P.10 (2007) Vocabulary for performance and Quality of Service. International Telecommunication Union, Geneva

    Google Scholar 

  19. Jekosch U (2005) Voice and speech quality perception. Assessment and evaluation. Springer, Berlin

    Google Scholar 

  20. Fraser N (1997) Assessment of interactive systems. In: Gibbon D, Moore R, Winski R (eds) Handbook on standards and resources for spoken language systems. Mouton de Gruyter, Berlin, pp 564–615

    Google Scholar 

  21. ITU-T Suppl. 24 to P-Series Rec. (2005) Parameters describing the interaction with spoken dialogue systems. International Telecommunication Union, Geneva

    Google Scholar 

  22. ITU-T Rec. P.851 (2003) Subjective quality evaluation of telephone services based on spoken dialogue systems. International Telecommunication Union, Geneva

    Google Scholar 

  23. Hone KS, Graham R (2000) Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI). Nat Lang Eng 6(3/4):287–303

    Article  Google Scholar 

  24. Nielsen J (1993) Usability engineering. Academic Press, Boston

    MATH  Google Scholar 

  25. Alonso-Ríos D, Vázquez-García A, Mosqueira-Rey E, Moret-Bonillo V (2010) Usability: a critical analysis and a taxonomy. Int J Hum-Comput Interact 26(1):53–74

    Article  Google Scholar 

  26. Möller S, Kühnel C, Weiss B (2010) Extending Suppl 24 to P-series towards multimodal systems and services. International Telecommunication Union, Geneva. Source: Deutsche Telekom Laboratories, ITU-T SIG12 Meeting 18–27 May 2010

    Google Scholar 

  27. Möller S (2005) Quality of telephone-based spoken dialogue systems. Springer, New York

    Google Scholar 

  28. Pallett D, Fiscus J, Fisher W, Garofolo J (1993) Benchmark tests for the DARPA spoken language program. In: Proc DARPA human language technology workshop, Princeton, pp 7–18

    Chapter  Google Scholar 

  29. Bennett CL, Black AW (2006) The blizzard challenge 2006. In: Proc blizzard satellite workshop to interspeech, Pittsburgh

    Google Scholar 

  30. Möller S (2005) Quality of telephone-based spoken dialogue systems. Springer, New York

    Google Scholar 

  31. Hermann F, Niedermann I, Peissner M, Henke K, Naumann A (2007) Users interact differently: towards a usability-oriented taxonomy. In: Jacko J (ed) Interaction design and usability, proc HCII 2007, vol 1. Springer, Heidelberg, pp 812–817

    Chapter  Google Scholar 

  32. Naumann AB, Hermann F, Peissner M, Henke K (2008) Interaktion mit Informations- und Kommunikationstechnologie: Eine Klassifikation von Benutzertypen [interaction with information and communication technology: a classification of user types]. In: Herczeg M, Kindsmüller MC (eds) Mensch & computer 2008: Viel Mehr interaktion. Oldenbourg Wissenschaftsverlag, München, pp 37–45. http://mc.informatik.uni-hamburg.de/konferenzbaende/mc2008/konferenzband/mc2008_05_naumann.pdf

    Google Scholar 

  33. Heinssen RK, Glass CR, Knight LA (1987) Assessing computer anxiety: development and validation of the computer anxiety rating scale. Comput Hum Behav 3(1):49–59

    Article  Google Scholar 

  34. Van Vliet PJA, Kletke MG, Chakraborty G (1994) The measurement of computer literacy—a comparison of self-appraisal and objective tests. Int J Hum-Comput Stud 40(5):835–857

    Article  Google Scholar 

  35. Richter T, Naumann J, Groeben N (2000) Attitudes toward the computer: construct validation of an instrument with scales differentiated by content. Comput Hum Behav 16:473–491

    Article  Google Scholar 

  36. Compeau DR, Higgins CA (1995) Computer self efficacy: development of a measure and initial test. Manag Inf Syst Q June:189–211

    Article  Google Scholar 

  37. Smith B, Caputi P, Rawstorne P (2007) The development of a measure of subjective computer experience. Comput Hum Behav 23(1):127–145

    Article  Google Scholar 

  38. Kalbe E, Kessler J, Calabrese P, Smith R, Passmore AP, Brand M, Bullock R (2004) DemTect: a new, sensitive cognitive screening test to support the diagnosis of mild cognitive impairment and early dementia. Int J Geriatr Psychiatry 19:136–143

    Article  Google Scholar 

  39. Lopez-Cozar Delgado R, Araki M (2005) Spoken, multilingual and multimodal dialogue systems—development and assessment. Wiley, New York

    Book  Google Scholar 

  40. Kühnel C, Weiss B, Möller S (2010) Parameters describing multimodal interaction—definitions and three usage scenarios. In: Proc INTERSPEECH, pp 2014–2017

    Google Scholar 

  41. Mostefa D, Garcia M-N, Choukri K (2006) Evaluation of multimodal components within CHIL: the evaluation packages and results. In: Proc LREC, Genoa, pp 915–918

    Google Scholar 

  42. Bernsen NO (2002) Multimodality in language and speech systems—from theory to design support tool. In: Granström B, House D, Karlsson I (eds) Multimodality in language and speech systems. Kluwer Academic, Dordrecht, pp 93–148

    Google Scholar 

  43. Grice H (1975) Logic and conversation. In: Cole P, Morgan JL (eds) Syntax and semantics. Speech acts, vol 3. Academic Press, New York, pp 41–58

    Google Scholar 

  44. Zimbardo PG (1995) Psychologie. Springer, Berlin

    Google Scholar 

  45. Borg G (1982) Psychophysical bases of perceived exertion. Med Sci Sports Exerc 14:377–381

    Google Scholar 

  46. De Waard D (1996) The measurement of drivers’ mental workload. PhD thesis, University of Groningen, Haren

  47. Hart SG, Staveland LE (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Hancock P, Meshkati N (eds) Human mental workload. North Holland, Amsterdam, pp 139–183

    Chapter  Google Scholar 

  48. Zijlstra FRH (1993) Efficiency in work behavior. A design approach for modern tools. PhD thesis, Delft University of Technology, Delft University Press, Delft

  49. Cacioppo JT, Tassinary LG, Berntson GG (eds) (2002) Handbook of psychophysiology, 3rd edn. Cambridge University Press, New York

    Google Scholar 

  50. Hassenzahl M, Platz A, Burmester M, Lehner K (2000) Hedonic and ergonomic quality aspects determine a software’s appeal. In: Proc CHI 2000, Den Haag, pp 201–208

    Google Scholar 

  51. Mandryk RL, Inkpen K, Calvert TW (2006) Using psycho-physiological techniques to measure user experience with entertainment technologies. Behav Inf Technol 25(2):141–158

    Article  Google Scholar 

  52. Chateau N, Gros L, Durin V, Macé A (2006) Redrawing the link between customer satisfaction and speech quality. In: Proc 2nd ISCA/DEGA tutorial and research workshop on perceptual quality of systems, Berlin, pp 88–94

    Google Scholar 

  53. Hornbæk K, Law EL (2007) Meta-analysis of correlations among usability measures. In: Proc CHI 2007. ACM Press, New York, pp 617–626

    Google Scholar 

  54. ISO 9241-11 (1999) Ergonomic requirements for office work with visual display terminals (VDTs). Part 11: guidance on usability. International Organization for Standardization, Geneva

    Google Scholar 

  55. ISO DIS 9241-210 (2010) Ergonomics of human system interaction—part 210: human-centred design for interactive systems (formerly known as 13407). International Organization for Standardization (ISO), Geneva

    Google Scholar 

  56. Bevan N (2009) What is the difference between the purpose of usability and user experience evaluation methods. In: Proc UXEM’09 (INTERACT’09), Uppsala

    Google Scholar 

  57. Kirakowski J, Corbett M (1993) SUMI: the software usability measurement inventory. Br J Educ Technol 24(3):210–212

    Article  Google Scholar 

  58. Lewis JR (1995) IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum-Comput Interact 7:57–78

    Article  Google Scholar 

  59. Hassenzahl M (2004) The interplay of beauty, goodness, and usability in interactive products. Hum-Comput Interact 19:319–349

    Article  Google Scholar 

  60. Kelly GA (1955) The psychology of personal constructs. Norton, New York

    Google Scholar 

  61. Burmester M, Mast M, Jäger K, Homans H (2010) Valence method for formative evaluation of user experience. In: Proc DIS ’10. ACM Press, New York, pp 364–367

    Google Scholar 

  62. Kujala S, Roto V, Vaananen-Vainio-Mattila K, Karapanos E, Sinnela A (2011) UX curve: a method for evaluating long-term user experience. Interact Comput. doi:10.1016/j.intcom.2011.06.005

    Google Scholar 

  63. Hassenzahl M, Wessler R (2000) Capturing design space from a user perspective: the repertory grid technique revisited. Int J Hum-Comput Interact 12(3/4):441–459

    Google Scholar 

  64. Hassenzahl M, Trautmann T (2001) Analysis of web sites with the repertory grid technique. In: Proc CHI 2001. ACM Press, New York, pp 167–168

    Google Scholar 

  65. Hassenzahl M, Diefenbach S, Göritz AS (2010) Needs, affect, and interactive products—facets of user experience. Interact Comput 22(5):353–362

    Article  Google Scholar 

  66. Karapanos E, Martens J-B, Hassenzahl M (2010) On the retrospective assessment of users’ experiences over time: memory or actuality? In: CHI 2010 extended abstracts on human factors in computing systems, Atlanta. ACM Press, New York, pp 4075–4080

    Google Scholar 

  67. Davis F (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. Manag Inf Syst Q 13(3):319–340

    Article  Google Scholar 

  68. Chin J, Diehl V, Norman K (1988) Development of an instrument measuring user satisfaction of the human–computer interface. In: Proceedings of SIGCHI 1988, pp 213–218

    Google Scholar 

  69. Brooke J (1996) SUS: A “quick and dirty” usability scale. In: Jordan P, Thomas B, Weerdmeester B, McClelland I (eds) Usability evaluation in industry. Taylor & Francis, London, pp 189–194

    Google Scholar 

  70. Gediga G, Hamborg K-C, Düntsch I (1999) The IsoMetrics usability inventory: an operationalisation of ISO 9241-10. Behav Inf Technol 18:151–164

    Article  Google Scholar 

  71. Lund AM (2001) Measuring usability with the USE questionnaire. In: Usability and user experience 8, STC usability SIG newsletter.

    Google Scholar 

  72. Ullrich D, Diefenbach S (2010) INTUI. Exploring the facets of intuitive interaction. In: Mensch & computer 2010 interaktive kulturen.

    Google Scholar 

  73. Naumann A, Hurtienne J, Israel JH, Mohs C, Kindsmüller MC, Meyer HA, Husslein S (2007) Intuitive use of user interfaces: defining a vague concept. In: Harris D (ed) Engineering psychology and cognitive ergonomics, vol 13, Proc HCII 2007. LNAI, vol 4562. Springer, Heidelberg, pp 128–136

    Chapter  Google Scholar 

  74. Polson PG, Lewis C, Rieman J, Wharton C (1992) Cognitive walkthroughs: a method for theory-based evaluation of user interfaces. Int J Man-Mach Stud 36:741–773

    Article  Google Scholar 

  75. John BE, Kieras DE (1996) Using GOMS for user interface design and evaluation: which technique? ACM Trans Comput-Hum Interact 3(4):287–319

    Article  Google Scholar 

  76. Kieras DE, Meyer DE (1997) An overview of the EPIC architecture for cognition and performance with application to human–computer interaction. Hum-Comput Interact 12(4):391–438

    Article  Google Scholar 

  77. Schleicher R, Trösterer S (2009) The ‘joy-of-use’-button: recording pleasant moments while using a PC. In: Human–computer interaction—INTERACT 2009, vol 5727/2009. Springer, Heidelberg

    Google Scholar 

  78. Hekkert P (2006) Design aesthetics: principles of pleasure in product design. Psychol Sci 48:157–172

    Google Scholar 

  79. Isbister K, Nass C (2000) Consistency of personality in interactive characters: verbal cues, non-verbal cues, and user characteristics. Int J Hum-Comput Stud 53(2):251–267

    Article  Google Scholar 

  80. Marcus A (1995) Principles of effective visual communication for graphical user interface design. In: Baecker R, Grudin J, Buxton W, Greenberg S (eds) Human-computer interaction: toward the year 2000. Morgan Kaufmann, San Francisco, pp 425–441

    Google Scholar 

  81. Mairesse F, Walker M, Mehl M, Moore R (2007) Using linguistic cues for the automatic recognition of personality in conversation and text. J Artif Intell Res 30. doi:10.1613/jair.2349

  82. Stelmaszweska H, Fields B, Blandford A (2004) Conceptualising user hedonic experience. In: Reed DJ, Baxter G, Blythe M (eds) Proc ECCE-12, living and working with technology. EACE, York, pp 83–89

    Google Scholar 

  83. Tractinsky N, Katz AS, Ikar D (2000) What is beautiful is usable. Interact Comput 13(2):127–145

    Article  Google Scholar 

  84. Mahlke S, Lindgaard G (2007) Emotional experiences and quality perceptions of interactive products. In: Jacko JA (ed) Proceedings of the 12th international conference on human–computer interaction: interaction design and usabilit. Lecture notes in computer science. Springer, Berlin/Heidelberg, pp 164–173

    Google Scholar 

  85. Hassenzahl M (2008) Aesthetics in interactive products: correlates and consequences of beauty. In: Schifferstein HNJ, Hekkert P (eds) Product experience. Elsevier, San Diego, pp 287–302

    Chapter  Google Scholar 

  86. Lavie T, Tractinsky N (2004) Assessing dimensions of perceived visual aesthetics of web sites. Int J Hum-Comput Stud 60:269–298

    Article  Google Scholar 

  87. Bradley MM, Lang PJ (1994) Measuring emotion: the self-assessment Manikin and the semantic differentia. J Behav Ther Exp Psychiatry 25:49–59

    Article  Google Scholar 

  88. Huisman G, Van Hout M (2008) The development of a graphical emotion measurement instrument using caricatured expressions: the LEMtool. In: Peter C, Crane E, Fabri M, Agius H, Axelrod L (eds) Emotion in HCI—designing for people: proceedings of the 2008 international workshop. Fraunhofer, Stuttgart, pp 5–7

    Google Scholar 

  89. Desmet PMA (2004) Measuring emotions: development and application of an instrument to measure emotional responses to products. In: Blythe MA, Overbeeke K, Monk AF, Wright PC (eds) Funology: from usability to enjoyment. Kluwer Academic, Norwell, pp 111–123

    Google Scholar 

  90. Bargas-Avila JA, Hornbæk K (2011) Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience. In: Proc CHI 2011, pp 2689–2698

    Google Scholar 

  91. Bevan N (1995) Usability is quality of use. In: Proc HCII 1995. Elsevier, Amsterdam, pp 349–354

    Google Scholar 

  92. Eurescom Project P.807 Deliverable 1 (1998) Jupiter II—usability, performability and interoperability trials in Europe. European Institute for Research and Strategic Studies in Telecommunications, Heidelberg

    Google Scholar 

  93. Davis FD (1993) User acceptance of information technology system characteristics, user perceptions and behavioral impacts. Int J Man-Mach Stud 38(3):475–487

    Article  Google Scholar 

  94. Ajzen I, Fishbein M (1980) Understanding attitudes and predicting social behavior. Prentice Hall, Englewood Cliffs/New York

    Google Scholar 

  95. Möller S, Engelbrecht K-P, Kühnel C, Wechsung I, Weiss B (2009) Evaluation of multimodal interfaces for ambient intelligence. In: Aghajan H, López-Cózar Delgado R, Augusto JC (eds) Human-centric interfaces for ambient intelligence. Elsevier, Amsterdam, pp 347–370

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Quality and Usability Lab, Deutsche Telekom Laboratories, TU Berlin, Berlin, Germany

    Ina Wechsung, Klaus-Peter Engelbrecht, Christine Kühnel, Sebastian Möller & Benjamin Weiss

Authors
  1. Ina Wechsung
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Klaus-Peter Engelbrecht
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christine Kühnel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sebastian Möller
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Benjamin Weiss
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Benjamin Weiss.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wechsung, I., Engelbrecht, KP., Kühnel, C. et al. Measuring the Quality of Service and Quality of Experience of multimodal human–machine interaction. J Multimodal User Interfaces 6, 73–85 (2012). https://doi.org/10.1007/s12193-011-0088-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-011-0088-y

Keywords

Search

Navigation