Skip to main content
SpringerLink
Log in

Do Not Shoot the Messenger: Effect of System Critical Feedback on User-Perceived Usability

  • Conference paper
  • First Online:
Human-Computer Interaction (HCII 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14012))

Included in the following conference series:

  • 677 Accesses

Abstract

Measuring perceived usability with questionnaires is a common practice for usability researchers and practitioners. This paper investigates whether there is any user bias towards the perceived usability of a system, when this system administers critical feedback to its users. These systems make decisions that substantially affect their users’ lives, such as automated medical diagnosis, bank loan approval etc. In our study, we gathered data from three, almost identical, systems used to apply for a consumer loan and communicate the decision to the applicant. Our dataset involves a total of 332 applicants who completed the UMUX-LITE questionnaire after receiving the system decision (approved, rejected) for their loan. Results showed that participants who had their loans approved (positive system critical feedback) provided significantly higher UMUX-LITE scores compared to participants who had their loans rejected (negative system critical feedback). This finding suggests that one should pay attention when measuring perceived usability of critical feedback administering systems as it tends to be biased from the critical feedback that users received.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Hertzum, M.: Images of usability. Int. J. Hum.-Comput. Interact. 26, 567–600 (2010). https://doi.org/10.1080/10447311003781300

    Article  Google Scholar 

  2. Brooke, J.: SUS: a “quick and dirty” usability scale. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, A.L. (eds.) Usability Evaluation in Industry. Taylor and Francis, London (1996)

    Google Scholar 

  3. Lewis, J.R.: IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int. J. Hum.-Comput. Interact. 7, 57–78 (1995). https://doi.org/10.1080/10447319509526110

    Article  Google Scholar 

  4. Lund, A.M.: Measuring usability with the USE questionnaire. Usabil. Interface 8, 3–6 (2001)

    Google Scholar 

  5. Lin, H.X., Choong, Y.-Y., Salvendy, G.: A proposed index of usability: a method for comparing the relative usability of different software systems. Behav. Inf. Technol. 16, 267–277 (1997). https://doi.org/10.1080/014492997119833

    Article  Google Scholar 

  6. Chin, J.P., Diehl, V.A., Norman, K.L.: Development of an instrument measuring user satisfaction of the human-computer interface. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 213–218. ACM, New York (1988). https://doi.org/10.1145/57167.57203

  7. Katsanos, C., Tselios, N., Xenos, M.: Perceived usability evaluation of learning management systems: a first step towards standardization of the system usability scale in Greek. In: Proceedings of the16th Panhellenic Conference on Informatics (PCI 2012), pp. 302–307 (2012)

    Google Scholar 

  8. Blažica, B., Lewis, J.R.: A Slovene translation of the system usability scale: the SUS-SI. Int. J. Hum.-Comput. Interact. 31, 112–117 (2015). https://doi.org/10.1080/10447318.2014.986634

    Article  Google Scholar 

  9. Borkowska, A., Jach, K.: Pre-testing of Polish translation of system usability scale (SUS). In: Borzemski, L., Grzech, A., Świątek, J., Wilimowska, Z. (eds.) Information Systems Architecture and Technology: Proceedings of 37th International Conference on Information Systems Architecture and Technology—ISAT 2016—Part I. AISC, vol. 521, pp. 143–153. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-46583-8_12

    Chapter  Google Scholar 

  10. Taheri, F.,Kavusi, A., Faghihnia Torshozi, Y., Farshad, A.A., Saremi, M.: Assessment of validity and reliability of Persian version of system usability scale (SUS) for traffic signs. Iran Occup. Health 14, 12–22 (2017)

    Google Scholar 

  11. Wang, Y., Lei, T., Liu, X.: Chinese system usability scale: translation, revision, psychological measurement. Int. J. Hum.-Comput. Interact. 36, 953–963 (2020). https://doi.org/10.1080/10447318.2019.1700644

    Article  Google Scholar 

  12. Gao, M., Kortum, P., Oswald, F.L.: Multi-language toolkit for the system usability scale. Int. J. Hum.-Comput. Interact. 36, 1883–1901 (2020). https://doi.org/10.1080/10447318.2020.1801173

    Article  Google Scholar 

  13. Hvidt, J.C.S., Christensen, L.F., Sibbersen, C., Helweg-Jørgensen, S., Hansen, J.P., Lichtenstein, M.B.: Translation and validation of the system usability scale in a Danish mental health setting using digital technologies in treatment interventions. Int. J. Hum.-Comput. Interact. 36, 709–716 (2020). https://doi.org/10.1080/10447318.2019.1680922

    Article  Google Scholar 

  14. Katsanos, C., Tselios, N., Liapis, A.: PSSUQ-GR: a first step towards standardization of the post-study system usability questionnaire in Greek. In: CHI Greece 2021: 1st International Conference of the ACM Greek SIGCHI Chapter, pp. Article23:1–Article23:6. ACM, New York (2021)

    Google Scholar 

  15. Erdinç, O., Lewis, J.R.: Psychometric evaluation of the T-CSUQ: the Turkish version of the computer system usability questionnaire. Int. J. Hum.-Comput. Interact. 29, 319–326 (2013). https://doi.org/10.1080/10447318.2012.711702

    Article  Google Scholar 

  16. Al-Hassan, A.A., AlGhannam, B., Naser, M.B., Alabdulrazzaq, H.: An Arabic translation of the computer system usability questionnaire (CSUQ) with psychometric evaluation using Kuwait university portal. Int. J. Hum.–Comput. Interact, pp. 1–8 (2021). https://doi.org/10.1080/10447318.2021.1926117

  17. Lewis, J.R.: The system usability scale: past, present, and future. Int. J. Hum.-Comput. Interact. 34, 577–590 (2018). https://doi.org/10.1080/10447318.2018.1455307

    Article  Google Scholar 

  18. Tullis, T., Stetson, J.: A comparison of questionnaires for assessing website usability. In: usability professionals association (UPA). In: 2004 Conference, pp. 7–11 (2004)

    Google Scholar 

  19. Finstad, K.: The usability metric for user experience. Interact. Comput. 22, 323–327 (2010). https://doi.org/10.1016/j.intcom.2010.04.004

    Article  Google Scholar 

  20. Lewis, J.R., Utesch, B.S., Maher, D.E.: UMUX-LITE: when there’s no time for the SUS. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2099–2102 (2013)

    Google Scholar 

  21. Lewis, J.R.: Psychometric evaluation of an after-scenario questionnaire for computer usability studies: the ASQ. SIGCHI Bull. 23, 78–81 (1991). https://doi.org/10.1145/122672.122692

    Article  Google Scholar 

  22. Sauro, J., Dumas, J.S.: Comparison of three one-question, post-task usability questionnaires. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1599–1608. ACM, New York (2009). https://doi.org/10.1145/1518701.1518946

  23. Lewis, J.R.: Measuring perceived usability: the CSUQ, SUS, and UMUX. Int. J. Hum.-Comput. Interact. 34, 1148–1156 (2018). https://doi.org/10.1080/10447318.2017.1418805

    Article  Google Scholar 

  24. Borsci, S., Buckle, P., Walne, S.: Is the LITE version of the usability metric for user experience (UMUX-LITE) a reliable tool to support rapid assessment of new healthcare technology? Appl. Ergon. 84, 103007 (2020). https://doi.org/10.1016/j.apergo.2019.103007

    Article  Google Scholar 

  25. Lewis, J.R., Utesch, B.S., Maher, D.E.: Investigating the correspondence between UMUX-LITE and SUS scores. In: Marcus, A. (ed.) DUXU 2015. LNCS, vol. 9186, pp. 204–211. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-20886-2_20

    Chapter  Google Scholar 

  26. Granic, A., Cukusic, M.: Usability testing and expert inspections complemented by educational evaluation: a case study of an e-Learning platform. Educ. Technol. Soc. 14, 107–123 (2011)

    Google Scholar 

  27. Bangor, A., Kortum, P., Miller, J.: An empirical evaluation of the system usability scale. Int. J. Hum.-Comput. Interact. 24, 574–594 (2008). https://doi.org/10.1080/10447310802205776

    Article  Google Scholar 

  28. Orfanou, K., Tselios, N., Katsanos, C.: Perceived usability evaluation of learning management systems: empirical evaluation of the system usability scale. Int. Rev. Res. Open Distrib. Learn. 16, 227–246 (2015)

    Google Scholar 

  29. Berkman, M.I., Karahoca, D.: Re-assessing the usability metric for user experience (UMUX) scale. J. Usabil. Stud. 11, 89–109 (2016)

    Google Scholar 

  30. Kortum, P., Bangor, A.: Usability ratings for everyday products measured with the system usability scale. Int. J. Hum.-Comput. Interact. 29, 67–76 (2013). https://doi.org/10.1080/10447318.2012.681221

    Article  Google Scholar 

  31. Kortum, P., Oswald, F.L.: The impact of personality on the subjective assessment of usability. Int. J. Hum.-Comput. Interact. 34, 177–186 (2018). https://doi.org/10.1080/10447318.2017.1336317

    Article  Google Scholar 

  32. Sauro, J.: Does prior experience affect perceptions of usability?. https://measuringu.com/prior-exposure. Accessed 30 Jan 2023

  33. McLellan, S., Muddimer, A., Peres, S.C.: The effect of experience on system usability scale ratings. J. Usabil. Stud. 7, 56–67 (2012)

    Google Scholar 

  34. Cutumisu, M., Schwartz, D.L.: The impact of critical feedback choice on students’ revision, performance, learning, and memory. Comput. Hum. Behav. 78, 351–367 (2018)

    Article  Google Scholar 

  35. Nunnally, J., Bernstein, I.: Psychometric Theory. McGraw-Hill Humanities/Social Sciences/Languages (1994)

    Google Scholar 

  36. Cohen, J.: A power primer. Psychol. Bull. 112, 155–159 (1992)

    Article  Google Scholar 

  37. Robinson, M.M., Morsella, E.: The subjective effort of everyday mental tasks: attending, assessing, and choosing. Motiv. Emot. 38(6), 832–843 (2014). https://doi.org/10.1007/s11031-014-9441-2

    Article  Google Scholar 

  38. Liapis, A., Katsanos, C., Xenos, M., Orphanoudakis, T.: Effect of personality traits on UX evaluation metrics: a study on usability issues, valence-arousal and skin conductance. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pp. LBW2721:1–LBW2721:6. ACM, New York (2019). https://doi.org/10.1145/3290607.3312995

  39. Liapis, A., Katsanos, C., Sotiropoulos, D., Xenos, M., Karousos, N.: Stress recognition in human-computer interaction using physiological and self-reported data: a study of gender differences. In: Proceedings of the 19th Panhellenic Conference on Informatics, pp. 323–328. ACM, New York (2015). https://doi.org/10.1145/2801948.2801964

Download references

Acknowledgments

We thank Multitude for allowing us to use the collected usability data for our research purposes in this paper. We also thank the anonymous participants that volunteered to assess the perceived usability of the evaluated system and thus made this research possible.

Author information

Authors and Affiliations

  1. Multitude SE, Helsinki, Finland

    Georgios Melissourgos

  2. Department of Informatics, Aristotle University of Thessaloniki, Thessaloniki, Greece

    Christos Katsanos

Authors
  1. Georgios Melissourgos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christos Katsanos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christos Katsanos .

Editor information

Editors and Affiliations

  1. The Open University of Japan, Chiba, Japan

    Masaaki Kurosu

  2. Hosei University, Tokyo, Japan

    Ayako Hashizume

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Melissourgos, G., Katsanos, C. (2023). Do Not Shoot the Messenger: Effect of System Critical Feedback on User-Perceived Usability. In: Kurosu, M., Hashizume, A. (eds) Human-Computer Interaction. HCII 2023. Lecture Notes in Computer Science, vol 14012. Springer, Cham. https://doi.org/10.1007/978-3-031-35599-8_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-35599-8_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-35598-1

  • Online ISBN: 978-3-031-35599-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation