Skip to main content
Springer Nature Link
Log in

Artifacts, Tools and Generalizing Usability Test Results

  • Conference paper
  • First Online:
Nordic Contributions in IS Research (SCIS 2015)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 223))

Included in the following conference series:

  • 664 Accesses

Abstract

Usability testing has gained a rather stable status as a method for usability evaluation even though it has both low reliability and validity. The sources of result variance are well acknowledged among researchers and practitioners. However, the validity problem has not been explicated or exemplified although it is frequently discussed in the literature how the results of usability tests should be interpreted and to what extent results are generalizable. We employ Activity Theory and a case example to argue that the validity problem is mainly caused by the fact that what we are testing are artifacts and what people are using in their real life activities are tools and these two entities are qualitatively different. Basing on our analysis, the effects of the reliability and validity problems on the application of usability testing and its role as one of the tools in the design process are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Barnum, C.M.: Usability Testing Essentials: Ready, Set… Test! Morgan Kaufmann, Burlington (2011)

    Google Scholar 

  2. Hvannberg, E.T., Law, E.L.-C., Lárusdóttir, M.K.: Heuristic evaluation: comparing ways of finding and reporting usability problems. Interact. Comput. 19, 225–240 (2007)

    Article  Google Scholar 

  3. Andreasen, M.S., Nielsen, H.V., Schrøder, S.O., Stage, J.: What happened to remote usability testing? an empirical study of three methods. In: Proceedings of CHI 2007, pp. 1405–1414. ACM Press (2007)

    Google Scholar 

  4. Alonso-Ríos, D., Vázquez-García, A., Mosqueira-Rey, E., Moret-Bonillo, V.: Usability: a critical analysis and a taxonomy. Int. J. Hum.-Comput. Interact. 26(1), 53–74 (2009)

    Article  Google Scholar 

  5. Hornbæk, K.: Usability evaluation as idea generation. In: Cockton, G.G., Hvannberg, E.T., Law, E. (eds.) Maturing Usability: Quality in Software, Interaction and Value, pp. 267–286. Springer, London (2008)

    Chapter  Google Scholar 

  6. Riemer, K., Vehring, N.: It’s not a property! exploring the sociomateriality of software usability. In: Proceedings of the International Conference on Information Systems (ICIS), Phoenix, Arizona, pp. 1–19 (2010)

    Google Scholar 

  7. Kuutti, K.: Activity theory as a potential framework for human-computer interaction research. In: Nardi, B.A. (ed.) Context and Consciousness, pp. 17–44. MIT Press, Cambridge (1995)

    Google Scholar 

  8. Suchman, L.A.: Plans and Situated Actions. The Problem of Human-Machine Communication. Thesis, XEROX PARC. ISL-6 (1985)

    Google Scholar 

  9. Molich, R., Dumas, J.S.: Comparative usability evaluation (CUE-4). Behav. Inf. Technol. 27(3), 263–281 (2008)

    Article  Google Scholar 

  10. Nielsen, J., Landauer, T.K.: A mathematical model of the finding of usability problems. In: Proceedings of CHI, pp. 206–213. ACM (1993)

    Google Scholar 

  11. Følstad, A., Law, E.L.-C., Hornbæk, K.: Analysis in practical usability evaluation: a survey study. In: Proceedings of CHI 2012, pp. 2127–2136. ACM Press (2012)

    Google Scholar 

  12. Molich, R., Ede, M.E., Kaasgaard, K., Karyakin, B.: Comparative usability evaluation. Behav. Inf. Technol. 23, 65–74 (2004)

    Article  Google Scholar 

  13. Jeffries, R., Miller, J.R., Wharton, C., Uyeda, K.M.: User interface evaluation in the real world: a comparison of four techniques. In: Proceedings of CHI 1991, pp. 119–124. ACM (1991)

    Google Scholar 

  14. Vermeeren, A., van Kesteren, I., Bekker, M.: Managing the evaluator effect in user testing. In: Proceedings of Interact 2003, pp. 647–654. IOS Press (2003)

    Google Scholar 

  15. Boren, T., Ramey, J.: Thinking aloud: reconciling theory and practice. IEEE Trans. Prof. Commun. 43, 261–278 (2000)

    Article  MATH  Google Scholar 

  16. Molich, K., Jeffries, R., Dumas, J.S.: Making usability recommendations useful and usable. J. Usability Stud. 2, 162–179 (2007)

    Google Scholar 

  17. Nielsen, J.: Usability Engineering. Academic Press, Boston (1993)

    MATH  Google Scholar 

  18. Andre, T.S., Belz, S.M., McCrearys, F.A., Hartson, H.R.: Testing a framework for reliable classification of usability problems. In: Proceedings of Human Factors and Ergonomics Society Annual Meeting, vol. 44, pp. 573–576. SAGE Publications (2000)

    Google Scholar 

  19. Lindgaard, G., Chattratichart, J.: Usability testing: what have we overlooked? In: Proceedings of CHI 2007, pp. 1415–1424. ACM (2007)

    Google Scholar 

  20. Hertzum, M., Molich, R., Jacobsen, N.E.: What you get is what you see: revisiting the evaluator effect in usability tests. Behav. Inf. Technol. 33(2), 144–162 (2014)

    Article  Google Scholar 

  21. Duh, H.B.-L., Tan, G.C.B., Chen, V.H.: Usability evaluation for mobile device: a comparison of laboratory and field tests. In: Proceedings of MobileHCI, pp. 181–186. ACM (2006)

    Google Scholar 

  22. Drost, E.A.: Validity and reliability in social science research. Educ. Res. Perspect. 38, 105–123 (2011)

    Google Scholar 

  23. Trochim, W.M.: The Research Methods Knowledge Base, 2nd edn. (2006). http://www.socialresearchmethods.net/kb/, version current as of 20 October 2006. Retrieved 20 Jan 2015

  24. Tarkkanen, K., Reijonen, P., Tétard, F., Harkke, V.: Back to user-centered usability testing. In: Holzinger, A., Ziefle, M., Hitz, M., Debevc, M. (eds.) SouthCHI 2013. LNCS, vol. 7946, pp. 91–106. Springer, Heidelberg (2013)

    Chapter  Google Scholar 

  25. Arhippainen, L.: Studying User Experience: Issues and Problems of Mobile Services - Case ADAMOS: User Experience (Im)possible to Catch? Acta Universitatis Ouluensis. Series A, Scientiae rerum naturalium (528) (2013)

    Google Scholar 

  26. Woolrych, A., Hornbæk, K., Frøkjær, E., Cockton, G.: Ingredients and meals rather than recipes: a proposal for research that does not treat usability evaluation methods as indivisible wholes. Int. J. Hum.-Comput. Interact. 27(10), 940–970 (2011)

    Article  Google Scholar 

  27. Merriam-Webster Online dictionary. http://www.merriam-webster.com/dictionary/

  28. Suchman, L.: Plans and Situated Actions. The Problem of Human-Machine Communication. Cambridge University Press, Cambridge (1987)

    Google Scholar 

  29. Suchman, L.: Human-Machine Reconfigurations. Plans and Situated Actions. Cambridge University Press, Cambridge (2007)

    Google Scholar 

  30. Duguid, P.: On Rereading. Suchman and Situated Action. Le Libellio d’ AEGIS 8, 2 Été, 3–9 (2012)

    Google Scholar 

  31. Bardram, J., Doryab, A.: Activity analysis – applying activity theory to analyze complex work in hospitals. In: CSCW 2011, pp. 455–464. ACM, New York (2011)

    Google Scholar 

  32. Cockton, G., Lavery, D.: A framework for usability problem extraction. In: Sasse, M.A., Johnson, C.V. (eds.) Proceedings of Interact 1999, pp. 344–352. IOS Press (1999)

    Google Scholar 

  33. Butler, S.: The Note-Books of Samuel Butler. Edited by Henry Festing Jones (1912). http://www.gutenberg.org/ebooks/6173. Accessed 8 Feb 2015

  34. ISO 9241-11:1998 Guidance on Usability. International Organization for Standardization, ISO 9241-11 (1998). http://www.iso.org (1998)

  35. Kjeldskov, Jesper, Skov, Mikael B., Als, Benedikte S., Høegh, Rune Thaarup: Is it worth the hassle? exploring the added value of evaluating the usability of context-aware mobile systems in the field. In: Brewster, Stephen, Dunlop, Mark D. (eds.) Mobile HCI 2004. LNCS, vol. 3160, pp. 61–73. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  36. Rogers, Y., Connelly, K.H., Tedesco, L., Hazlewood, W., Kurtz, A., Hall, R.E., Hursey, J., Toscos, T.: Why it’s worth the Hassle: the value of in-situ studies when designing Ubicomp. In: Krumm, J., Abowd, G.D., Seneviratne, A., Strang, T. (eds.) UbiComp 2007. LNCS, vol. 4717, pp. 336–353. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  37. Nielsen, C.M., Overgaard, M., Pedersen, M.B., Stage, J., Stenild, S.: It’s worth the Hassle! the added value of evaluating the usability of mobile systems in the field. In: Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles, pp. 272–280. ACM (2006)

    Google Scholar 

  38. Nevo, B.: Face validity revisited. J. Educ. Meas. 22(4), 287–293 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  39. Hornbæk, K., Frøkjær, E.: Making use of business goals in usability evaluation: an experiment with novice evaluators. In: Proceedings of CHI 2008, pp. 903–912. ACM (2008)

    Google Scholar 

  40. Uldall-Espersen, T., Frøkjær, E., Hornbæk, K.: Tracing impact in a usability improvement process. Interact. Comput. 20(1), 48–63 (2008)

    Article  MATH  Google Scholar 

  41. Hornbæk, K.: Dogmas in the assessment of usability evaluation methods. Behav. Inf. Technol. 29(1), 97–111 (2010)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Information Systems Science, Turku School of Economics, University of Turku, Rehtorinpellonkatu 3, 20500, Turku, Finland

    Pekka Reijonen & Kimmo Tarkkanen

Authors
  1. Pekka Reijonen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kimmo Tarkkanen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kimmo Tarkkanen .

Editor information

Editors and Affiliations

  1. University of Oulu, Oulu, Finland

    Harri Oinas-Kukkonen

  2. University of Oulu, Oulu, Finland

    Netta Iivari

  3. University of Oulu, Oulu, Finland

    Kari Kuutti

  4. University of Oulu, Oulu, Finland

    Anssi Öörni

  5. University of Oulu, Oulu, Finland

    Mikko Rajanen

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Reijonen, P., Tarkkanen, K. (2015). Artifacts, Tools and Generalizing Usability Test Results. In: Oinas-Kukkonen, H., Iivari, N., Kuutti, K., Öörni, A., Rajanen, M. (eds) Nordic Contributions in IS Research. SCIS 2015. Lecture Notes in Business Information Processing, vol 223. Springer, Cham. https://doi.org/10.1007/978-3-319-21783-3_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-21783-3_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-21782-6

  • Online ISBN: 978-3-319-21783-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics