Skip to main content
SpringerLink
Log in

Coherent Heuristic Evaluation (CoHE): Toward Increasing the Effectiveness of Heuristic Evaluation for Novice Evaluators

  • Conference paper
  • First Online:
Design, User Experience, and Usability. Interaction Design (HCII 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12200))

Included in the following conference series:

Abstract

Heuristic evaluation (HE) is an inspection-based usability evaluation method in which a number of evaluators, typically 3–5, assess the usability of a system based on a set of usability guidelines. HE was first introduced by Nielsen and Molich and then revised by Nielsen. Since its introduction, HE has gained wide popularity among human-computer interaction (HCI) and user experience (UX) practitioners and is now one of the most common usability evaluation methods. A few years after the introduction of HE, some researchers realized that novice evaluators perform poorly on complex systems. This issue, known as the expertise effect, is one of the major challenges of HE; i.e., the more experienced the evaluator is, the better the results of the evaluation. Consequently, some researchers argue that the results of HE are the product of the evaluator’s experience rather than the method itself. To address this issue, we interviewed 15 usability experts, all of whom had at least four years of experience and who came from both academia and industry. We analyzed their responses and developed a step-by-step protocol.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Issa, T., Isaias, P.: Usability and human computer interaction (HCI). In: Issa, T., Isaias, P. (eds.) Sustainable Design: HCI, Usability and Environmental Concerns, pp. 19–36. Springer, London (2015). https://doi.org/10.1007/978-1-4471-6753-2_2

  2. Molich, R., Nielsen, J.: Improving a human-computer dialogue. Commun. ACM 33, 338–348 (1990). https://doi.org/10.1145/77481.77486

    Article  Google Scholar 

  3. Nielsen, J., Molich, R.: Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 249–256. Association for Computing Machinery, Seattle (1990). https://doi.org/10.1145/97243.97281

  4. Nielsen, J.: Enhancing the explanatory power of usability heuristics. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 152–158. Association for Computing Machinery, Boston (1994). https://doi.org/10.1145/191666.191729

  5. Rosenbaum, S., Rohn, J.A., Humburg, J.: A toolkit for strategic usability: results from workshops, panels, and surveys. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 337–344. Association for Computing Machinery, The Hague (2000). https://doi.org/10.1145/332040.332454

  6. Fernandez, A., Insfran, E., Abrahão, S.: Usability evaluation methods for the web: a systematic mapping study. Inf. Softw. Technol. 53, 789–817 (2011). https://doi.org/10.1016/j.infsof.2011.02.007

    Article  Google Scholar 

  7. Cockton, G., Woolrych, A.: Sale must end: should discount methods be cleared off HCI’s shelves? Interactions 9, 13–18 (2002). https://doi.org/10.1145/566981.566990

    Article  Google Scholar 

  8. Hertzum, M., Jacobsen, N.E.: The evaluator effect: a chilling fact about usability evaluation methods. Int. J. Hum. Comput. Interact. 13, 421–443 (2001). https://doi.org/10.1207/S15327590IJHC1304_05

    Article  Google Scholar 

  9. Nielsen, J.: Finding usability problems through heuristic evaluation. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 373–380. Association for Computing Machinery, Monterey (1992). https://doi.org/10.1145/142750.142834

  10. Al-Razgan, M.S., Al-Khalifa, H.S., Al-Shahrani, M.D.: Heuristics for evaluating the usability of mobile launchers for elderly people. In: Marcus, A. (ed.) DUXU 2014. LNCS, vol. 8517, pp. 415–424. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07668-3_40

  11. Paz, F., Paz, F.A., Pow-Sang, J.A.: Experimental case study of new usability heuristics. In: Marcus, A. (ed.) Design, User Experience, and Usability 2015: Design Discourse. LNCS, vol. 9186, pp. 212–223. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-20886-2_21

  12. de Salgado, A.L., Amaral, L.A., Freire, A.P., Fortes, R.P.M.: Usability and UX practices in small enterprises: lessons from a survey of the Brazilian context. In: Proceedings of the 34th ACM International Conference on the Design of Communication, pp. 1–9. Association for Computing Machinery, Silver Spring (2016). https://doi.org/10.1145/2987592.2987616

  13. Rajanen, D., Clemmensen, T., Iivari, N., Inal, Y., Rızvanoğlu, K., Sivaji, A., Roche, A.: UX professionals’ definitions of usability and UX – a comparison between Turkey, Finland, Denmark, France and Malaysia. In: Bernhaupt, R., Dalvi, G., Joshi, A.K., Balkrishan, D., O’Neill, J., Winckler, M. (eds.) INTERACT 2017. LNCS, vol. 10516, pp. 218–239. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68059-0_14

  14. Iivari, N.: Understanding the work of an HCI practitioner. In: Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles, pp. 185–194. Association for Computing Machinery, Oslo (2006). https://doi.org/10.1145/1182475.1182495

  15. Hussein, I., Mahmud, M., Tap, A.O.M.: A survey of user experience practice: a point of meet between academic and industry. In: 2014 3rd International Conference on User Science and Engineering (i-USEr), pp. 62–67. IEEE, Shah Alam (2014). https://doi.org/10.1109/iuser.2014.7002678

  16. Aydin, B., Palikhe, H., Beruvides, M.: The impact of usability on the cost of quality. In: Annual International Conference of the American Society for Engineering Management ASEM, pp. 632–637. ASEM, Virginia (2012)

    Google Scholar 

  17. Kushniruk, A.W., Triola, M.M., Borycki, E.M., Stein, B., Kannry, J.L.: Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int. J. Med. Inform. 74, 519–526 (2005). https://doi.org/10.1016/j.ijmedinf.2005.01.003

    Article  Google Scholar 

  18. Han, Y.Y., et al.: Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 116, 1506–1512 (2005). https://doi.org/10.1542/peds.2005-1287

    Article  Google Scholar 

  19. Sripathi, V., Sandru, V.: Effective usability testing–knowledge of user centered design is a key requirement. Int. J. Emerg. Technol. Adv. Eng. 3, 627–635 (2013)

    Google Scholar 

  20. Shneiderman, B.: Designing the User Interface: Strategies for Effective Human-Computer Interaction. Addison-Wesley Publishing Co., Reading (1987)

    Google Scholar 

  21. Tognazzini, B.: First Principles, HCI Design, Human Computer Interaction (HCI), Principles of HCI Design, Usability Testing. http://www.asktog.com/basics/firstPrinciples.html. Accessed 5 Mar 2014

  22. Gerhardt-Powals, J.: Cognitive engineering principles for enhancing human-computer performance. Int. J. Hum. Comput. Interact. 8, 189–211 (1996). https://doi.org/10.1080/10447319609526147

    Article  Google Scholar 

  23. Mandel, T.: The golden rules of user interface design. In: Mandel, T. (ed.) The Elements of User Interface Design, pp. 1–28. Wiley, Hoboken (1997)

    Google Scholar 

  24. Granollers, T.: Usability evaluation with heuristics. New proposal from integrating two trusted sources. In: Marcus, A., Wang, W. (eds.) DUXU 2018. LNCS, vol. 10918, pp. 396–405. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91797-9_28

  25. Hvannberg, E.T., Law, E.L.-C., Lárusdóttir, M.K.: Heuristic evaluation: comparing ways of finding and reporting usability problems. Interact. Comput. 19, 225–240 (2007). https://doi.org/10.1016/j.intcom.2006.10.001

    Article  Google Scholar 

  26. Inostroza, R., Rusu, C., Roncagliolo, S., Rusu, V.: Usability heuristics for touchscreen-based mobile devices: update. In: Proceedings of the 2013 Chilean Conference on Human - Computer Interaction, pp. 24–29. Association for Computing Machinery, Temuco, Chile (2013). https://doi.org/10.1145/2535597.2535602

  27. Chanco, C., Moquillaza, A., Paz, F.: Development and validation of usability heuristics for evaluation of interfaces in ATMs. In: Marcus, A., Wang, W. (eds.) HCII 2019. LNCS, vol. 11586, pp. 3–18. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23535-2_1

  28. Saavedra, M.-J., Rusu, C., Quiñones, D., Roncagliolo, S.: A set of usability and user experience heuristics for social networks. In: Meiselwitz, G. (ed.) HCII 2019. LNCS, vol. 11578, pp. 128–139. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-21902-4_10

  29. Ariffin, S.A., Dyson, L.E.: Culturally appropriate design of mobile learning applications in the Malaysian context. In: Rau, P. (eds.) CCD 2015. LNCS, vol. 9181, pp. 3–14. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-20934-0_1

  30. Cronholm, S.: The usability of usability guidelines: a proposal for meta-guidelines. In: Proceedings of the 21st Annual Conference of the Australian Computer-Human Interaction Special Interest Group: Design: Open 24/7, pp. 233–240. Association for Computing Machinery, Melbourne (2009). https://doi.org/10.1145/1738826.1738864

  31. Friess, E.: Personas in heuristic evaluation: an exploratory study. IEEE Trans. Prof. Commun. 58, 176–191 (2015). https://doi.org/10.1109/TPC.2015.2429971

    Article  Google Scholar 

  32. Po, S., Howard, S., Vetere, F., Skov, M.B.: Heuristic evaluation and mobile usability: bridging the realism gap. In: Brewster, S., Dunlop, M. (eds.) MobileHCI 2004. LNCS, vol. 3160, pp. 49–60. Springer, Berlin (2004). https://doi.org/10.1007/978-3-540-28637-0_5

  33. Varsaluoma, J.: Scenarios in the heuristic evaluation of mobile devices: emphasizing the context of use. In: Kurosu, M. (eds.) HCD 2009. LNCS, vol. 5619, pp. 332–341. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02806-9_38

  34. Muller, M.J., Matheson, L., Page, C., Gallup, R.: Methods & tools: participatory heuristic evaluation. Interactions 5, 13–18 (1998). https://doi.org/10.1145/285213.285219

    Article  Google Scholar 

  35. Sivaji, A., Abdullah, M.R., Downe, A.G., Ahmad, W.F.W.: Hybrid usability methodology: integrating heuristic evaluation with laboratory testing across the software development lifecycle. In: 2013 10th International Conference on Information Technology: New Generations, pp. 375–383. IEEE, Las Vegas (2013). https://doi.org/10.1109/itng.2013.60

  36. Kurosu, M., Matsuura, S., Sugizaki, M.: Categorical inspection method-structured heuristic evaluation (sHEM). In: 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation, vol. 2613, pp. 2613–2618. IEEE, Orlando (1997). https://doi.org/10.1109/icsmc.1997.635329

  37. Kurosu, M., Sugizaki, M., Matsuura, S.: A comparative study of sHEM (structured heuristic evaluation method). In: Proceedings of HCI International (The 8th International Conference on Human-Computer Interaction) on Human-Computer Interaction: Ergonomics and User Interfaces-Volume I, pp. 938–942. L. Erlbaum Associates Inc., Mahwah (1999)

    Google Scholar 

  38. Cockton, G., Woolrych, A., Hall, L., Hindmarch, M.: Changing analysts’ tunes: the surprising impact of a new instrument for usability inspection method assessment. In: O’Neill, E., Palanque, P., Johnson, P. (eds.) People and Computers XVII—Designing for Society, pp. 145–161. Springer, London (2004). https://doi.org/10.1007/978-1-4471-3754-2_9

  39. Slavkovic, A., Cross, K.: Novice heuristic evaluations of a complex interface. In: CHI 1999 Extended Abstracts on Human Factors in Computing Systems, pp. 304–305. Association for Computing Machinery, Pittsburgh (1999). https://doi.org/10.1145/632716.632902

  40. Botella, F., Rusu, C., Rusu, V., Quiñones, D.: How novel evaluators perceive their first heuristic evaluation. In: Proceedings of the XIX International Conference on Human Computer Interaction, pp. 1–4. Association for Computing Machinery, Palma (2018). https://doi.org/10.1145/3233824.3233835

  41. Rusu, C., Botella, F., Rusu, V., Roncagliolo, S., Quiñones, D.: An online travel agency comparative study: heuristic evaluators perception. In: Meiselwitz, G. (eds.) SCSM 2018. LNCS, vol. 10913, pp. 112–120. Springer, Cham, Switzerland (2018). https://doi.org/10.1007/978-3-319-91521-0_9

  42. de Salgado, A.L., de Fortes, R.P.M.: Heuristic evaluation for novice evaluators. In: Marcus, A. (ed.) DUXU 2016. LNCS, vol. 9746, pp. 387–398. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40409-7_37

  43. de Salgado, A.L., de Lara, S.M., Freire, A.P., de Fortes, R.P.M.: What is hidden in a heuristic evaluation: tactics from the experts. In: 13th International Conference on Information Systems & Technology Management - CONTECSI, pp. 2931–2946. CONTECSI, São Paulo (2016). https://doi.org/10.5748/9788599693124-13CONTECSI/PS-4068

  44. Petrie, H., Buykx, L.: Collaborative heuristic evaluation: improving the effectiveness of heuristic evaluation. In: Proceedings of UPA 2010 International Conference, Muinch, Germany (2010)

    Google Scholar 

  45. Salgado, A.D.: Adaptations to the heuristic evaluation (HE) method for novice evaluators. Master Dissertation. Universidade de São Paulo, São Paulo (2017)

    Google Scholar 

  46. Alqurni, J., Alroobaea, R., Alqahtani, M.: Effect of user sessions on the heuristic usability method. Int. J. Open Source Softw. Process. 9, 62–81 (2018). https://doi.org/10.4018/ijossp.2018010104

    Article  Google Scholar 

  47. Thomas, R.P., Lawrence, A.: Assessment of expert performance compared across professional domains. J. Appl. Res. Mem. Cogn. 7, 167–176 (2018). https://doi.org/10.1016/j.jarmac.2018.03.009

    Article  Google Scholar 

  48. Botella, F., Alarcon, E., Peñalver, A.: How to classify to experts in usability evaluation. In: Proceedings of the XV International Conference on Human Computer Interaction, p. Article 25. Association for Computing Machinery, Puerto de la Cruz (2014). https://doi.org/10.1145/2662253.2662278

  49. Ericsson, K.A., Prietula, M.J., Cokely, E.T.: The making of an expert. Harv. Bus. Rev. 85, 114–121, 193 (2007)

    Google Scholar 

  50. Hambrick, D.Z., Oswald, F.L., Altmann, E.M., Meinz, E.J., Gobet, F., Campitelli, G.: Deliberate practice: is that all it takes to become an expert? Intelligence 45, 34–45 (2014). https://doi.org/10.1016/j.intell.2013.04.001

    Article  Google Scholar 

  51. Guest, G., Bunce, A., Johnson, L.: How many interviews are enough?: an experiment with data saturation and variability. Field Methods 18, 59–82 (2006). https://doi.org/10.1177/1525822X05279903

    Article  Google Scholar 

  52. Norman, D.A.: The Design of Everyday Things: Revised and Expanded. Basic Books, New York (2013)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Computing and Digital Media, DePaul University, 243 South Wabash Avenue, Chicago, IL, 60604, USA

    Anas Abulfaraj & Adam Steele

Authors
  1. Anas Abulfaraj
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adam Steele
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anas Abulfaraj .

Editor information

Editors and Affiliations

  1. Aaron Marcus and Associates, Berkeley, CA, USA

    Aaron Marcus

  2. World Usability Day and Bentley User Experience Center, Newton Center, MA, USA

    Elizabeth Rosenzweig

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Abulfaraj, A., Steele, A. (2020). Coherent Heuristic Evaluation (CoHE): Toward Increasing the Effectiveness of Heuristic Evaluation for Novice Evaluators. In: Marcus, A., Rosenzweig, E. (eds) Design, User Experience, and Usability. Interaction Design. HCII 2020. Lecture Notes in Computer Science(), vol 12200. Springer, Cham. https://doi.org/10.1007/978-3-030-49713-2_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-49713-2_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-49712-5

  • Online ISBN: 978-3-030-49713-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation