Skip to main content

A Coding Framework for Usability Evaluation of Digital Health Technologies

  • Conference paper
  • First Online:
Human-Computer Interaction. Theoretical Approaches and Design Methods (HCII 2022)

Abstract

Several studies have reported low adherence and high resistance from clinicians to adopt digital health technologies into clinical practice, particularly the use of computer-based clinical decision support systems. Poor usability and lack of integration with the clinical workflow have been identified as primary issues. Few guidelines exist on how to analyze the collected data associated with the usability of digital health technologies. In this study, we aimed to develop a coding framework for the systematic evaluation of users’ feedback generated during focus groups and interview sessions with clinicians, underpinned by fundamental usability principles and design components. This codebook also included a coding category to capture the user’s clinical role associated with each specific piece of feedback, providing a better understanding of role-specific challenges and perspectives, as well as the level of shared understanding across the multiple clinical roles. Furthermore, a voting system was created to quantitatively inform modifications of the digital system based on usability data. As a use case, we applied this method to an electronic cognitive aid designed to improve coordination and communication in the cardiac operating room, showing that this framework is feasible and useful not only to better understand suboptimal usability aspects, but also to recommend relevant modifications in the design and development of the system from different perspectives, including clinical, technical, and usability teams. The framework described herein may be applied in other highly complex clinical settings, in which digital health systems may play an important role in improving patient care and enhancing patient safety.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Trinkley, K.E., Blakeslee, W.W., Matlock, D.D., Kao, D.P., Van Matre, A.G., Harrison, R., et al.: Clinician preferences for computerised clinical decision support for medications in primary care: a focus group study. BMJ Health Care Inform. 26, e000015 (2019)

    Article  Google Scholar 

  2. Book, B.A.: Crossing the quality chasm: a new health system for the 21st century. BMJ 323, 1192 (2001). https://doi.org/10.1136/bmj.323.7322.1192

    Article  Google Scholar 

  3. Hessels, A., Flynn, L., Cimiotti, J.P., Bakken, S., Gershon, R.: Impact of heath information technology on the quality of patient care. Online J. Nurs. Inform. 19 (2015). https://www.ncbi.nlm.nih.gov/pubmed/27570443

  4. Furukawa, M.F., Eldridge, N., Wang, Y., Metersky, M.: Electronic health record adoption and rates of in-hospital adverse events. J. Patient Saf. 16, 137–142 (2020)

    Article  Google Scholar 

  5. Ebnali, M., Shah, M., Mazloumi, A.: How mHealth apps with higher usability effects on patients with Breast Cancer? In: Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, pp. 81–84 (2019). https://doi.org/10.1177/2327857919081018

  6. Guo, C., Ashrafian, H., Ghafur, S., Fontana, G., Gardner, C., Prime, M.: Challenges for the evaluation of digital health solutions—a call for innovative evidence generation approaches. NPJ Digital Med. 3, 1–14 (2020)

    Article  Google Scholar 

  7. Zenati, M.A., Kennedy-Metz, L., Dias, R.D.: Cognitive engineering to improve patient safety and outcomes in cardiothoracic surgery. Semin. Thorac. Cardiovasc. Surg. 32, 1–7 (2020)

    Article  Google Scholar 

  8. Brown, W., Yen, P.-Y., Rojas, M., Schnall, R.: Assessment of the health IT usability evaluation model (Health-ITUEM) for evaluating mobile health (mHealth) technology. J. Biomed. Inform. 46, 1080–1087 (2013). https://doi.org/10.1016/j.jbi.2013.08.001

    Article  Google Scholar 

  9. Leech, N.L., Onwuegbuzie, A.J.: An array of qualitative data analysis tools: a call for data analysis triangulation. School Psychol. Q. 22, 557–584 (2007). https://doi.org/10.1037/1045-3830.22.4.557

    Article  Google Scholar 

  10. Willis, G.: Pretesting of health survey questionnaires: cognitive interviewing, usability testing, and behavior coding. In: Health Survey Methods, pp. 217–242 (2014). https://doi.org/10.1002/9781118594629.ch9

  11. Nielsen, J.: Enhancing the explanatory power of usability heuristics. In: Conference Companion on Human Factors in Computing Systems - CHI 1994 (1994). https://doi.org/10.1145/259963.260333

  12. Mosier, J.N., Smith, S.L.: Application of guidelines for designing user interface software. Behav. Inf. Technol. 5, 39–46 (1986). https://doi.org/10.1080/01449298608914497

    Article  Google Scholar 

  13. Kontio, J., Lehtola, L., Bragge, J.: Using the focus group method in software engineering: obtaining practitioner and user experiences. In: Proceedings of 2004 International Symposium on Empirical Software Engineering, ISESE 2004. IEEE (2004). https://doi.org/10.1109/isese.2004.1334914

  14. Mandel, T.: The Elements of User Interface Design. Wiley, New York (1997)

    Google Scholar 

  15. Rind, A., Wang, T.D., Aigner, W., Miksch, S., Wongsuphasawat, K., Plaisant, C., et al.: Interactive information visualization to explore and query electronic health records. HCI 5, 207–298 (2013)

    Google Scholar 

  16. Avrunin, G.S., Christov, S.C., Clarke, L.A., Conboy, H.M., Osterweil, L.J., Zenati, M.A.: Process driven guidance for complex surgical procedures. AMIA Annu. Symp. Proc. 2018, 175–184 (2018)

    Google Scholar 

Download references

Acknowledgment

This work was supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health Under Award Number R01HL126896 (PI: Zenati). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. No conflicts of interest were declared.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Roger D. Dias .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ebnali, M. et al. (2022). A Coding Framework for Usability Evaluation of Digital Health Technologies. In: Kurosu, M. (eds) Human-Computer Interaction. Theoretical Approaches and Design Methods. HCII 2022. Lecture Notes in Computer Science, vol 13302. Springer, Cham. https://doi.org/10.1007/978-3-031-05311-5_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-05311-5_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-05310-8

  • Online ISBN: 978-3-031-05311-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics