Usability testing of mobile ICT for clinical settings: Methodological and practical challenges
Introduction
Most Electronic Patient Record (EPR) systems currently run only on stationary computers, while empirical studies of clinical work in hospitals show that health workers are constantly on the move in a highly event-driven working environment [1]. Clinical work is information and communication intensive and highly mobile [2]. EPR content is currently to a large extent produced and utilized in point-of-care settings away from the computers through the use of paper printouts, handwritten notes, and voice memos; while actual interaction with the EPR is done while sitting down at a stationary computer. This creates an obvious potential for mobile computing in healthcare.
To best support health workers in their everyday work, the hospital's EPR system should allow for interaction with the patient's medical information at the point of care. A number of studies of existing systems have documented the benefits of mobile computing in health care [3], [4], and other studies indicate additional benefits from the use of context information such as the health worker's location and electronic patient identification [5], [6], [7].
Moving the user interfaces of EPR systems on to mobile devices creates new challenges for system design and usability evaluation. Since its infancy at Xerox Parc in the late 1970s [8], usability testing of information systems has matured to an established practice in the software industry, with an ISO-defined common industry format for reporting test results [9]. Up until recently, most software products being tested were desktop based, i.e. single-user software running on a desktop computer with input through a keyboard and a mouse. This situation is now changing as more software is being produced for mobile devices such as mobile phones and PDAs. This creates new methodological and technological challenges.
From a usability perspective, the main difference between desktop-based and mobile computing is related to the use situation. The prototypical use situation for desktop-based applications is one-user sitting on a chair in front of a table looking at a screen with his or her hands on the keyboard and the mouse. Mobile technology, on the other hand, is to a much larger degree embedded into the user's web of physical and social life. Dourish [10] uses the concept of embodied interaction when referring to this. Embodied interaction, as argued by Dourish, is characterized by presence and participation in the world. As such, interaction with mobile technology is not a foreground activity to the same extent as interaction with desktop-based systems, but switches between being at the foreground of the user's attention and residing silently in the background.
The hospital as a work environment makes usability evaluations even harder, as compared to for example everyday use of mobile phones. Mobile ICT in healthcare is often integrated with a number of other ICT systems, serves a number of different user groups, and must allow for use in a number of different physical environments. Usability testing of mobile technology in healthcare consequently requires new ways of designing and doing the tests, new ways of recording user and system behavior, and new ways of analyzing the test data.
In the present paper we will address some of the methodological and practical challenges related to usability testing of mobile ICT for healthcare. This will be done by summing up our experience from two usability evaluation projects of mobile EPR done in a full-scale model of a hospital ward.
We have posed two research questions. (1) What classes of usability problems should a usability test of mobile ICT for clinical settings be able to identify? (2) What are the consequences concerning test methodology, lab setup and recording equipment? We will answer the first question by analyzing the usability issues that emerged in the two projects. The next question will be answered by analyzing what aspects of our existing test methodology, lab setup and recording equipment that contributed to the identification of these usability issues. Based on this, we will give some general recommendations for usability testing of mobile ICT for clinical settings.
We are aware of the limitations given by the low number of projects, and will discuss the threats to validity that this poses.
Section snippets
Mobile technology defined
There is at present no consensus on a definition of mobile technology. In [11], Weilenmann does a review of the literature on mobile usability and ends with a fairly open definition of mobile technology: “…a technology which is designed to be mobile” (p. 24). For the purpose of the present analysis we prefer a more precise definition. We define mobile technology as technology that provides digital information and communication services to users on the move either through devices that are
A usability laboratory for mobile ICT in medical settings
As part of a national research initiative on health informatics in Norway (NSEP), we got funding to build a usability laboratory for evaluation of mobile applications in the health domain. Being aware of the drawbacks of traditional desktop-based usability tests for mobile technology, we started out by conducting a comparative usability evaluation to verify the results of Kjeldskov et al. [17]. The study [19] verified their results and motivated the construction of a laboratory that allows for
The two experiments
We will report here from two usability evaluations done in the usability laboratory by the authors. Both evaluations were controlled experiments exploring the potential for mobile and ubiquitous computing in the hospital. The aim of the two studies was to compare specific technological solutions. The results from the comparison tests have been reported elsewhere [22], [23], while the consequences for test methodology were not discussed. We will here summarize the lessons learned from the two
Factors that affect the usability of mobile EPR
A number of factors that affected the overall usability were identified in the two experiments. We have grouped them into three large classes: GUI usability, physical and bodily aspects of usability, and social aspects of usability.
Consequences for usability testing of mobile EPR
Based on the identified factors that affect the usability of mobile EPR, we will present a set of recommendations concerning usability testing of such systems. These recommendations come in addition to accepted best practice for usability testing and reporting as defined in the ISO/CIF document [9]. For all usability testing it is important to identify the right user group(s), make tasks that are realistic, and create a physical and social test environment that mimics that of the intended use
Discussion
The analysis and recommendations in this study are based on a limited number of tests with a limited number of test subjects. In addition, the experiments were done with very simple prototypes in simplified use scenarios.
The experiments have allowed us to identify some usability issues for mobile EPR, but our findings should not be seen as an attempt at making a complete list of such issues. More studies of mobile EPR are necessary to get a more complete picture of the usability challenges for
Conclusion
Clinical work in hospitals is information and communication intensive and highly mobile. Health workers are constantly on the move in a highly event-driven working environment. Most current Electronic Patient Record (EPR) systems only allow for access on stationary computers, while future systems also will allow for access on mobile devices at the point of care. While much is known about how to do usability testing of stationary EPR systems, less is known about how to do usability testing of
Acknowledgements
We would like to thank Arild Faxvaag, Øystein Nytrø, Terje Røsand and Reidar Martin Svendsen for help of various kinds in the projects. The research was funded by the Norwegian University of Science and Technology and the Norwegian Research Council.
References (27)
- et al.
Context awareness in health care: a review
International Journal of Medical Informatics
(2007) - et al.
Moving to get ahead: local mobility and collaboratory work
Proceedings of the 5th European Conference on Computer Supported Cooperative Work (ECSCW 2003)
(2003) - et al.
Communication behaviours in a hospital setting: an observational study
British Medical Journal
(1998) - et al.
Supporting work activities in healthcare by mobile electronic patient records
- et al.
Handheld computing in medicine
Journal of the American Medical Informatics Association
(2003) - et al.
Context in care-requirements for mobile context-aware patient charts
Proc. MEDINFO
(2004) - et al.
Context-aware mobile communication in hospitals
Computer
(2003) - W.L. Bewley, T.L. Roberts, D. Schroit, W.L. Verplank, Human factors testing in the design of Xerox's 8010 “Star” office...
- ISO. ISO/IEC 25062:2006 Software Engineering-Software Product Quality Requirements and Evaluation (SQuaRE)-Common...
Where the Action Is: The Foundations of Embodied Interaction
(2001)
Cited by (52)
Inpatient satisfaction and usage patterns of personalized smart bedside station system for patient-centered service at a tertiary university hospital
2016, International Journal of Medical InformaticsThe value of Retrospective and Concurrent Think Aloud in formative usability testing of a physician data query tool
2015, Journal of Biomedical InformaticsCitation Excerpt :Formative usability studies provide means to improve on a system design by uncovering those interface design flaws that clinicians might encounter when interacting with a system in a clinical setting [6]. To perform a formative usability study, a broad spectrum of usability evaluation methods (UEMs) is available, and these methods are increasingly used in interactive healthcare application design and evaluation [7,8]. However, selection of a UEM in a specific healthcare setting is often limited by practicality, accessibility of required human resources, and time to perform the evaluation study.
Determinants of e-Learning user satisfaction at a South African University of Technology
2022, 2022 IEEE 28th International Conference on Engineering, Technology and Innovation, ICE/ITMC 2022 and 31st International Association for Management of Technology, IAMOT 2022 Joint Conference - ProceedingsUnintended consequences of technology-enabled work activities experienced by healthcare professionals in tertiary hospitals of sub-Saharan Africa
2022, African Journal of Science, Technology, Innovation and DevelopmentUser-Centered Design of a Scenario-Based Serious Game: Game-Based Teaching of Future Healthcare
2020, IEEE Transactions on Games