Usability testing of mobile ICT for clinical settings: Methodological and practical challenges

Abstract

Background

While much is known about how to do usability testing of stationary Electronic Patient Record (EPR) systems, less is known about how to do usability testing of mobile ICT systems intended for use in clinical settings.

Aim

Our aim is to provide a set of empirically based recommendations for usability testing of mobile ICT for clinical work.

Method

We have conducted usability tests of two mobile EPR systems. Both tests have been done in full-scale models of hospital settings, and with multiple users simultaneously. We report here on the methodological aspects of these tests.

Results

We found that the usability of the mobile EPR systems to a large extent were determined by factors that went beyond that of the graphical user interface. These factors include ergonomic aspects such as the ability to have both hands free, and social aspects such as to what extent the systems disturbs the face-to-face interaction between the health worker and the patient.

Conclusions

To be able to measure usability issues that go beyond what can be found by a traditional stationary user interface evaluation, it is necessary to conduct usability tests of mobile EPR systems in physical environments that simulate the conditions of the work situation at a high level of realism. It is further in most cases necessary to test with a number of test subjects simultaneously.

Introduction

Most Electronic Patient Record (EPR) systems currently run only on stationary computers, while empirical studies of clinical work in hospitals show that health workers are constantly on the move in a highly event-driven working environment [1]. Clinical work is information and communication intensive and highly mobile [2]. EPR content is currently to a large extent produced and utilized in point-of-care settings away from the computers through the use of paper printouts, handwritten notes, and voice memos; while actual interaction with the EPR is done while sitting down at a stationary computer. This creates an obvious potential for mobile computing in healthcare.

To best support health workers in their everyday work, the hospital's EPR system should allow for interaction with the patient's medical information at the point of care. A number of studies of existing systems have documented the benefits of mobile computing in health care [3], [4], and other studies indicate additional benefits from the use of context information such as the health worker's location and electronic patient identification [5], [6], [7].

Moving the user interfaces of EPR systems on to mobile devices creates new challenges for system design and usability evaluation. Since its infancy at Xerox Parc in the late 1970s [8], usability testing of information systems has matured to an established practice in the software industry, with an ISO-defined common industry format for reporting test results [9]. Up until recently, most software products being tested were desktop based, i.e. single-user software running on a desktop computer with input through a keyboard and a mouse. This situation is now changing as more software is being produced for mobile devices such as mobile phones and PDAs. This creates new methodological and technological challenges.

From a usability perspective, the main difference between desktop-based and mobile computing is related to the use situation. The prototypical use situation for desktop-based applications is one-user sitting on a chair in front of a table looking at a screen with his or her hands on the keyboard and the mouse. Mobile technology, on the other hand, is to a much larger degree embedded into the user's web of physical and social life. Dourish [10] uses the concept of embodied interaction when referring to this. Embodied interaction, as argued by Dourish, is characterized by presence and participation in the world. As such, interaction with mobile technology is not a foreground activity to the same extent as interaction with desktop-based systems, but switches between being at the foreground of the user's attention and residing silently in the background.

The hospital as a work environment makes usability evaluations even harder, as compared to for example everyday use of mobile phones. Mobile ICT in healthcare is often integrated with a number of other ICT systems, serves a number of different user groups, and must allow for use in a number of different physical environments. Usability testing of mobile technology in healthcare consequently requires new ways of designing and doing the tests, new ways of recording user and system behavior, and new ways of analyzing the test data.

In the present paper we will address some of the methodological and practical challenges related to usability testing of mobile ICT for healthcare. This will be done by summing up our experience from two usability evaluation projects of mobile EPR done in a full-scale model of a hospital ward.

We have posed two research questions. (1) What classes of usability problems should a usability test of mobile ICT for clinical settings be able to identify? (2) What are the consequences concerning test methodology, lab setup and recording equipment? We will answer the first question by analyzing the usability issues that emerged in the two projects. The next question will be answered by analyzing what aspects of our existing test methodology, lab setup and recording equipment that contributed to the identification of these usability issues. Based on this, we will give some general recommendations for usability testing of mobile ICT for clinical settings.

We are aware of the limitations given by the low number of projects, and will discuss the threats to validity that this poses.