User-centered design techniques for a computerised antibiotic decision support system in an intensive care unit

doi:10.1016/j.ijmedinf.2006.07.011

International Journal of Medical Informatics

Volume 76, Issue 10, October 2007, Pages 760-768

https://doi.org/10.1016/j.ijmedinf.2006.07.011 Get rights and content

Abstract

Objective

To explore the use of user-centered design techniques for developing the requirements for an antibiotic decision support system (DSS) in an intensive care unit (ICU).

Design and methodology

The setting was a 21-bed mixed medical/surgical adult ICU. This was an observational study with unstructured interviews and participatory design process. Models were constructed to demonstrate cultural, workflow, sequence/trigger events and other artefacts used to support antibiotic prescribing in the ICU. Using participatory design, a paper prototype was developed and case studies were used to simulate antibiotic prescribing for bacterial isolates. This information was used to design and pilot the decision support tool.

Results

The key users were identified as residents, registrars and the unit pharmacist. They identified the major requirements: ability to collate and print microbiology results, and to provide education and antibiotic advice for isolates. The final product was a real time microbiology browser and decision support tool for antibiotic prescribing (ADVISE). Uptake of the system was rapid with over 6000 encounters in the first 6 months. An audit of antibiotic use performed on all consecutive patients 6 months before and after introducing the DSS demonstrated a reduction in total and broad-spectrum antibiotics.

Conclusion

Contextual design methodology in conjunction with participatory design was an effective method to design this antibiotic decision support tool. The process facilitated physician and pharmacist ownership of the system that resulted in immediate uptake and ongoing use.

Section snippets

Background

Antibiotic prescribing for the critically ill patient is a cognitively demanding task [1]. Patients generate dozens of microbiology, pathology and radiology results that need to be processed by the treating doctor. In many cases the decision to start antibiotic therapy is based on clinical suspicion of infection so that the clinician must use appropriate diagnostic criteria, consider the likely pathogen, local patterns of common bacteria and antibiotic resistance. In the presence of an isolate

Methods

This study was performed in a 21-bed mixed medical/surgical intensive care unit of the Royal Melbourne Hospital, a tertiary referral and teaching hospital in Melbourne, Victoria. This ICU has 2000 admissions per year, with a median length of stay of 2 days. The unit admitted patients from both general and specialist medical and surgical units including nephrology, cardiothoracic surgery and bone marrow transplants. The hospital did not have electronic medical records or computerised prescribing

Flow and cultural model

The flow (Fig. 1) identified the resident and registrar as the key personnel responsible for collating and interpreting microbiology results, identifying the need for antibiotic therapy and feeding this information back to the senior ICU physician on call. Therefore, the key users would primarily be registrars and residents, and accordingly the content and functionality of the DSS should target their level of knowledge. The resident and registrars demonstrated a wide range of experience—from

Discussion

We have illustrated the use of user-centred design methodologies for the development of a computerised decision support system to assist with antibiotic prescribing in the complex environment of an intensive care unit.

The clinician researchers working with the developers found the Contextual Inquiry methodology very useful for identifying the major issues facing the design team. First and foremost it was possible to characterise the users real needs and the clinical circumstances in which it

Acknowledgments

Many thanks to the staff of the Royal Melbourne Hospital Intensive Care Unit and the Victorian Infectious Diseases service who participated in this study, and to Dr. Julian Kelly and Tabish Zaidi for reviewing this manuscript.

Financial support the development of the software and its evaluation was provided in part by a grant from the Quality Branch, Victorian Department of Human Services, Australia. The funding agreement ensured the authors’ independence in designing the study, interpreting the

References (34)

B. Kaplan
Evaluating informatics applications-some alternative approaches: theory, social interactionism, and call for methodological pluralism
Int. J. Med. Inform.
(2001)
D.W. Bates
Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality
J. Am. Med. Inform Assoc.
(2003)
J. Nielsen
Estimating the number of subjects needed for a thinking aloud test
Int. J. Hum. Comput. Stud.
(1994)
A. Blandford et al.
Situation awareness in emergency medical dispatch
Int. J. Hum. Comput. Stud.
(2004)
A. Kushniruk et al.
Analysis of medical decision making: a cognitive perspective on medical informatics
D.L. Hunt
Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review
JAMA
(1998)
S.L. Pestotnik
Implementing antibiotic practice guidelines through computer-assisted decision support: clinical and financial outcomes
Ann. Intern. Med.
(1996)
R.S. Evans
A computer-assisted management program for antibiotics and other antiinfective agents
N. Engl. J. Med.
(1998)
R.A. Larsen
Improved perioperative antibiotic use and reduced surgical wound infections through use of computer decision analysis
Infect. Control. Hosp. Epidemiol.
(1989)
Clinical Advisory Board, Computerised physician order entry: lessons from pioneering institutions, in: Best Practice...

L. Barenfanger et al.

Improved antimicrobial interventions have benefits

J. Clin. Microbiol.

(2001)

S. Grau

Monitoring of antimicrobial therapy by an integrated computer program

Pharm. World Sci.

(1999)

E.T. Jozefiak et al.

Computer-assisted antimicrobial surveillance in a community teaching hospital

Am. J. Health Syst. Pharm.

(1995)

M.T. Inaraja

Computer-assisted antimicrobial-use monitoring

Am. J. Hosp. Pharm.

(1986)

M.J. Richards

Impact of a web-based antimicrobial approval system on broad-spectrum cephalosporin use at a teaching hospital

Med. J. Aust.

(2003)

M.G. Olson et al.

Human-computer interaction: psychological aspects of the human use of computing

Annu. Rev. Psychol.

(2003)

C.H. Cheng

The effects of CPOE on ICU workflow: an observational study

AMIA Annu. Symp. Proc.

(2003)

Cited by (67)

The role of Design Science Research Methodology in developing pharmacy eHealth services
2021, Research in Social and Administrative Pharmacy
The increasing focus on efficiency of care has prompted health systems to look for innovative solutions that yield maximum value for care. Integration of care and eHealth are seen as the most promising solutions in the high technological environment of the coming decades. Pharmacy services have been developed to a point where the face of community pharmacy has changed dramatically. Thus, service design and implementation has become an area of increased attention by Pharmacy practice researchers, and the arrival of eHealth concepts to Pharmacy is driving the need to find alternative ways to design new services. Moreover, known barriers and challenges remain when securing integration of pharmacy services with the health systems. In this paper, Design Science Research Methodology (DSRM), a user-centered alternative methodology to the design, development and implementation of health services, especially eHealth services, is presented. This alternative, originally from the Information Sciences field, has been adopted as a service design methodology in diverse settings, including health care. Here, case studies are used to explain how a DSRM process should be conducted in a health care setting, illustrating what methods to choose in each step of the process. Finally, the advantages of DSRM compared to other user-centered methodologies for service design are presented, hopefully prompting the discussion on the use of DSRM for the study of the implementation and sustainability of pharmacy services.
Human-centred design processes for clinical decision support: A pulmonary embolism case study
2020, International Journal of Medical Informatics
Clinical Decision Support Systems (CDSS) can make patient care more efficient, cost-effective, and guideline-concordant. Many are created by clinicians who understand the challenges, but may publish concepts before considering subtle but important design details. Human-Centred Design (HCD) approaches provide necessary methods ensuring solid CDSS design. This article highlights HCD approaches in a pulmonary embolism CDSS case study context.
This pulmonary embolism CDSS results from collaborative work between computer science, psychology, and medicine. HCD methods used include: evaluations of pre-clinical prototype recordings, iterative usability expert reviews with software refinement, formative usability testing, and (separately-published) clinical pilot study.
HCD methods were instrumental in iteratively creating an easy to use and functionally-sound CDSS. Retrospective evaluations revealed that participants spent considerable time on items that were out of order from natural cognitive diagnostic workflows. Features missing between original and study version were noted, confusing interface elements reworked, and currently-active decision tree branches were visually emphasized. From iterative usability reviews, positioning of information within the decision tree was radically reworked, information separated into levels of support for different user groups, and supportive versus directive language issues addressed. Formative studies identified issues such as interface adjustments and hospital workflow integration.
Human-centred design approaches provide methods for integrating the skills and knowledge of many disciplines, illustrated by example in this pulmonary embolism CDSS creation. Advantages of leveraging many design guidelines as well as revealing new design considerations that would otherwise have remained hidden are described. The findings reported here support future CDSS design through HCD inclusion.
Explainable decision support through the learning and visualization of preferences from a formal ontology of antibiotic treatments
2020, Journal of Biomedical Informatics
The aim of eXplainable Artificial Intelligence (XAI) is to design intelligent systems that can explain their predictions or recommendations to humans. Such systems are particularly desirable for therapeutic decision support, because physicians need to understand rcommendations to have confidence in their application and to adapt them if required, e.g. in case of patient contraindication. We propose here an explainable and visual approach for decision support in antibiotic treatment, based on an ontology. There were three steps to our method. We first generated a tabular dataset from the ontology, containing features defined on various domains and n-ary features. A preference model was then learned from patient profiles, antibiotic features and expert recommendations found in clinical practice guidelines. This model made the implicit rationale of the expert explicit, including the way in which missing data was treated. We then visualized the preference model and its application to all antibiotics available on the market for a given clinical situation, using rainbow boxes, a recently developed technique for set visualization. The resulting preference model had an error rate of 3.5% on the learning data, and 5.2% on test data (10-fold validation). These findings suggest that our system can help physicians to prescribe antibiotics correctly, even for clinical situations not present in the guidelines (e.g. due to allergies or contraindications for the recommended treatment).
PARS, a system combining semantic technologies with multiple criteria decision aiding for supporting antibiotic prescriptions
2019, Journal of Biomedical Informatics
Citation Excerpt :
Unless the many stakeholders do not act in a coordinated manner urgently, the world is moving towards a post-antibiotic era, where current and minor injury infections who were treated for decades could kill again [79]. On the upside, implementing antibiotic decision support systems has been shown to be a promising direction for reducing inappropriate antibiotic prescription and decreasing local resistance to antibiotics [38,107,82]. We now discuss five main categories of work relating to our problem statement, namely: Medical Guidelines, Rule-based systems, Semantic Technologies, Machine Learning and MCDA.
Motivated by the well documented worldwide spread of adverse drug events, as well as the increased danger of antibiotic resistance (caused mainly by inappropriate prescribing and overuse), we propose a novel recommendation system for antibiotic prescription (PARS).
Our approach is based on the combination of semantic technologies with MCDA (Multiple Criteria Decision Aiding) that allowed us to build a two level decision support model. Given a specific domain, the approach assesses the adequacy of an alternative/action (prescription of antibiotic) for a specific subject (patient) with an issue (bacterial infection) in a given context (medical). The goal of the first level of the decision support model is to select the set of alternatives which have the potential to be suitable. Then the second level sorts the alternatives into categories according to their adequacy using an MCDA sorting method (MR–Sort with Veto) and a structured set of description logic queries.
We applied this approach in the domain of antibiotic prescriptions, working closely with the EpiCura Hospital Center (BE). Its performance was compared to the EpiCura recommendation guidelines which are currently in use. The results showed that the proposed system is more consistent in its recommendations when compared with the static EpiCura guidelines. Moreover, with PARS the antibiotic prescribing workflow becomes more flexible. PARS allows the user (physician) to update incrementally and dynamically a patient’s profile with more information, or to input knowledge modifications that accommodate the decision context (like the introduction of new side effects and antibiotics, the development of germs that are resistant, etc). At the end of our evaluation, we detail a number of limitations of the current version of PARS and discuss future perspectives.
Critical care information display approaches and design frameworks: A systematic review and meta-analysis
2019, Journal of Biomedical Informatics: X
Citation Excerpt :
For each display, we documented whether any of these 5 innovations were represented by the display (see Table 1). Examples of displays that were designed to combine information from different sources to support more meaningful grouping of related information included: (1) displays that integrated a broad range of critical care data to be presented in a high resolution display (see Fig. 2 for an example) [46–55], (2) displays that integrated data from bedside devices and monitors (see Fig. 3) [56–61], (3) displays that integrated data for the purpose of antibiotic selection and management [62–64], and (4) displays that integrated information into a novel graphical presentation [65–68]. Display designers frequently described an intention to organize the information in more clinically meaningful ways, for example, by human body systems, medical concepts, or provider tasks.
To systematically review original user evaluations of patient information displays relevant to critical care and understand the impact of design frameworks and information presentation approaches on decision-making, efficiency, workload, and preferences of clinicians.
We included studies that evaluated information displays designed to support real-time care decisions in critical care or anesthesiology using simulated tasks. We searched PubMed and IEEExplore from 1/1/1990 to 6/30/2018. The search strategy was developed iteratively with calibration against known references. Inclusion screening was completed independently by two authors. Extraction of display features, design processes, and evaluation method was completed by one and verified by a second author.
Fifty-six manuscripts evaluating 32 critical care and 22 anesthesia displays were included. Primary outcome metrics included clinician accuracy and efficiency in recognizing, diagnosing, and treating problems. Implementing user-centered design (UCD) processes, especially iterative evaluation and redesign, resulted in positive impact in outcomes such as accuracy and efficiency. Innovative display approaches that led to improved human-system performance in critical care included: (1) improving the integration and organization of information, (2) improving the representation of trend information, and (3) implementing graphical approaches to make relationships between data visible.
Our review affirms the value of key principles of UCD. Improved information presentation can facilitate faster information interpretation and more accurate diagnoses and treatment. Improvements to information organization and support for rapid interpretation of time-based relationships between related quantitative data is warranted. Designers and developers are encouraged to involve users in formal iterative design and evaluation activities in the design of electronic health records (EHRs), clinical informatics applications, and clinical devices.
Design principles for simulation games for learning clinical reasoning: A design-based research approach
2018, Nurse Education Today
Citation Excerpt :
Previous research has highlighted a gap between game developers, learners, educators and curriculum designers and thus in this study, participatory methodologies were used (Winters and Mor, 2008; Wu et al., 2012) with an emphasis on involving end users throughout the game design process (see Abras et al., 2004). This enhances the likelihood of the product being used as intended (see de Vito Dabbs et al., 2009; Thursky and Mahemoff, 2007). The present study adopts the following definition of ‘game-based simulation’: a learning method that combines game elements, simulations, and learning objectives and focuses on game-based simulations delivered in web-based, mobile, or virtual-reality learning environments.
Nurses sometimes lack the competence needed for recognising deterioration in patient conditions and this is often due to poor clinical reasoning. There is a need to develop new possibilities for learning this crucial competence area. In addition, educators need to be future oriented; they need to be able to design and adopt new pedagogical innovations. The purpose of the study is to describe the development process and to generate principles for the design of nursing simulation games.
A design-based research methodology is applied in this study. Iterative cycles of analysis, design, development, testing and refinement were conducted via collaboration among researchers, educators, students, and game designers.
The study facilitated the generation of reusable design principles for simulation games to guide future designers when designing and developing simulation games for learning clinical reasoning.
This study makes a major contribution to research on simulation game development in the field of nursing education. The results of this study provide important insights into the significance of involving nurse educators in the design and development process of educational simulation games for the purpose of nursing education.

View all citing articles on Scopus

View full text

User-centered design techniques for a computerised antibiotic decision support system in an intensive care unit

Abstract

Objective

Design and methodology

Results

Conclusion

Section snippets

Background

Methods

Flow and cultural model

Discussion

Acknowledgments

Int. J. Med. Inform.

J. Am. Med. Inform Assoc.

Int. J. Hum. Comput. Stud.

Int. J. Hum. Comput. Stud.

Analysis of medical decision making: a cognitive perspective on medical informatics

Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review

JAMA

Implementing antibiotic practice guidelines through computer-assisted decision support: clinical and financial outcomes

Ann. Intern. Med.

A computer-assisted management program for antibiotics and other antiinfective agents

N. Engl. J. Med.

Improved perioperative antibiotic use and reduced surgical wound infections through use of computer decision analysis

Infect. Control. Hosp. Epidemiol.

Improved antimicrobial interventions have benefits

J. Clin. Microbiol.

Monitoring of antimicrobial therapy by an integrated computer program

Pharm. World Sci.

Computer-assisted antimicrobial surveillance in a community teaching hospital

Am. J. Health Syst. Pharm.

Computer-assisted antimicrobial-use monitoring

Am. J. Hosp. Pharm.

Impact of a web-based antimicrobial approval system on broad-spectrum cephalosporin use at a teaching hospital

Med. J. Aust.

Human-computer interaction: psychological aspects of the human use of computing

Annu. Rev. Psychol.

The effects of CPOE on ICU workflow: an observational study

AMIA Annu. Symp. Proc.