Technical Notes
Integrating natural language processing expertise with patient safety event review committees to improve the analysis of medication events

https://doi.org/10.1016/j.ijmedinf.2017.05.005Get rights and content

Highlights

  • NLP models categorized medication related events into pharmacy delays, dispensing errors, Pyxis discrepancies, and prescriber errors.

  • This study highlights and discusses differences when incorporating brief factual descriptions and resolutions free-texts in NLP models.

  • We demonstrate the integration of NLP models and discuss the importance of having mechanisms for users to update the models over time.

Abstract

Objectives

Many healthcare providers have implemented patient safety event reporting systems to better understand and improve patient safety. Reviewing and analyzing these reports is often time consuming and resource intensive because of both the quantity of reports and length of free-text descriptions in the reports.

Methods

Natural language processing (NLP) experts collaborated with clinical experts on a patient safety committee to assist in the identification and analysis of medication related patient safety events. Different NLP algorithmic approaches were developed to identify four types of medication related patient safety events and the models were compared.

Results

Well performing NLP models were generated to categorize medication related events into pharmacy delivery delays, dispensing errors, Pyxis discrepancies, and prescriber errors with receiver operating characteristic areas under the curve of 0.96, 0.87, 0.96, and 0.81 respectively. We also found that modeling the brief without the resolution text generally improved model performance. These models were integrated into a dashboard visualization to support the patient safety committee review process.

Conclusions

We demonstrate the capabilities of various NLP models and the use of two text inclusion strategies at categorizing medication related patient safety events. The NLP models and visualization could be used to improve the efficiency of patient safety event data review and analysis.

Introduction

Adverse drug events are a leading cause of preventable patient harm [1], [2], [3]. In an effort to reduce patient harm events associated with medications many healthcare systems have implemented patient safety event reporting systems to better identify safety hazards associated with pharmacy and medication administration, as well as other types of events [4], [5]. The reporting systems generally provide a method for provider staff to submit a description of a safety hazard ranging from a near miss, where no patient harm occurred, to a serious safety event that resulted in patient harm.

Many patient safety event reporting systems contain hundreds to thousands of medication related events and have the potential to dramatically improve care and reduce adverse drug events [6], [7]. However, there are several challenges associated with the data from these reporting systems [8]. Often, the data are difficult to interpret and act on because of the large number of reports, amount of free-text, and variability in category assignment by reporters.

In order to utilize the patient safety event data more rigorously many hospitals have created review committees, composed of clinicians focused on safety and quality, to review each event, categorize them appropriately to better understand trends, and develop solutions once trends are recognized. The committee review of the events is an incredibly labor intensive process given the large volume of reports generated each week. This difficulty is compounded in large healthcare systems where data from multiple hospitals need to be efficiently analyzed to understand overall patterns and trends across the system. Each report can take several minutes to initially review and then additional time during the committee meeting to further discuss.

Our goal is to develop a more efficient and streamlined method for categorizing patient safety event reports based on modeling the free-text of event reports to reduce the review time of the committee. We describe a collaborative effort in which informatics and safety science experts joined a clinical safety committee to develop an algorithmic approach to more automatically review and categorize medication events. The intent is to eventually develop a computational system that can categorize events in near real-time, hence reducing the time for committee review and expediting the process of identifying meaningful trends that can then be acted on to reduce adverse drug events. There are three main contributions of this case report. First, we develop and evaluate the performance of different modeling techniques to categorize four medication safety issues. Second, we evaluate model performance of two text inclusion conditions. The first condition includes only the brief factual description from medication related event reports as provided by the frontline staff member entering the report. The second combines both the brief factual description and resolution text, which is a short description typically provided by a manager that has reviewed the event report. Lastly we deploy the best models in an interactive visualization which categorizes reports in near real-time and allows users to provide feedback to the algorithm allowing for continued model training.

Section snippets

Data elements in patient safety event reports

Patient safety event reporting systems are generally composed of structured and unstructured data [9], [10]. When entering a report, the frontline staff selects a general category from a predefined list of categories (e.g. medication, fall, surgery) and a specific event type category. The reporter then enters a free-text description (brief factual description) of the safety hazards which can vary in length. Lastly, reports can sometimes be accompanied with additional free-text about how the

Data sources

To train and validate our models, we started with 774 medication safety events that have been manually annotated and reviewed by the safety and quality committee (2 MDs, 1 PharmD, 3 RNs). Every report has a free-text brief factual description ranging from 9 to 424 words (77.9 mean, 59 median, 63.3 std). Six hundred ninety-five reports (90%) have resolution free-text averaging 50.6 words (29 median 29, 61.7 std) and were used for the model development efforts, Fig. 1. This study was approved by

Performance

Accuracy, precision, F1-score, sensitivity, specificity, and ROC AUC metrics for each approach are summarized in Table 2. ROC AUC was used to rank the relative performance of the models as this metrics combines both sensitivity and specificity. Pyxis Discrepancy and Pharmacy Delivery Delay SVM models had high and similar AUC results between the [B] and [B + R] conditions. While there are no previous studies to compare these prediction of medication workflow related patient safety events, the

Discussion

Integrating data analytic and safety science expertise with the clinical safety committee to streamline the analysis and categorization of patient safety events has led to promising results. The clinicians provided the data analytics experts with the necessary domain specific knowledge to develop NLP techniques to recategorize medication patient safety events into specific workflow related categories. The models can serve to dramatically reduce the time investment currently required by the

Conclusion

We evaluated different NLP modeling techniques and text inclusion strategies to categorize four specific medication workflow safety events. We demonstrated the predictive capabilities of these models while highlighting the cautionary benefits with using additional resolution text in the model. Lastly, the models were incorporated into an interactive visualization that provide users a way to directly update model results.

Contributorship statement

According to the definition given by the International Committee of Medical Journal Editors (ICMJE), Allan Fong qualifies for authorship including making substantial contributions to the intellectual content of conception and design, acquisition of data, and analysis and interpretation of data. Furthermore, Allan Fong has participated in drafting of the manuscript and critical revision of the manuscript for important intellectual content. Allan Fong is the corresponding author.

According to the

Statement on conflict of interest

The authors have no competing interests or conflicts of interest.

Funding statement

N/A.

Summary table

What was known

  • Extracting information from patient safety reports is challenging in large part due to the variability in reporting

  • NLP techniques can assist in understanding free-text

Added knowledge

  • Considering only the brief descriptions of patient safety reports was generally sufficient for developing reliable classification models

  • Integration of models into visualization requires mechanism for users to provide feedback to the models

Acknowledgements

We are very thankful to the entire review committee and the dedication of the frontline reporters working to make our hospital and systems safer.

References (20)

  • P.J. Pronovost et al.

    Toward learning from patient safety reporting systems

    J. Clin. Nurs.

    (2006)
  • A.W. Wu et al.

    ICU incident reporting systems

    J. Crit. Care

    (2002)
  • A. Fong et al.

    Exploring methods for identifying related patient safety events using structured and unstructured data

    J. Biomed. Inf.

    (2015)
  • D.W. Bates et al.

    Incidence and preventability of adverse drug events in hospitalized adults

    J. Gen. Intern. Med.

    (1993)
  • A. Malpass et al.

    A classification for adverse drug events

    J. Qual. Clin. Pract.

    (1999)
  • L.L. Leape et al.

    The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II

    N. Engl. J. Med.

    (1991)
  • P. Aspden et al.

    Patient safety reporting systems and applications

    Patient Safety: Achieving a New Standard of Care

    (2004)
  • J. Rosenthal et al.

    Maximizing the Use of State Adverse Event Data to Improve Patient Safety

    (2005)
  • D.R. Longo et al.

    The long road to patient safety: a status report on patient safety systems

    JAMA

    (2005)
  • J. White

    Adverse Event Reporting and Learning Systems: A Review of the Relevant Literature

    (2007)
There are more references available in the full text version of this article.

Cited by (26)

  • Looking to the future: Artificial intelligence in enhancing patient care

    2023, Deprescribing and Polypharmacy in an Aging Population
  • Natural Language Processing Combined with ICD-9-CM Codes as a Novel Method to Study the Epidemiology of Allergic Drug Reactions

    2020, Journal of Allergy and Clinical Immunology: In Practice
    Citation Excerpt :

    Although these methods can be easily applied to epidemiologic research, they can also be used to improve patient care by facilitating advanced queries to accurately determine patients' allergies. Prior studies have shown the strength of using NLP to efficiently study large patient populations and improve patient safety.15,18 The use of computer algorithms like the one we described could be modified to find all relevant information within the EHR on allergic drug reactions for that specific patient.

  • A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysis

    2019, International Journal of Medical Informatics
    Citation Excerpt :

    A summary of the data extraction form is presented in Fig. 2. The full citation list for studies included in the review is presented in the bibliography under references [7,11,36–45,28,46–55,29,56–60,30–35]. Thirty-five studies were included in this review.

  • Identification of Inpatient Allergic Drug Reactions Using ICD-9-CM Codes

    2019, Journal of Allergy and Clinical Immunology: In Practice
    Citation Excerpt :

    NLP has been used to identify patient cohorts with gastrointestinal pathologies such as hepatocellular carcinoma and cirrhosis.24,25 For ADRs, NLP models were recently shown to accurately categorize medication-related events to help improve the efficiency of patient safety event review.26 We are currently studying using billing codes with NLP to improve drug allergy case-finding.

View all citing articles on Scopus
View full text