Improving Prediction Models’ Propriety in Intensive-Care Unit, by Enforcing an Advance Notice Period

Abstract

Intensive-Care-Units (ICUs) are time-critical, and sufficient reaction time is crucial. Previous studies of systems for alerting life-threatening events in the ICU, suffer from “immediate” events bias. In this research, we present a new approach for outcome prediction in ICU admissions, which takes into consideration the constraint of an advance notice of a predicted outcome. We showcase the approach over mortality and sepsis-3 predictions and compare it to existing approaches. We’ve created a set of Neural Network models that implement and evaluate the existing and the suggested approaches using the MIMIC-III data. We show that the performance is affected significantly when enforcing a notice period for mortality prediction, but not affected for sepsis-3 prediction. Further, we examine whether models need to be trained for a specific notice period, or whether the approach could be incorporated at the evaluation level. We found that adding notice enforcement post-model training, has no significant performance loss compared to incorporating the notice period during training, within the bounds of the trained lookahead. The concept of adding Alert-Interval could be applied to other clinical scenarios, where having advance notice is essential.

Appendix A

We demonstrate here the difference between the Area Under the Curve (AUC) and Weighted AUC (WAUC) metrics used in the paper, using a simplified example. We define a negative window as period of time when the outcome of interest was not observed (e.g., the patient didn’t die during this window) and a positive window as a time-window when the outcome of interest was observed. Our example composed of 3 admitted patients and two potential models with the following attributes:

• Patient-0 has a single positive prediction-window (to enable ROC AUC calculation).

• Patient-1 has 4 negative prediction-windows.

• Patient-2 has 20 negative prediction-windows

• Both models are correct on Patient-0’s prediction.

• Model-1 is mistaken in 2 prediction-windows of patient-1 (50% mistake rate) and correct in the rest of the prediction-windows.

• Model-2 is mistaken in 2 of the prediction-windows for patient-2 (10% mistake rate) and correct in the rest of the prediction-windows.

• When calculating the AUC or WAUC, the models’ mistaken prediction-windows are given the highest prediction-score and the rest of the prediction-windows are given prediction-scores in a correct order according to their labels.

In such a case, the standard AUC metric will be indifferent regarding which model to choose (both AUC scores are 0.92), as number of mistaken prediction-windows are similar to both models and it doesn’t matter for which admission the mistakes were made. However, when weighting each admission equally (looking on the quality of prediction per admission), model-2 outperformed model-1 as its achieves an WAUC of 0.95 while model-1 gets only 0.75 (Fig. 3).

Fig. 3.

Illustration of the given example. The two models predict for the same patients and the same set of prediction windows (yellow rectangles). Correct classifications are marked with green ‘v’s and misclassifications with red ‘x’s. (Color figure online)