Improving Hospital Outpatient Clinics Appointment Schedules by Prediction Models

Abstract

Patient no-shows and suboptimal patient appointment length scheduling reduce clinical efficiency and impair the clinic’s quality of service. The main objective of this study is to improve appointment scheduling in hospital outpatient clinics. We developed generic supervised machine learning models to predict patient no-shows and patient’s length of appointment (LOA). We performed a retrospective study using more than 100,000 records of patient appointments in a hospital outpatient clinic. Several machine learning algorithms were used for the development of our prediction models. We trained our models on a dataset that contained patients’, physicians’, and appointments’ characteristics. Our feature set combines both unstudied features and features adopted from previous studies. In addition, we identified the influential features for predicting LOA and no-show. Our LOA model’s performance was 6.92 in terms of MAE, and our no-show model’s performance was 92.1% in terms of F-score. We compared our models’ performance to the performance of previous research models by applying their methods to our dataset; our models demonstrated better performance. We show that the major effector of such differences is the use of our novel features. To evaluate the effect of our prediction results on the quality of schedules produced by appointment systems (AS), we developed an interface layer between our prediction models and the AS, where prediction results comprise the AS input. Using our prediction models, there was an 80% improvement in the daily cumulative patient waiting time and a 33% reduction in the daily cumulative physician idle time.

Appendices

Appendix A

F-score is the weighted harmonic mean of a model’s precision and recall. It is used to measure the performance of a model and is calculated as follows:

$${F}_{1}=\left(\frac{2}{{\mathrm{recall}}^{-1}+{\mathrm{precision}}^{-1}}\right)=2\cdot \frac{\mathrm{precision}\cdot \mathrm{recall}}{\mathrm{precision}+\mathrm{recall}}.$$

$$\mathrm{Precision}=\frac{tp}{tp+fp}$$

$$\mathrm{Recall}=\frac{tp}{tp+fn}$$

where tp is the number of true positive predictions, and fn and fp are the number of false negative and false positive predictions, respectively.

The Brier Score measures the accuracy of probabilistic predictions and is calculated as follows:

$$\mathrm{BR}=\frac{1}{\mathrm{n}}\sum_{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{y}}_{\mathrm{i}}-{\widehat{\mathrm{y}}}_{\mathrm{i}}\right)}^{2}$$

where y is the true value, i.e., 1 or 0, ŷ is the is the predicted probability and n is the number of records in the evaluation set.

A ROC curve plots precision vs. recall at different classification thresholds and AUC measures the entire two-dimensional area under the curve. AUC and F-score metrics range from 0 to 1. A higher score indicates better performance and is, therefore, preferred.

Accuracy is a metric for evaluating classification models, it as follows:

$$\mathrm{Accuracy}=\frac{tp+tn}{tp+tn+fp+fn}$$

MAE measures the average absolute magnitude of the errors in a set of predictions. RMSE is the average of the quadratic magnitude of the error.

$$\mathrm{RMSE}=\frac{1}{\mathrm{n}}\sum_{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{y}}_{\mathrm{i}}-{\widehat{\mathrm{y}}}_{\mathrm{i}}\right)}^{2}$$

$$\mathrm{MAE}=\frac{1}{\mathrm{n}}\sum_{\mathrm{i}=1}^{\mathrm{n}}|{\mathrm{y}}_{\mathrm{i}}-{\widehat{\mathrm{y}}}_{\mathrm{i}}|$$

where y is the true value of the dependent variable in our LOA, ŷ is the predicted value and n is the number of records in the evaluation set. MAE and RMSE metrics range from 0 to infinity. A lower score indicates better performance and is, therefore, preferred.

Appendix B

We have calculated the LOAM’s uncertainty as follows. First, for each record in the test set, we used the model to predict a list of probabilities from each tree in the LightGBM. Then, for each record, we calculated the STD from the list of probabilities. Finally, we calculated the mean of the STDs. Following this flow, the derived uncertainty of the model was 1.35 min.

Appendix C

The outputs of running the LightGBM Internal Feature Importance Algorithm (LGFIA) on LOAM and NSM are shown in Figs. 1 and 2, respectively. The features are presented in decreasing order of importance. In Figs. 7 and 8, the number at the end of each bar indicates the split value, i.e., the number of times the feature is used when the model is trained. In Figs. 9 and 10, the number at the end of each bar indicates the average gain of the feature when the model is trained where the gain is the reduction in training loss that results from adding a split point, i.e., feature.

Fig. 7

LOAM – LGFIA – Split method

Fig. 8

NSM – LightGBM feature importance – Split method

Fig. 9

LOAM – LGFIA – Gain method

Fig. 10

NSM – LightGBM feature importance – Gain method