Skip to main content

Advertisement

SpringerLink
Log in

Prediction of Incident Delirium Using a Random Forest classifier

  • Systems-Level Quality Improvement
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

Delirium is a serious medical complication associated with poor outcomes. Given the complexity of the syndrome, prevention and early detection are critical in mitigating its effects. We used Confusion Assessment Method (CAM) screening and Electronic Health Record (EHR) data for 64,038 inpatient visits to train and test a model predicting delirium arising in hospital. Incident delirium was defined as the first instance of a positive CAM occurring at least 48 h into a hospital stay. A Random Forest machine learning algorithm was used with demographic data, comorbidities, medications, procedures, and physiological measures. The data set was randomly partitioned 80% / 20% for training and validating the predictive model, respectively. Of the 51,240 patients in the training set, 2774 (5.4%) experienced delirium during their hospital stay; and of the 12,798 patients in the validation set, 701 (5.5%) experienced delirium. Under-sampling of the delirium negative population was used to address the class imbalance. The Random Forest predictive model yielded an area under the receiver operating characteristic curve (ROC AUC) of 0.909 (95% CI 0.898 to 0.921). Important variables in the model included previously identified predisposing and precipitating risk factors. This machine learning approach displayed a high degree of accuracy and has the potential to provide a clinically useful predictive model for earlier intervention in those patients at greatest risk of developing delirium.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Confusion Assessment Method. © 1988, 2003, Hospital Elder Life Program. All rights reserved. Not to be reproduced without permission. Instructions for correct usage available at: http://www.hospitalelderlifeprogram.org

References

  1. Inouye, S. K., Westendorp, R. G. J., and Saczynski, J. S., Delirium in elderly people. Lancet 383:911–922, 2014. https://doi.org/10.1016/S0140-6736(13)60688-1.

    Article  PubMed  Google Scholar 

  2. Inouye, S. K., Rushing, J. T., Foreman, M. D. et al., Does delirium contribute to poor hospital outcomes?: A three-site epidemiologic study. J. Gen. Intern. Med. 13:234–242, 1998. https://doi.org/10.1046/j.1525-1497.1998.00073.x.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  3. Ely, E. W., Gautam, S., Margolin, R. et al., The impact of delirium in the intensive care unit on hospital length of stay. Intensive Care Med. 27:1892–1900, 2001. https://doi.org/10.1007/s00134-001-1132-2.

    Article  CAS  PubMed  Google Scholar 

  4. Witlox, J., Eurelings, L. S. M., De Jonghe, J. F. M. et al., Delirium in Elderly Patients and the Risk of Postdischarge Mortality. JAMA 304:443–451, 2010. https://doi.org/10.1001/jama.2010.1013.

    Article  CAS  PubMed  Google Scholar 

  5. Rudolph, J. L., and Marcantonio, E. R., Postoperative Delirium: Acute change with long-term implications. Anesth. Analg. 112:1202–1211, 2011. https://doi.org/10.1213/ANE.0b013e3182147f6d.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Girard, T. D., Jackson, J. C., Pandharipande, P. P. et al., Delirium as a predictor of long-term cognitive impairment in survivors of critical illness. Crit. Care Med. 38:1513–1520, 2010. https://doi.org/10.1097/CCM.0b013e3181e47be1.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Langan, C., Sarode, D. P., Russ, T. C. et al., Psychiatric symptomatology after delirium: a systematic review. Psychogeriatrics, 2017. https://doi.org/10.1111/psyg.12240.

    Article  Google Scholar 

  8. Davis, D. H. J., Muniz-Terrera, G., Keage, H. A. D. et al., Association of Delirium With Cognitive Decline in Late Life. JAMA Psychiatry 74:244, 2017. https://doi.org/10.1001/jamapsychiatry.2016.3423.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Brown, E., and Douglas, V., Moving Beyond Metabolic Encephalopathy: An Update on Delirium Prevention, Workup, and Management. Semin. Neurol. 35:646–655, 2015. https://doi.org/10.1055/s-0035-1564685.

    Article  PubMed  Google Scholar 

  10. Hshieh, T. T., Yue, J., Oh, E. et al., Effectiveness of Multicomponent Nonpharmacological Delirium Interventions. JAMA Intern. Med. 175:512, 2015. https://doi.org/10.1001/jamainternmed.2014.7779.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Lawlor, P. G., and Bush, S. H., Delirium diagnosis, screening and management. Curr Opin Support Palliat Care 8:286–295, 2014. https://doi.org/10.1097/SPC.0000000000000062.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Inouye, S. K., Van Dyck, C. H., Alessi, C. A. et al., Clarifying confusion: The confusion assessment method: A new method for detection of delirium. Ann. Intern. Med. 113:941–948, 1990. https://doi.org/10.7326/0003-4819-113-12-941.

    Article  CAS  PubMed  Google Scholar 

  13. Waszynski, C., and Petrovic, K., Nurses’ evaluation of the Confusion Assessment Method: a pilot study. J. Gerontol. Nurs. 34:49–56, 2008. https://doi.org/10.3928/00989134-20080401-06.

    Article  PubMed  Google Scholar 

  14. Ely, E. W., Margolin, R., Francis, J. et al., Evaluation of delirium in critically ill patients: validation of the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU). Crit. Care Med. 29:1370–1379, 2001.

    Article  CAS  Google Scholar 

  15. Inouye, S. K., Kosar, C. M., Tommet, D. et al., The CAM-S: Development and validation of a new scoring system for delirium severity in 2 cohorts. Ann. Intern. Med. 160:526–533, 2014. https://doi.org/10.7326/M13-1927.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Van Meenen, L. C. C., van Meenen, D. M. P., de Rooij, S. E., and ter Riet, G., Risk prediction models for postoperative delirium: A systematic review and meta-analysis. J. Am. Geriatr. Soc. 62:2383–2390, 2014. https://doi.org/10.1111/jgs.13138.

    Article  PubMed  Google Scholar 

  17. van den Boogaard, M., Pickkers, P., Slooter, A. J. C. et al., Development and validation of PRE-DELIRIC (PREdiction of DELIRium in ICu patients) delirium prediction model for intensive care patients: observational multicentre study. BMJ 344:e420–e420, 2012. https://doi.org/10.1136/bmj.e420.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Kim MY, Park UJ, Kim HT, Cho WH (2016) DELirium Prediction Based on Hospital Information (Delphi) in General Surgery Patients. 95:1–7. doi: https://doi.org/10.1097/MD.0000000000003072

    Article  Google Scholar 

  19. Kennedy, M., Enander, R. A., Tadiri, S. P. et al., Delirium risk prediction, healthcare use and mortality of elderly adults in the emergency department. J. Am. Geriatr. Soc. 62, 2014. https://doi.org/10.1111/jgs.12692.

    Article  Google Scholar 

  20. Breiman, L., Random Forests. Mach. Learn. 45:5–32, 2001. https://doi.org/10.1023/A:1010933404324.

    Article  Google Scholar 

  21. Quan, H., Li, B., Couris, C. M. et al., Updating and validating the charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am. J. Epidemiol. 173:676–682, 2011. https://doi.org/10.1093/aje/kwq433.

    Article  PubMed  Google Scholar 

  22. Wasey JO (2016) ICD: Tools for Working with ICD-9 and ICD-10 Codes, and Finding Comorbidities

  23. Team TH a. (2017) h2o: R Interface for H2O

  24. Chen C, Liaw A, Breiman L (2004) Using random forest to learn imbalanced data. Univ California, Berkeley 1–12. doi: ley.edu/sites/default/files/tech-reports/666.pdf

  25. Khalilia, M., Chakraborty, S., and Popescu, M., Predicting disease risks from highly imbalanced data using random forest. BMC Med Inform Decis Mak 11:51, 2011. https://doi.org/10.1186/1472-6947-11-51.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Platt, J., Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Adv Large Margin Classif 10:61–74, 1999. 10.1.1.41.1639.

  27. Kuhn M (2008) Building Predictive Models in R Using the caret Package. J Stat Software, Artic 28:1–26. https://doi.org/10.18637/jss.v028.i05.

  28. Robin, X., Turck, N., Hainard, A. et al., pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12:77, 2011. https://doi.org/10.1186/1471-2105-12-77.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Grau, J., Grosse, I., and Keilwagen, J., PRROC: computing and visualizing precision-recall and receiver operating characteristic curves in R. Bioinformatics 31:2595–2597, 2015. https://doi.org/10.1093/bioinformatics/btv153.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Boulesteix, A.-L., Janitza, S., Kruppa, J., and König, I. R., Overview of random forest methodology and practical guidance with emphasis on computational biology and bioinformatics. Wiley Interdiscip Rev Data Min Knowl Discov 2:493–507, 2012. https://doi.org/10.1002/widm.1072.

    Article  Google Scholar 

  31. Semancik, L., Waszynski, C., and Udeh, E., Delirium in hospitalized patients: recognition, prevention, and management. Conn. Med. 78:105–109, 2014.

    PubMed  Google Scholar 

  32. Tieges, Z., McGrath, A., Hall, R. J., and MacLullich, A. M. J., Abnormal Level of Arousal as a Predictor of Delirium and Inattention: An Exploratory Study. Am. J. Geriatr. Psychiatry 21:1244–1253, 2013. https://doi.org/10.1016/j.jagp.2013.05.003.

    Article  PubMed  Google Scholar 

  33. Han, J. H., Vasilevskis, E. E., Schnelle, J. F. et al., The Diagnostic Performance of the Richmond Agitation Sedation Scale for Detecting Delirium in Older Emergency Department Patients. Acad. Emerg. Med. 22:878–882, 2015. https://doi.org/10.1111/acem.12706.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Morandi, A., Han, J. H., Meagher, D. et al., Detecting Delirium Superimposed on Dementia: Evaluation of the Diagnostic Performance of the Richmond Agitation and Sedation Scale. J. Am. Med. Dir. Assoc. 17:828–833, 2016. https://doi.org/10.1016/j.jamda.2016.05.010.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Corradi, J. P., Chhabra, J., Mather, J. F. et al., Analysis of multi-dimensional contemporaneous EHR data to refine delirium assessments. Comput. Biol. Med. 5799:1–24, 2016. https://doi.org/10.1016/j.compbiomed.2016.06.013.

    Article  Google Scholar 

  36. Gulshan, V., Peng, L., Coram, M. et al., Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. Jama 304:649–656, 2016. https://doi.org/10.1001/jama.2016.17216.

    Article  Google Scholar 

  37. Dawes, T. J. W., de Marvao, A., Shi, W. et al., Machine Learning of Three-dimensional Right Ventricular Motion Enables Outcome Prediction in Pulmonary Hypertension: A Cardiac MR Imaging Study. Radiology 161315, 2017. https://doi.org/10.1148/radiol.2016161315.

    Article  Google Scholar 

  38. Ghose, S., Mitra, J., Khanna, S., and Dowling, J., An Improved Patient-Specific Mortality Risk Prediction in ICU in a Random Forest Classification Framework. Stud Health Technol Inform 214:56–61, 2015. https://doi.org/10.3233/978-1-61499-558-6-56.

    Article  PubMed  Google Scholar 

  39. Futoma, J., Morris, J., and Lucas, J., A comparison of models for predicting early hospital readmissions. J. Biomed. Inform. 56:229–238, 2015. https://doi.org/10.1016/j.jbi.2015.05.016.

    Article  PubMed  Google Scholar 

  40. Kessler, R. C., Warner, C. H., Ivany, C. et al., Predicting Suicides After Psychiatric Hospitalization in US Army Soldiers: The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry 72:1–9, 2014. https://doi.org/10.1001/jamapsychiatry.2014.1754.

    Article  Google Scholar 

  41. Mortazavi, B. J., Downing, N. S., Bucholz, E. M. et al., Analysis of Machine Learning Techniques for Heart Failure Readmissions. Circ Cardiovasc Qual Outcomes 9:629–640, 2016. https://doi.org/10.1161/CIRCOUTCOMES.116.003039.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Taylor, R. A., Pare, J. R., Venkatesh, A. K. et al., Prediction of In-hospital Mortality in Emergency Department Patients with Sepsis: A Local Big Data-Driven, Machine Learning Approach. Acad. Emerg. Med. 23:269–278, 2016. https://doi.org/10.1111/acem.12876.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Kramer, D., Veeranki, S., Hayn, D., and Quehenberger, F., Development and Validation of a Multivariable Prediction Model for the Occurrence of Delirium in Hospitalized Gerontopsychiatry and Internal Medicine Patients. Stud Health Technol Inform:32–39, 2017. https://doi.org/10.3233/978-1-61499-759-7-32.

  44. Wong, A., Young, A. T., Liang, A. S. et al., Development and Validation of an Electronic Health Record–Based Machine Learning Model to Estimate Delirium Risk in Newly Hospitalized Patients Without Known Cognitive Impairment. JAMA Netw. Open 1:e181018, 2018. https://doi.org/10.1001/jamanetworkopen.2018.1018.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Halladay, C. W., Sillner, A. Y., and Rudolph, J. L., Performance of Electronic Prediction Rules for Prevalent Delirium at Hospital Admission. JAMA Netw. Open 1:e181405, 2018. https://doi.org/10.1001/JAMANETWORKOPEN.2018.1405.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Research Department, Hartford Hospital, 80 Seymour Street, ERD-223W, Hartford, CT, 06102, USA

    John P. Corradi, Stephen Thompson & Jeffrey F. Mather

  2. Division of Geriatric Medicine, Hartford Hospital, Hartford, CT, USA

    Christine M. Waszynski & Robert S. Dicks

Authors
  1. John P. Corradi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephen Thompson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jeffrey F. Mather
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christine M. Waszynski
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Robert S. Dicks
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to John P. Corradi.

Ethics declarations

Conflicts of Interest

The authors do not have any conflicts of interest to declare. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

For this type of study formal consent is not required.

Additional information

This article is part of the Topical Collection on Systems-Level Quality Improvement

Electronic supplementary material

ESM 1

(XLSX 16 kb)

ESM 2

(PDF 255 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Corradi, J.P., Thompson, S., Mather, J.F. et al. Prediction of Incident Delirium Using a Random Forest classifier. J Med Syst 42, 261 (2018). https://doi.org/10.1007/s10916-018-1109-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10916-018-1109-0

Keywords

Search

Navigation