Skip to main content

Advertisement

SpringerLink
Log in

Improving clinical documentation: automatic inference of ICD-10 codes from patient notes using BERT model

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Electronic health records provide a vast amount of text health data written by physicians as patient clinical notes. The world health organization released the international classification of diseases version 10 (ICD-10) system to monitor and analyze clinical notes. ICD-10 is system physicians and other healthcare providers use to classify and code all diagnoses and symptom records in conjunction with hospital care. Therefore, the data can be easily stored, retrieved, and analyzed for decision-making. In order to address the problem, this paper introduces a system to classify the clinical notes to ICD-10 codes. This paper examines 7541 clinical notes collected from a health institute in Jordan and annotated by ICD-10’s coders. In addition, the research uses another outsource dataset to augment the actual dataset. The research presented many approaches, such as the baseline and pipeline models. The Baseline model employed several methods like Word2vec embedding for representing the text. The model structure also involves long-short-term memory a convolutional neural network, and two fully-connected layers. The second Pipeline approach adopts the transformer model, such as Bidirectional Encoder Representations from Transformers (BERT), which is pre-trained on a similar health domain. The Pipeline model builds on two BERT models. The first model classifies the category codes representing the first three characters of ICD-10. The second BERT model uses the outputs from the general BERT model (first model) as input for the special BERT (second model) to classify the clinical notes into total codes of ICD-10. Moreover, Baseline and Pipeline models applied the Focal loss function to eliminate the imbalanced classes. However, The Pipeline model demonstrates a significant performance by evaluating it over the F1 score, recall, precision, and accuracy metric, which are 92.5%, 84.9%, 91.8%, and 84.97%, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

Data are available from the authors upon reasonable request.

Notes

  1. https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT.

  2. https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT.

  3. https://colab.research.google.com/signup.

  4. https://keras.io/.

  5. https://huggingface.co/.

  6. https://huggingface.co/Emran/ClinicalBERT_ICD10_Categories.

  7. https://huggingface.co/Emran/ClinicalBERT_ICD1.

References

  1. Kalra D (2006) Electronic health record standards. Yearb Med Inf 15(01):136–144

    Article  MathSciNet  Google Scholar 

  2. Cimino JJ (2013) Improving the electronic health record-are clinicians getting what they wished for? Jama 309(10):991–992

    Article  Google Scholar 

  3. Organization WH et al (1992) The icd-10 classification of mental and behavioural disorders: clinical descriptions and diagnostic guidelines. Week Epidemiol Record Relevé épidémiologique hebdomadaire 67(30):227–227

    Google Scholar 

  4. Zivetz L (1992) The ICD-10 classification of mental and behavioural disorders: clinical descriptions and diagnostic guidelines, vol 1. World Health Organization

  5. Movig K, Leufkens H, Lenderink A, Egberts A (2003) Validity of hospital discharge international classification of diseases (icd) codes for identifying patients with hyponatremia. J Clin Epidemiol 56(6):530–535

    Article  Google Scholar 

  6. Organization W.H (2004) International statistical classification of diseases and related health problems, vol 1. World Health Organization

  7. AlZu’bi S, Elbes M, Mughaid A, Bdair N, Abualigah L, Forestiero A, Zitar RA (2023) Diabetes monitoring system in smart health cities based on big data intelligence. Fut Internet 15(2):85

    Article  Google Scholar 

  8. Alzu’bi D, Abdullah M, Hmeidi I, AlAzab R, Gharaibeh M, El-Heis M, Almotairi KH, Forestiero A, Hussein AM, Abualigah L, et al (2022) Kidney tumor detection and classification based on deep learning approaches: A new dataset in ct scans. J Healthc Eng

  9. Comito C, Falcone D, Forestiero A (2022) Convergence between iot and ai for smart health and predictive medicine. In: Integrating Artificial Intelligence and IoT for Advanced Health Informatics: AI in the Healthcare Sector. Springer. pp 69–84

  10. Brà C et al (1999) A hospital-wide clinical findings dictionary based on an extension of the international classification of diseases (icd). In: Proceedings of the AMIA Symposium. American Medical Informatics Association, 706

  11. Lovis C, Baud R, Rassinoux A-M, Michel P-A, Scherrer J-R (1998) Medical dictionaries for patient encoding systems: a methodology. Artif Intell Med 14(1–2):201–214

    Article  Google Scholar 

  12. Murphy K.P (2012) Machine learning: a probabilistic perspective. MIT press

  13. Park DJ, Park MW, Lee H, Kim Y-J, Kim Y, Park YH (2021) Development of machine learning model for diagnostic disease prediction based on laboratory tests. Sci Rep 11(1):1–11

    Google Scholar 

  14. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  15. Miotto R, Li L, Kidd BA, Dudley JT (2016) Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Sci Rep 6(1):1–10

    Article  Google Scholar 

  16. Jagannatha A.N, Yu H (2016) Structured prediction models for rnn based sequence labeling in clinical text. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, vol 2016. NIH Public Access. p 856

  17. Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Sigl Process 45(11):2673–2681

    Article  Google Scholar 

  18. Pascual D, Luck S, Wattenhofer R, Towards bert-based automatic icd coding: limitations and opportunities. arXiv preprint arXiv:2104.06709

  19. López-Úbeda P, Díaz-Galiano MC, Martín-Noguerol T, Luna A, Ureña-López LA, Martín-Valdivia MT (2021) Automatic medical protocol classification using machine learning approaches. Comput Methods Progr Biomed 200:105939

    Article  Google Scholar 

  20. Borjali A, Magneli M, Shin D, Malchau H, Muratoglu OK, Varadarajan KM (2021) Natural language processing with deep learning for medical adverse event detection from free-text medical narratives: A case study of detecting total hip replacement dislocation. Comput Biol Med 129:104140

    Article  Google Scholar 

  21. Duarte F, Martins B, Pinto CS, Silva MJ (2018) Deep neural models for icd-10 coding of death certificates and autopsy reports in free-text. J Biomed Inf 80:64–77

    Article  Google Scholar 

  22. Atutxa A, de Ilarraza AD, Gojenola K, Oronoz M, Perez-de Viñaspre O (2019) Interpretable deep learning to map diagnostic texts to icd-10 codes. Int J Med Inf 129:49–59

    Article  Google Scholar 

  23. Zhan X, Humbert-Droz M, Mukherjee P, Gevaert O, Structuring clinical text with ai: old vs. new natural language processing techniques evaluated on eight common cardiovascular diseases. medRxiv

  24. Bagheri A, Sammani A, Van der Heijden P.G, Asselbergs F.W, Oberski D.L (2020) Automatic icd-10 classification of diseases from dutch discharge letters. In: BIOINFORMATICS 2020-11th International Conference on Bioinformatics Models, Methods and Algorithms, Proceedings; Part of 13th International Joint Conference on Biomedical Engineering Systems and Technologies. BIOSTEC 2020, vol 13, SciTePress pp 281–289

  25. Velichkov B, Gerginov S, Panayotov P, Vassileva S, Velchev G, Koychev I, Boytcheva S (2020)Automatic icd-10 codes association to diagnosis: Bulgarian case. In: CSBio’20: Proceedings of the Eleventh International Conference on Computational Systems-Biology and Bioinformatics, pp 46–53

  26. Silvestri S, Gargiulo F, Ciampi M, De Pietro G (2020) Exploit multilingual language model at scale for icd-10 clinical text classification. In (2020) IEEE Symposium on Computers and Communications (ISCC). IEEE: 1–7

  27. Della Mea V, Popescu MH, Roitero K (2020) Underlying cause of death identification from death certificates using reverse coding to text and a nlp based deep learning approach. Inf Med Unlock 21:100456

    Article  Google Scholar 

  28. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  29. Brahim AO, Belaidi I, Khatir S, Le Thanh C, Mirjalili S, Wahab MA (2023) Strength prediction of a steel pipe having a hemi-ellipsoidal corrosion defect repaired by gfrp composite patch using artificial neural network. Compos Struct 304:116299

    Article  Google Scholar 

  30. Mikolov T, Karafiát M, Burget L, Cernockỳ J, Khudanpur S (2010) Recurrent neural network based language model. In: Interspeech vol. 2, Makuhari pp 1045–1048

  31. Cuong-Le T, Nghia-Nguyen T, Khatir S, Trong-Nguyen P, Mirjalili S, Nguyen KD (2021) An efficient approach for damage identification based on improved machine learning using pso-svm. Eng Comput 1–16

  32. Zhang Y, Wallace B, A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification. arXiv preprint arXiv:1510.03820

  33. Tolias G, Sicre R, Jégou H, Particular object retrieval with integral max-pooling of cnn activations. arXiv preprint arXiv:1511.05879

  34. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  35. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, PMLR. pp 448–456

  36. Huang G-B, Zhu Q-Y, Siew C-K, (2004) Extreme learning machine: a new learning scheme of feedforward neural networks, in, (2004) IEEE international joint conference on neural networks (IEEE Cat. No. 04CH37541), Vol. 2. Ieee :985–990

  37. Agarap AF, Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375

  38. Gold S, Rangarajan A et al (1996) Softmax to softassign: Neural network algorithms for combinatorial optimization. J Artif Neural Netw 2(4):381–399

    Google Scholar 

  39. Mikolov T, Sutskever I, Chen K, Corrado G.S, Dean J, (2013) Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp 3111–3119

  40. Devlin J, Chang M.-W, Lee K, Toutanova K Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  41. Johnson AE, Pollard TJ, Shen L, Li-Wei HL, Feng M, Ghassemi M, Moody B, Szolovits P, Celi LA, Mark RG (2016) Mimic-iii, a freely accessible critical care database. Sci Data 3(1):1–9

    Article  Google Scholar 

  42. Alsentzer E, Murphy JR, Boag W, Weng W-H, Jin D, Naumann T, McDermott M, Publicly available clinical bert embeddings. arXiv preprint arXiv:1904.03323

  43. Lin T.-Y, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision. pp 2980–2988

  44. Zhang Z, Sabuncu MR (2018) Generalized cross entropy loss for training deep neural networks with noisy labels. In: 32nd Conference on Neural Information Processing Systems (NeurIPS)

Download references

Acknowledgments

We thankfully acknowledge the Deanship of Research at the Jordan University of Science and Technology (JUST) for financing this work via Grant number: 20210379. We further admit the efforts of the E-learning Center, Health Center, and King Abdullah University Hospital (KAUH) at JUST for providing this project with the dataset.

Author information

Authors and Affiliations

  1. Department of Computer Information Systems, Jordan University of Science and Technology, 22110, Irbid, Jordan

    Emran Al-Bashabsheh, Ahmad Alaiad, Mahmoud Al-Ayyoub & Othman Beni-Yonis

  2. Sorbonne Center of Artificial Intelligence, Sorbonne University-Abu Dhabi, 38044, Abu Dhabi, United Arab Emirates

    Raed Abu Zitar

  3. Computer Science Department, Prince Hussein Bin Abdullah Faculty for Information Technology, Al al-Bayt University, Mafraq, 25113, Jordan

    Laith Abualigah

  4. College of Engineering, Yuan Ze University, Taoyuan City, Taiwan

    Laith Abualigah

  5. Hourani Center for Applied Scientific Research, Al-Ahliyya Amman University, 19328, Amman, Jordan

    Laith Abualigah

  6. Faculty of Information Technology, Middle East University, Amman, 11831, Jordan

    Laith Abualigah

  7. Applied science research center, Applied science private university, 11931, Amman, Jordan

    Laith Abualigah

  8. School of Computer Sciences, Universiti Sains Malaysia, George Town, Pulau Pinang, 11800, Malaysia

    Laith Abualigah

Authors
  1. Emran Al-Bashabsheh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmad Alaiad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mahmoud Al-Ayyoub
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Othman Beni-Yonis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Raed Abu Zitar
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Laith Abualigah
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

EAB contributed to writing—original draft preparation, visualization, investigation. AA contributed to supervision, conceptualization, methodology, software, investigation, validation, writing—original draft preparation. MAA contributed to writing—original draft preparation, visualization, investigation. OB-Y contributed to writing—original draft preparation, visualization, investigation. RAZ contributed to writing—original draft preparation, visualization, investigation. LA contributed to writing—original draft preparation, visualization, investigation.

Corresponding author

Correspondence to Laith Abualigah.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this paper.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Al-Bashabsheh, E., Alaiad, A., Al-Ayyoub, M. et al. Improving clinical documentation: automatic inference of ICD-10 codes from patient notes using BERT model. J Supercomput 79, 12766–12790 (2023). https://doi.org/10.1007/s11227-023-05160-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-023-05160-z

Keywords

Search

Navigation