Abstract
This work describes applying a transformer-based ranking solution to the specific problem of ordering ICD diagnoses codes. Taking advantage of the TFR-BERT framework and adapting it to the biomedical context using pre-trained and publicly available language representation models, namely BioBERT, BlueBERT and ClinicalBERT (Bio + Discharge Summary BERT Model), we demonstrate the effectiveness of such a framework and the strengths of using pre-trained models adapted to the biomedical domain. We showcase this by using a benchmark dataset in the healthcare field—MIMIC-III—showing how it was possible to learn how to sequence the main or primary diagnoses and the order in which the secondary diagnoses are presented. A window-based approach and a summary approach (using only the sentences with diagnoses) were also tested in an attempt to circumvent the maximum sequence length limitation of BERT-based models. BioBERT demonstrated superior performance in all approaches, achieving the best results in the summary approach.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
BERT-Base-Cased checkpoint can be downloaded from: https://github.com/google-research/bert.
- 2.
BioBERT-Base v1.1, based on BERT-base-Cased (same vocabulary), checkpoint can be downloaded from: https://github.com/dmislab/biobert.
- 3.
BlueBERT-Base-Uncased, PubMed+MIMIC-III checkpoint can be downloaded from: https://github.com/ncbinlp/bluebert; or from Hugging Face Hub: https://huggingface.co/bionlp/bluebert_pubmed_mimic_uncased_L12_H768_A12.
- 4.
ClinicalBERT (Bio + Discharge Summary BERT model) checkpoint can be downloaded from: https://github.com/EmilyAlsentzer/clinicalBERT; or from Hugging Face Hub: https://huggingface.co/emilyalsentzer/Bio_Discharge_Summary_BERT.
References
Alsentzer, E., Murphy, J., Boag, W., Weng, W.H., Jindi, D., Naumann, T., McDermott, M.: Publicly available clinical BERT embeddings. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop. pp. 72–78. Association for Computational Linguistics, Minneapolis, Minnesota, USA (2019). https://doi.org/10.18653/v1/W19-1909
Amin, S., Neumann, G., Dunfield, K.A., Vechkaeva, A., Chapman, K.A., Wixted, M.K.: Mlt-dfki at clef ehealth 2019: Multi-label classification of icd-10 codes with bert. In: Conference and Labs of the Evaluation Forum (2019)
Aronson, A.R.: Effective mapping of biomedical text to the umls metathesaurus: the metamap program. In: Proceedings of the AMIA Symposium, p. 17. American Medical Informatics Association (2001)
Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document transformer (2020). arXiv:2004.05150
Bruch, S., Zoghi, M., Bendersky, M., Najork, M.: Revisiting approximate metric optimization in the age of deep neural networks. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1241–1244. SIGIR’19. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3331184.3331347
Carla Smith, S.B., Dooling, J.A.: Measuring and benchmarking coding productivity: a decade of Ahima leadership (2015)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding (2019)
Han, S., Wang, X., Bendersky, M., Najork, M.: Learning-to-rank with BERT in TF-ranking (2020). arXiv:2004.08476
Honnibal, M., Montani, I.: spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing (2017), to appear
Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. (TOIS) 20(4), 422–446 (2002)
Johnson, A.E., Pollard, T.J., Shen, L., Lehman, L.W.H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., Mark, R.G.: MIMIC-III, a freely accessible critical care database. Sci. Data 3(1), 1–9 (2016)
Kaur, R., Ginige, J.A., Obst, O.: A systematic literature review of automated ICD coding and classification systems using discharge summaries (2021)
Kavuluru, R., Rios, A., Lu, Y.: An empirical evaluation of supervised learning approaches in assigning diagnosis codes to electronic medical records. Artif. Intell. Med. 65 (2015). https://doi.org/10.1016/j.artmed.2015.04.007
Kocaman, V., Talby, D.: Spark NLP: natural language understanding at scale. Softw. Impacts 100058 (2021). https://doi.org/10.1016/j.simpa.2021.100058
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics (2019). https://doi.org/10.1093/bioinformatics/btz682
Pascual, D., Luck, S., Wattenhofer, R.: Towards BERT-based automatic ICD coding: limitations and opportunities. In: Workshop on Biomedical Natural Language Processing (2021)
Pasumarthi, R.K., Bruch, S., Wang, X., Li, C., Bendersky, M., Najork, M., Pfeifer, J., Golbandi, N., Anil, R., Wolf, S.: TF-ranking: Scalable tensorflow library for learning-to-rank. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2970–2978 (2019)
Peng, Y., Yan, S., Lu, Z.: Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In: Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019), pp. 58–65 (2019)
Pirie, W.: Spearman Rank Correlation Coefficient. Wiley (2006). https://doi.org/10.1002/0471667196.ess2499.pub2
Qin, T., Liu, T.Y., Li, H.: A general approximation framework for direct optimization of information retrieval measures. Inf. Retr. 13, 375–397 (2010). https://doi.org/10.1007/s10791-009-9124-x
Silva, A., Silva, A., Oliveira, T., Novais, P.: Mapping a clinical case description to an argumentation framework: a preliminary assessment. In: Intelligent Data Engineering and Automated Learning-IDEAL 2020: 21st International Conference, Guimaraes, Portugal, Proceedings, Part I, vol. 21, pp. 338–349. Springer (2020)
Wang, Y., Wang, L., Li, Y., He, D., Liu, T.Y.: A theoretical analysis of NDCG type ranking measures. In: Conference on Learning Theory, pp. 25–54. PMLR (2013)
WHO: ICD-10: international statistical classification of diseases and related health problems: tenth revision (2004)
Zhang, Z., Liu, J., Razavian, N.: BERT-XML: Large scale automated ICD coding using BERT pretraining. In: Proceedings of the 3rd Clinical Natural Language Processing Workshop, pp. 24–34. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.clinicalnlp-1.3
Acknowledgment
We would like to express our appreciation to Select Data, Inc. for supporting this research and publication. The work of Paulo Novais has been supported by FCT—Fundação para a Ciência e Tecnologia within the R &D Units Project Scope: UIDB/00319/2020.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Silva, A., Chaves, P., Rijo, S., Boné, J., Oliveira, T., Novais, P. (2023). Leveraging TFR-BERT for ICD Diagnoses Ranking. In: Moniz, N., Vale, Z., Cascalho, J., Silva, C., Sebastião, R. (eds) Progress in Artificial Intelligence. EPIA 2023. Lecture Notes in Computer Science(), vol 14116. Springer, Cham. https://doi.org/10.1007/978-3-031-49011-8_25
Download citation
DOI: https://doi.org/10.1007/978-3-031-49011-8_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-49010-1
Online ISBN: 978-3-031-49011-8
eBook Packages: Computer ScienceComputer Science (R0)