Skip to main content

Leveraging TFR-BERT for ICD Diagnoses Ranking

  • Conference paper
  • First Online:
Progress in Artificial Intelligence (EPIA 2023)

Abstract

This work describes applying a transformer-based ranking solution to the specific problem of ordering ICD diagnoses codes. Taking advantage of the TFR-BERT framework and adapting it to the biomedical context using pre-trained and publicly available language representation models, namely BioBERT, BlueBERT and ClinicalBERT (Bio + Discharge Summary BERT Model), we demonstrate the effectiveness of such a framework and the strengths of using pre-trained models adapted to the biomedical domain. We showcase this by using a benchmark dataset in the healthcare field—MIMIC-III—showing how it was possible to learn how to sequence the main or primary diagnoses and the order in which the secondary diagnoses are presented. A window-based approach and a summary approach (using only the sentences with diagnoses) were also tested in an attempt to circumvent the maximum sequence length limitation of BERT-based models. BioBERT demonstrated superior performance in all approaches, achieving the best results in the summary approach.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    BERT-Base-Cased checkpoint can be downloaded from: https://github.com/google-research/bert.

  2. 2.

    BioBERT-Base v1.1, based on BERT-base-Cased (same vocabulary), checkpoint can be downloaded from: https://github.com/dmislab/biobert.

  3. 3.

    BlueBERT-Base-Uncased, PubMed+MIMIC-III checkpoint can be downloaded from: https://github.com/ncbinlp/bluebert; or from Hugging Face Hub: https://huggingface.co/bionlp/bluebert_pubmed_mimic_uncased_L12_H768_A12.

  4. 4.

    ClinicalBERT (Bio + Discharge Summary BERT model) checkpoint can be downloaded from: https://github.com/EmilyAlsentzer/clinicalBERT; or from Hugging Face Hub: https://huggingface.co/emilyalsentzer/Bio_Discharge_Summary_BERT.

References

  1. Alsentzer, E., Murphy, J., Boag, W., Weng, W.H., Jindi, D., Naumann, T., McDermott, M.: Publicly available clinical BERT embeddings. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop. pp. 72–78. Association for Computational Linguistics, Minneapolis, Minnesota, USA (2019). https://doi.org/10.18653/v1/W19-1909

  2. Amin, S., Neumann, G., Dunfield, K.A., Vechkaeva, A., Chapman, K.A., Wixted, M.K.: Mlt-dfki at clef ehealth 2019: Multi-label classification of icd-10 codes with bert. In: Conference and Labs of the Evaluation Forum (2019)

    Google Scholar 

  3. Aronson, A.R.: Effective mapping of biomedical text to the umls metathesaurus: the metamap program. In: Proceedings of the AMIA Symposium, p. 17. American Medical Informatics Association (2001)

    Google Scholar 

  4. Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document transformer (2020). arXiv:2004.05150

  5. Bruch, S., Zoghi, M., Bendersky, M., Najork, M.: Revisiting approximate metric optimization in the age of deep neural networks. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1241–1244. SIGIR’19. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3331184.3331347

  6. Carla Smith, S.B., Dooling, J.A.: Measuring and benchmarking coding productivity: a decade of Ahima leadership (2015)

    Google Scholar 

  7. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding (2019)

    Google Scholar 

  8. Han, S., Wang, X., Bendersky, M., Najork, M.: Learning-to-rank with BERT in TF-ranking (2020). arXiv:2004.08476

  9. Honnibal, M., Montani, I.: spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing (2017), to appear

    Google Scholar 

  10. Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. (TOIS) 20(4), 422–446 (2002)

    Article  Google Scholar 

  11. Johnson, A.E., Pollard, T.J., Shen, L., Lehman, L.W.H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., Mark, R.G.: MIMIC-III, a freely accessible critical care database. Sci. Data 3(1), 1–9 (2016)

    Google Scholar 

  12. Kaur, R., Ginige, J.A., Obst, O.: A systematic literature review of automated ICD coding and classification systems using discharge summaries (2021)

    Google Scholar 

  13. Kavuluru, R., Rios, A., Lu, Y.: An empirical evaluation of supervised learning approaches in assigning diagnosis codes to electronic medical records. Artif. Intell. Med. 65 (2015). https://doi.org/10.1016/j.artmed.2015.04.007

  14. Kocaman, V., Talby, D.: Spark NLP: natural language understanding at scale. Softw. Impacts 100058 (2021). https://doi.org/10.1016/j.simpa.2021.100058

  15. Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics (2019). https://doi.org/10.1093/bioinformatics/btz682

  16. Pascual, D., Luck, S., Wattenhofer, R.: Towards BERT-based automatic ICD coding: limitations and opportunities. In: Workshop on Biomedical Natural Language Processing (2021)

    Google Scholar 

  17. Pasumarthi, R.K., Bruch, S., Wang, X., Li, C., Bendersky, M., Najork, M., Pfeifer, J., Golbandi, N., Anil, R., Wolf, S.: TF-ranking: Scalable tensorflow library for learning-to-rank. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2970–2978 (2019)

    Google Scholar 

  18. Peng, Y., Yan, S., Lu, Z.: Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In: Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019), pp. 58–65 (2019)

    Google Scholar 

  19. Pirie, W.: Spearman Rank Correlation Coefficient. Wiley (2006). https://doi.org/10.1002/0471667196.ess2499.pub2

  20. Qin, T., Liu, T.Y., Li, H.: A general approximation framework for direct optimization of information retrieval measures. Inf. Retr. 13, 375–397 (2010). https://doi.org/10.1007/s10791-009-9124-x

  21. Silva, A., Silva, A., Oliveira, T., Novais, P.: Mapping a clinical case description to an argumentation framework: a preliminary assessment. In: Intelligent Data Engineering and Automated Learning-IDEAL 2020: 21st International Conference, Guimaraes, Portugal, Proceedings, Part I, vol. 21, pp. 338–349. Springer (2020)

    Google Scholar 

  22. Wang, Y., Wang, L., Li, Y., He, D., Liu, T.Y.: A theoretical analysis of NDCG type ranking measures. In: Conference on Learning Theory, pp. 25–54. PMLR (2013)

    Google Scholar 

  23. WHO: ICD-10: international statistical classification of diseases and related health problems: tenth revision (2004)

    Google Scholar 

  24. Zhang, Z., Liu, J., Razavian, N.: BERT-XML: Large scale automated ICD coding using BERT pretraining. In: Proceedings of the 3rd Clinical Natural Language Processing Workshop, pp. 24–34. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.clinicalnlp-1.3

Download references

Acknowledgment

We would like to express our appreciation to Select Data, Inc. for supporting this research and publication. The work of Paulo Novais has been supported by FCT—Fundação para a Ciência e Tecnologia within the R &D Units Project Scope: UIDB/00319/2020.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ana Silva .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Silva, A., Chaves, P., Rijo, S., Boné, J., Oliveira, T., Novais, P. (2023). Leveraging TFR-BERT for ICD Diagnoses Ranking. In: Moniz, N., Vale, Z., Cascalho, J., Silva, C., Sebastião, R. (eds) Progress in Artificial Intelligence. EPIA 2023. Lecture Notes in Computer Science(), vol 14116. Springer, Cham. https://doi.org/10.1007/978-3-031-49011-8_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-49011-8_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-49010-1

  • Online ISBN: 978-3-031-49011-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics