skip to main content
10.1145/3410530.3414436acmconferencesArticle/Chapter ViewAbstractPublication PagesubicompConference Proceedingsconference-collections
research-article

Improving fine-tuned question answering models for electronic health records

Published: 12 September 2020 Publication History

Abstract

The prevalence of voice assistants has strengthened the interest in a question answering for the medical domain, allowing both patients and healthcare providers to enter a question naturally and pinpoint useful information quickly. However, a large number of medical terms make the creation of such a system a demanding task. To address this challenge, we explore transfer learning techniques for constructing a personalized EHR-QA system. The goal is to answer questions regarding a discharge summary in an electronic health record (EHR). We present the experiments with a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model fine-tuned on different tasks and show the results obtained to provide insights into learning effects and training effectiveness.

References

[1]
Alan Akbik, Duncan Blythe, and Roland Vollgraf. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638--1649, 2018.
[2]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
[3]
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint arXiv:1803.11175, 2018.
[4]
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013.
[5]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[6]
Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014.
[7]
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 2019.
[8]
Fei Li, Yonghao Jin, Weisong Liu, Bhanu Pratap Singh Rawat, Pengshan Cai, and Hong Yu. Fine-tuning bidirectional encoder representations from transformers (bert)-based models on large-scale electronic health record notes: An empirical study. JMIR medical informatics, 7(3):e14830, 2019.
[9]
Yifan Peng, Shankai Yan, and Zhiyong Lu. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. In Proceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019), pages 58--65, 2019.
[10]
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016.
[11]
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104--3112, 2014.
[12]
Wonjin Yoon, Jinhyuk Lee, Donghyeon Kim, Minbyul Jeong, and Jaewoo Kang. Pre-trained language model for biomedical question answering. In Peggy Cellier and Kurt Driessens, editors, Machine Learning and Knowledge Discovery in Databases, pages 727--740, Cham, 2020. Springer International Publishing. ISBN 978-3-030-43887-6.
[13]
Xiaohui Zhang, Yaoyun Zhang, Qin Zhang, Yuankai Ren, Tinglin Qiu, Jianhui Ma, and Qiang Sun. Extracting comprehensive clinical information for breast cancer using deep learning methods. International Journal of Medical Informatics, 132:103985, 2019.

Cited By

View all
  • (2024)Question Answering for Electronic Health Records: Scoping Review of Datasets and ModelsJournal of Medical Internet Research10.2196/5363626(e53636)Online publication date: 30-Oct-2024
  • (2023)Pre-trained Language Models in Biomedical Domain: A Systematic SurveyACM Computing Surveys10.1145/361165156:3(1-52)Online publication date: 5-Oct-2023
  • (2022)Design and implementation of information extraction system for scientific literature using fine-tuned deep learning modelsACM SIGAPP Applied Computing Review10.1145/3530043.353004722:1(31-38)Online publication date: 1-Apr-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
UbiComp/ISWC '20 Adjunct: Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers
September 2020
732 pages
ISBN:9781450380768
DOI:10.1145/3410530
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 September 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. electronic health records
  2. question answering
  3. transfer learning

Qualifiers

  • Research-article

Conference

UbiComp/ISWC '20

Acceptance Rates

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)28
  • Downloads (Last 6 weeks)3
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Question Answering for Electronic Health Records: Scoping Review of Datasets and ModelsJournal of Medical Internet Research10.2196/5363626(e53636)Online publication date: 30-Oct-2024
  • (2023)Pre-trained Language Models in Biomedical Domain: A Systematic SurveyACM Computing Surveys10.1145/361165156:3(1-52)Online publication date: 5-Oct-2023
  • (2022)Design and implementation of information extraction system for scientific literature using fine-tuned deep learning modelsACM SIGAPP Applied Computing Review10.1145/3530043.353004722:1(31-38)Online publication date: 1-Apr-2022
  • (2021)Classification of Arabic healthcare questions based on word embeddings learned from massive consultations: a deep learning approachJournal of Ambient Intelligence and Humanized Computing10.1007/s12652-021-02948-w13:4(1811-1827)Online publication date: 8-Mar-2021

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media