Abstract
Extensive technologies have been employed to explore a best way for cross-lingual transfer learning. In medical domain, Named Entity Recognition is pivotal for many downstream tasks, such as medical entity linking and clinical decision support systems. Nevertheless, the lack of annotation limits the applicability in many languages without enough labeled data. To alleviate this issue and make use of languages with sufficient annotated data, we find a new way to obtain medical parallel corpus from medical terminology systems and knowledge bases and propose a methodology which combines cross-lingual language model pretraining and bilingual word embedding alignment with the help of the parallel corpus. Moreover, our combined architecture which maintains the framework of pretrained model can not only be used for NER task but also other downstream NLP tasks. Experiments demonstrated that incorporating Chinese and English medical data can effectively improve the performance for an English medical NER dataset (i2b2).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Artetxe, M., Labaka, G., Agirre, E.: Learning principled bilingual mappings of word embeddings while preserving monolingual invariance, pp. 2289–2294 (2016)
Artetxe, M., Labaka, G., Agirre, E.: Learning bilingual word embeddings with (almost) no bilingual data, vol. 1, pp. 451–462 (2017)
Artetxe, M., Schwenk, H.: Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond (2018). arXiv Computation and Language
Conneau, A., Kiela, D., Schwenk, H., Barrault, L., Bordes, A.: Supervised learning of universal sentence representations from natural language inference data (2017). arXiv Computation and Language
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding (2018). arXiv preprint arXiv:1810.04805
Doan, S., Bastarache, L., Klimkowski, S., Denny, J.C., Xu, H.: Integrating existing natural language processing tools for medication extraction from discharge summaries. J. Am. Med. Inf. Assoc. 17(5), 528–531 (2010)
Ehrmann, M., Turchi, M., Steinberger, R.: Building a multilingual named entity-annotated corpus using annotation projection, pp. 118–124 (2011)
Faruqui, M., Dyer, C.: Improving vector space word representations using multilingual correlation, pp. 462–471 (2014)
Friedman, C., Alderson, P.O., Austin, J.H.M., Cimino, J.J., Johnson, S.B.: A general natural-language text processor for clinical radiology. J. Am. Med. Inf. Assoc. 1(2), 161–174 (1994)
Gridach, M.: Character-level neural network for biomedical named entity recognition. J. Biomed. Inf. 70, 85–91 (2017)
Habibi, M., Weber, L., Neves, M.L., Wiegandt, D.L., Leser, U.: Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics 33(14), i37–i48 (2017)
Haug, P.J., Christensen, L.M., Gundersen, M.L., Clemons, B., Koehler, S.B., Bauer, K.: A natural language parsing system for encoding admitting diagnoses, pp. 814–818 (1997)
Haug, P.J., Koehler, S., Lau, L.M., Wang, P., Rocha, R.A., Huff, S.M.: Experience with a mixed semantic/syntactic parser, pp. 284–288 (1995)
Howard, J., Ruder, S.: Universal language model fine-tuning for text classification, vol. 1, pp. 328–339 (2018)
Johnson, A.E.W., et al.: Mimic-iii, a freely accessible critical care database. Sci. Data 3(1), 160035 (2016)
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition, pp. 260–270 (2016)
Lample, G., Conneau, A.: Cross-lingual language model pretraining (2019). arXiv preprint arXiv:1901.07291
Lample, G., Conneau, A., Ranzato, M., Denoyer, L., Jegou, H.: Word translation without parallel data (2018)
Li, Z., Liu, F., Antieau, L.D., Cao, Y., Yu, H.: Lancet: a high precision medication event extraction system for clinical text. J. Am. Med. Inf. Assoc. 17(5), 563–567 (2010)
Meystre, S.M., Thibault, J., Shen, S., Hurdle, J.F., South, B.R.: Textractor: a hybrid system for medications and reason for their prescription extraction from clinical text documents. J. Am. Med. Inf. Assoc. 17(5), 559–562 (2010)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space (2013)
Mikolov, T., Le, Q.V., Sutskever, I.: Exploiting similarities among languages for machine translation (2013). arXiv Computation and Language
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality, pp. 3111–3119 (2013)
Ni, J., Dinu, G., Florian, R.: Weakly supervised cross-lingual named entity recognition via effective annotation and representation projection, vol. 1, pp. 1470–1480 (2017)
Patrick, J., Li, M.: High accuracy information extraction of medication information from clinical notes: 2009 i2b2 medication extraction challenge. J. Am. Med. Inf. Assoc. 17(5), 524–527 (2010)
Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation, pp. 1532–1543 (2014)
Peters, M.E., et al.: Deep contextualized word representations (2018). arXiv Computation and Language
Savova, G., et al.: Mayo clinical text analysis and knowledge extraction system (ctakes): architecture, component evaluation and applications. J. Am. Med. Inf. Assoc. 17(5), 507–513 (2010)
Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units (2015). arXiv Computation and Language
Smith, S.L., Turban, D.H.P., Hamblin, S., Hammerla, N.Y.: Offline bilingual word vectors, orthogonal transformations and the inverted softmax (2017). arXiv Computation and Language
Spasic, I., Sarafraz, F., Keane, J.A., Nenadic, G.: Medication information extraction with linguistic pattern matching and semantic rules. J. Am. Med. Inf. Assoc. 17(5), 532–535 (2010)
Taylor, W.L.: “cloze procedure": A new tool for measuring readability. Journalism Bull. 30(30), 415–433 (1953)
Tsai, C., Roth, D.: Cross-lingual wikification using multilingual embeddings, pp. 589–598 (2016)
Wang, Q., Xia, Y., Zhou, Y., Ruan, T., Gao, D., He, P.: Incorporating dictionaries into deep neural networks for the chinese clinical named entity recognition (2018). arXiv Computation and Language
Xu, K., Zhou, Z., Gong, T., Hao, T., Liu, W.: Sblc: a hybrid model for disease named entity recognition based on semantic bidirectional lstms and conditional random fields. BMC Med. Inf. Decis. Making 18(5), 114 (2018)
Yarowsky, D., Ngai, G., Wicentowski, R.: Inducing multilingual text analysis tools via robust projection across aligned corpora, pp. 1–8 (2001)
Zeng, Q.T., Goryachev, S., Weiss, S.T., Sordo, M., Murphy, S.N., Lazarus, R.: Extracting principal diagnosis, co-morbidity and smoking status for asthma research: evaluation of a natural language processing system. BMC Med. Inf. Decis. Making 6(1), 30–30 (2006)
Zhang, M., Liu, Y., Luan, H., Sun, M.: Adversarial training for unsupervised bilingual lexicon induction, vol. 1, pp. 1959–1970 (2017)
Zitouni, I., Florian, R.: Mention detection crossing the language barrier, pp. 600–609 (2008)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Ding, P. et al. (2020). Cross-Lingual Transfer Learning for Medical Named Entity Recognition. In: Nah, Y., Cui, B., Lee, SW., Yu, J.X., Moon, YS., Whang, S.E. (eds) Database Systems for Advanced Applications. DASFAA 2020. Lecture Notes in Computer Science(), vol 12112. Springer, Cham. https://doi.org/10.1007/978-3-030-59410-7_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-59410-7_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59409-1
Online ISBN: 978-3-030-59410-7
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)