Abstract
Privacy protection is an essential issue in biomedical natural language processing (BioNLP). Recently, some researchers apply federated learning (FL) in BioNLP to protect the privacy of biomedical data. However, their methods are only applicable for small NLP models, whose effectiveness is heavily limited in processing biomedical data. In this paper, we propose a novel memetic federated learning framework named Mem-Fed, which is tailored for federated learning of large-scale NLP models in the biomedical scenario. Experiments with large-scale BioNLP model on the public dataset show that the proposed framework significantly outperforms the state-of-the-art counterparts.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alicante, A., Corazza, A., Isgrò, F., Silvestri, S.: Unsupervised entity and relation extraction from clinical records in Italian. Comput. Biol. Med. 72, 263–275 (2016)
Baker, S., et al.: Automatic semantic classification of scientific literature according to the hallmarks of cancer. Bioinform. 32(3), 432–440 (2016)
Boughorbel, S., Jarray, F., Venugopal, N., Moosa, S., Elhadi, H., Makhlouf, M.: Federated uncertainty-aware learning for distributed hospital EHR data. CoRR abs/1910.12191 (2019)
Brisimi, T.S., Chen, R., Mela, T., Olshevsky, A., Paschalidis, I.C., Shi, W.: Federated learning of predictive models from federated electronic health records. Int. J. Med. Inform. 112, 59–67 (2018)
Cotta, C., Mathieson, L., Moscato, P.: Memetic algorithms. In: MartÃ, R., Pardalos, P.M., Resende, M.G.C. (eds.) Handbook of Heuristics, pp. 607–638 (2018)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186, June 2019
Gehrmann, S., et al.: Comparing rule-based and deep learning models for patient phenotyping. arXiv preprint arXiv:1703.08705 (2017)
Hafiane, W., Legrand, J., Toussaint, Y., Coulet, A.: Experiments on transfer learning architectures for biomedical relation extraction. CoRR abs/2011.12380 (2020)
Ju, C., et al.: Privacy-preserving technology to help millions of people: federated prediction model for stroke prevention. CoRR abs/2006.10517 (2020)
Lee, J., et al.: BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4), 1234–1240 (2020)
Liu, D., Dligach, D., Miller, T.: Two-stage federated phenotyping and patient representation learning. In: Proceedings of the 18th BioNLP Workshop and Shared Task, pp. 283–291, August 2019
Liu, D., Miller, T.A.: Federated pretraining and fine tuning of BERT using clinical notes from multiple silos. CoRR abs/2002.08562 (2020)
Liu, D., Miller, T.A., Sayeed, R., Mandl, K.D.: FADL: federated-autonomous deep learning for distributed electronic health record. CoRR abs/1811.11400 (2018)
McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Singh, A., Zhu, X.J. (eds.) Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, Fort Lauderdale, FL, USA, 20–22 April 2017. Proceedings of Machine Learning Research, vol. 54, pp. 1273–1282 (2017)
Moscato, P., et al.: On evolution, search, optimization, genetic algorithms and martial arts: towards memetic algorithms. Caltech concurrent computation program, C3P Report 826, 1989 (1989)
Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., Eisenstein, J.: Explainable prediction of medical codes from clinical text. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), June 2018
Ormerod, M., MartÃnez-del Rincón, J., Robertson, N., McGuinness, B., Devereux, B.: Analysing representations of memory impairment in a clinical notes classification model. In: Proceedings of the 18th BioNLP Workshop and Shared Task, pp. 48–57 (2019)
Peng, Y., Yan, S., Lu, Z.: Transfer learning in biomedical natural language processing: an evaluation of BERT and ELMo on Ten benchmarking datasets. In: Proceedings of the 18th BioNLP Workshop and Shared Task, pp. 58–65, August 2019
Peters, M., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2227–2237, June 2018
Schumacher, E., Mulyar, A., Dredze, M.: Clinical concept linking with contextualized neural representations. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8585–8592, July 2020
Shin, H.C., et al.: BioMegatron: larger biomedical domain language model. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4700–4706, November 2020
Tan, C., Jiang, D., Peng, J., Wu, X., Xu, Q., Yang, Q.: A de novo divide-and-merge paradigm for acoustic model optimization in automatic speech recognition. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3709–3715 (2020)
Tang, B., Cao, H., Wu, Y., Jiang, M., Xu, H.: Recognizing clinical entities in hospital discharge summaries using structural support vector machines with word representation features. BMC Med. Inform. Decis. Mak. 13, 1–10 (2013). BioMed Central
Vaidhya, T., Kaushal, A.: IITKGP at W-NUT 2020 shared task-1: domain specific BERT representation for named entity recognition of lab protocol. In: Xu, W., Ritter, A., Baldwin, T., Rahimi, A. (eds.) Proceedings of the Sixth Workshop on Noisy User-Generated Text, W-NUT@EMNLP 2020 Online, 19 November 2020, pp. 268–272 (2020)
Zhu, H., Jin, Y.: Multi-objective evolutionary federated learning. IEEE Trans. Neural Netw. Learn. Syst. 31(4), 1310–1322 (2019)
Acknowledge
This work was supported by National Natural Science Foundation of China (61872338) and the Foundation of Guizhou Provincial Key Laboratory of Public Big Data (No. 2019BDKFJJ002).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, X. et al. (2021). Memetic Federated Learning for Biomedical Natural Language Processing. In: Wang, L., Feng, Y., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2021. Lecture Notes in Computer Science(), vol 13029. Springer, Cham. https://doi.org/10.1007/978-3-030-88483-3_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-88483-3_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88482-6
Online ISBN: 978-3-030-88483-3
eBook Packages: Computer ScienceComputer Science (R0)
