Infusing Biomedical Knowledge into BERT for Chinese Biomedical NLP Tasks with Adversarial Training

Biomedical text mining is becoming increasingly important. Recently, biomedical pre-trained language models such as BioBERT and SciBERT, which can capture biomedical knowledge from text, have achieved promising results in biomedical NLP tasks. However, most biomedical pre-trained language models rely on the traditional masked language model (MLM) pre-training strategy, which cannot fully capture the semantic relations of context. It is challenging to learn biomedical knowledge via language models in the Chinese biomedical fields due to the lack of training resources and the extreme complexity and diversity of Chinese medical terminologies. To this end, we propose MedBERT-adv, which utilizes a biomedical knowledge infusion method that can effectively complement BERT-like models. Instead of using time-consuming medical expert annotation and inaccurate automatic annotation, we use the article structure in Baidu Encyclopedia as a weakly supervised signal, utilizing each medical term and its category as labels to pre-train the model. We also leverage adversarial training strategies like FGM for fine-tuning downstream tasks to further improve the performance of MedBERT-adv. We experimented with MedBERT-adv on the Chinese biomedical dataset CBLUE using eight NLP tasks. Among all of them, our proposed model obtained an average 1.8% improvement in average score than four baseline models, demonstrating the effectiveness of MedBERT-adv on Chinese biomedical text mining.