Abstract
Identifying causal precedence relations among chemical interactions in biomedical literature is crucial for comprehending the underlying biological mechanisms. However, several issues persist, including the scarcity of labeled data, the complexity of domain transfer, and limited computing resources in this field. To tackle these challenges, we present a novel approach called Prompt-Ensemble Knowledge Distillation (PEKD). The PEKD model employs a BERT encoder combined with prompt templates to extract causal relationships between events. Additionally, model compression is achieved through a knowledge distillation framework that incorporates loss function regularization constraints, reducing resource overhead and computational time. To enhance the performance of knowledge distillation, an ensemble method with multiple teachers is utilized. Experimental results demonstrate that the proposed approach achieves a significant improvement in macro-F1 compared to the direct distillation methods. Importantly, it exhibits commendable performance when trained on few-shot datasets and compact models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Li, Z., Li, Q., Zou, X., Ren, J.: Causality extraction based on self-attentive BiLSTM-CRF with transferred embeddings. Neurocomputing 423, 207–219 (2021)
Rodchenkov, I., et al.: Pathway commons 2019 update: integration, analysis and exploration of pathway data. Nucleic Acids Res. 48(D1), D489–D497 (2020)
Frisoni, G., Moro, G., Carbonaro, A.: A survey on event extraction for natural language understanding: riding the biomedical literature wave. IEEE Access 9, 160721–160757 (2021)
Kang, H., et al.: TSPNet: translation supervised prototype network via residual learning for multimodal social relation extraction. Neurocomputing 507, 166–179 (2022)
Liang, Z., Noriega-Atala, E., Morrison, C., Surdeanu, M.: Low resource causal event detection from biomedical literature. In: Proceedings of the 21st Workshop on Biomedical Language Processing, Dublin, Ireland, pp. 252–263 (2022)
Chen, Y., Xu, L., Liu, K., Zeng, D., Zhao, J.: Event extraction via dynamic multi-pooling convolutional neural networks. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China, pp. 167–176 (2015)
Zeng, D., Liu, K., Lai, S., Zhou, G., Zhao, J.: Relation classification via convolutional deep neural network. In: Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin, Ireland, pp. 2335–2344 (2014)
Zheng, S., et al.: Joint entity and relation extraction based on a hybrid neural network. Neurocomputing 257, 59–66 (2017)
Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 (2018)
Wang, Y.-X., Hebert, M.: Learning from small sample sets by combining unsupervised meta-training with CNNs. In: The 30th Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 244–252 (2016)
Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al.: Matching networks for one shot learning. In: The 30th Conference on Neural Information Processing Systems, Barcelona, Spain, vol. 29 (2016)
Gao, X., Zhao, Y., Dudziak, Ł., Mullins, R., Xu, C.-Z.: Dynamic channel pruning: feature boosting and suppression. arXiv preprint arXiv:1810.05331 (2018)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
Mirza, P.: Extracting temporal and causal relations between events. In: Proceedings of the ACL 2014 Student Research Workshop, Baltimore, Maryland, pp. 10–17 (2014)
Hahn-Powell, G., Bell, D., Valenzuela-Escárcega, M.A., Surdeanu, M.: This before that: causal precedence in the biomedical domain. arXiv preprint arXiv:1606.08089 (2016)
Li, X., Wang, W., Fang, J., Jin, L., Kang, H., Liu, C.: PEINet: joint prompt and evidence inference network via language family policy for zero-shot multilingual fact checking. Appl. Sci. 12(19), 9688–9709 (2022)
Scao, T.L., Rush, A.M.: How many data points is a prompt worth?. arXiv preprint arXiv:2103.08493 (2021)
Acknowledgements
The work is supported by the National Natural Science Foundation of China (62206267).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, X., Liu, H., Jin, L., Li, G., Guan, S. (2024). PEKD: Joint Prompt-Tuning and Ensemble Knowledge Distillation Framework for Causal Event Detection from Biomedical Literature. In: Tan, Y., Shi, Y. (eds) Data Mining and Big Data. DMBD 2023. Communications in Computer and Information Science, vol 2017. Springer, Singapore. https://doi.org/10.1007/978-981-97-0837-6_10
Download citation
DOI: https://doi.org/10.1007/978-981-97-0837-6_10
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-0836-9
Online ISBN: 978-981-97-0837-6
eBook Packages: Computer ScienceComputer Science (R0)