Abstract
Automatic term extraction is the process of identifying domain-specific terms in a text using automated algorithms and is a key first step in ontology learning and knowledge graph creation. Large language models have shown good few-shot capabilities, thus, in this paper, we present a study to evaluate the few-shot in-context learning performance of GPT-3.5-Turbo on automatic term extraction. To benchmark the performance we compare the results with fine-tuning of a BERT-sized model. We also carry out experiments with count-based term extractors to assess their applicability to few-shot scenarios. We quantify prompt sensitivity with experiments to analyze the variation in performance of large language models across different prompt templates. Our results show that in-context learning with GPT-3.5-Turbo outperforms the BERT-based model and unsupervised count-based methods in few-shot scenarios.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
Here we refer to models with more than 1B parameters as large language models.
- 3.
Specifically we used paraphrase-MiniLM-L6-v2.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
References
Astrakhantsev, N.A., Fedorenko, D.G., Turdakov, D.Y.: Methods for automatic term recognition in domain-specific text collections: a survey. Ph.D. thesis (2015). https://doi.org/10.1134/S036176881506002X
Brown, T.B., et al.: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 6–12 December 2020, Virtual. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.-F., Lin, H.-T. (eds.) (2020)
Cabré, M.T.: Terminology: Theory, Methods, and Applications, vol. 1. John Benjamins Publishing (1999)
Carreras, X., Mà rquez, L., Padró, L.: Proceedings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HLT-NAACL 2003, Edmonton, Canada, 31 May–1 June 2003. In: Daelemans, W., Osborne, M. (eds.), pp. 152–155. ACL (2003)
Clouet, E.L., Gojun, A., Blancafort, H., Guegan, M., Gornostay, T., Heid, U.: Reference lists for the evaluation of term extraction tools (2012)
Fowler, M.: Domain-Specific Languages. Pearson Education (2010)
Frantzi, K.T., Ananiadou, S.: 16th International Conference on Computational Linguistics, Proceedings of the Conference, COLING 1996, Center for Sprogteknologi, Copenhagen, Denmark, 5–9 August 1996, pp. 41–46 (1996)
Gutierrez, B.J., et al.: Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, 7–11 December 2022. In: Goldberg, Y., Kozareva, Z., Zhang, Y. (eds.), pp. 4497–4512. Association for Computational Linguistics (2022). https://doi.org/10.18653/V1/2022.FINDINGS-EMNLP.329
Hazem, A., Bouhandi, M., Boudin, F., Daille, B.: Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20–25 June 2022. In: Calzolari, N., et al. (eds.), pp. 648–662. European Language Resources Association (2022)
Hoste, V., Vanopstal, K., Terryn, A.R., Lefever, E.: The trade-off between quantity and quality. Comparing a large crawled corpus and a small focused corpus for medical terminology extraction. Across Lang. Cult. 20(2), 197–211 (2019)
Lang, C., Wachowiak, L., Heinisch, B., Gromann, D.: Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, 1–6 August 2021. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.), pp. 3607–3620. Association for Computational Linguistics (2021). https://doi.org/10.18653/V1/2021.FINDINGS-ACL.316
Li, X., Qiu, X.: Finding supporting examples for in-context learning. CoRR abs/2302.13539 (2023). https://doi.org/10.48550/ARXIV.2302.13539
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., Chen, W.: Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO@ACL 2022, Dublin, Ireland and Online, 27 May 2022. In: Agirre, E., Apidianaki, M., Vulic, I. (eds.), pp. 100–114. Association for Computational Linguistics (2022). https://doi.org/10.18653/V1/2022.DEELIO-1.10
Naveed, H., et al.: A comprehensive overview of large language models. CoRR abs/2307.06435 (2023). https://doi.org/10.48550/ARXIV.2307.06435
Perez, E., Kiela, D., Cho, K.: Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, 6–14 December 2021, Virtual. In: Ranzato, M., Beygelzimer, A., Dauphin, Y.N., Liang, P., Vaughan, J.W. (eds.), pp. 11054–11070 (2021)
Rigouts Terryn, A., Hoste, V., Drouin, P., Lefever, E.: Proceedings of the 6th International Workshop on Computational Terminology. In: Daille, B., Kageura, K., Rigouts Terryn, A. (eds.), pp. 85–94. European Language Resources Association (2020). ISBN: 979-10-95546-57-3
Rigouts Terryn, A., Hoste, V., Lefever, E.: HAMLET: hybrid adaptable machine learning approach to extract terminology. Terminology 27(2), 254–293 (2021)
Rubin, O., Herzig, J., Berant, J.: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, 10–15 July 2022. In: Carpuat, M., de Marneffe, M.-C., RuÃz, I.V.M. (eds.), pp. 2655–2671. Association for Computational Linguistics (2022). https://doi.org/10.18653/V1/2022.NAACL-MAIN.191
Sclar, M., Choi, Y., Tsvetkov, Y., Suhr, A.: Quantifying language models’ sensitivity to spurious features in prompt design or: how i learned to start worrying about prompt formatting. CoRR abs/2310.11324 (2023). https://doi.org/10.48550/ARXIV.2310.11324
Tang, L., et al.: Evaluating large language models on medical evidence summarization. NPJ Digit. Med. 6 (2023). https://doi.org/10.1038/S41746-023-00896-7
Wadhwa, S., Amir, S., Wallace, B.C.: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, 9-14 July 2023. In: Rogers, A., Boyd-Graber, J.L., Okazaki, N. (eds.), pp. 15566–15589. Association for Computational Linguistics (2023). https://doi.org/10.18653/V1/2023.ACL-LONG.868
Wang, S., et al.: GPT-NER: named entity recognition via large language models. CoRR abs/2304.10428 (2023).https://doi.org/10.48550/ARXIV.2304.10428
Zhang, M., Wang, B., Fei, H., Zhang, M.: In-context learning for few-shot nested named entity recognition. CoRR abs/2402.01182 (2024). https://doi.org/10.48550/ARXIV.2402.01182
Acknowledgement
Author Shubhanker Banerjee was supported by Science Foundation Ireland under Grant Agreement No. 13/RC/2106_P2 at the ADAPT SFI Research Centre at University Of Galway.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Banerjee, S., Chakravarthi, B.R., McCrae, J.P. (2024). Large Language Models for Few-Shot Automatic Term Extraction. In: Rapp, A., Di Caro, L., Meziane, F., Sugumaran, V. (eds) Natural Language Processing and Information Systems. NLDB 2024. Lecture Notes in Computer Science, vol 14762. Springer, Cham. https://doi.org/10.1007/978-3-031-70239-6_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-70239-6_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-70238-9
Online ISBN: 978-3-031-70239-6
eBook Packages: Computer ScienceComputer Science (R0)