A Language Model Based Framework for New Concept Placement in Ontologies

Abstract

We investigate the task of inserting new concepts extracted from texts into an ontology using language models. We explore an approach with three steps: edge search which is to find a set of candidate locations to insert (i.e., subsumptions between concepts), edge formation and enrichment which leverages the ontological structure to produce and enhance the edge candidates, and edge selection which eventually locates the edge to be placed into. In all steps, we propose to leverage neural methods, where we apply embedding-based methods and contrastive learning with Pre-trained Language Models (PLMs) such as BERT for edge search, and adapt a BERT fine-tuning-based multi-label Edge-Cross-encoder, and Large Language Models (LLMs) such as GPT series, FLAN-T5, and Llama 2, for edge selection. We evaluate the methods on recent datasets created using the SNOMED CT ontology and the MedMentions entity linking benchmark. The best settings in our framework use fine-tuned PLM for search and a multi-label Cross-encoder for selection. Zero-shot prompting of LLMs is still not adequate for the task, and we propose explainable instruction tuning of LLMs for improved performance. Our study shows the advantages of PLMs and highlights the encouraging performance of LLMs that motivates future studies.

Appendices

Appendix 1: Detailed model settings and time usage

The approaches are implemented using PyTorch and Huggingface Transformers. Edge-Bi-encoder and Edge-Cross-encoder are originally based on the architectures of BLINKout [8] (based on BLINK [31]). Inverted index with ontology concepts is based on DeepOnto Library [15]. The batch sizes for Edge-Bi-encoder and Edge-Cross-encoder are 16 and 1, resp. The fine-tuning of Edge-Bi-encoder and Edge-Cross-encoder takes 1 and 4 epochs, resp. We limit the rows to 200,000 for training the Edge-Cross-encoder models given the sufficient amount the data for model convergence and the long time of training. The instruction tuning of Llama-2-7B uses a 4-bit quantisation and takes 3 epochs with a batch size of 4.

Time Usage. We run all models using an NVIDIA Quadro RTX 8000 GPU card (48GB GPU). We report the time usage estimate for MM-S14-Disease under the top-50 setting. Training bi-encoder took around 29 h. Training cross-encoder took around 4 h. Instruction tuning of Llama-2-7B took around 16 h. Inferencing with fixed embeddings and inverted index with edge enrichment is within around 0.5 and 1 s per mention, resp. Inferencing with Edge-Bi-encoder only takes around 0.2 s per mention. The whole inferencing with both Edge-Bi-encoder and Edge-Cross-encoder takes around 2.3 s per mention. The prompting of an explainable instruction-tuned Llama-2-7B model takes around 78 s per mention to output natural language explanations.

Appendix 2: Detailed results on edge enrichment

We applied edge formation enrichment over inverted index and fixed embedding approach. Results in Table 4 show a substantial improvement for $InR_{any}$ and $InR_{all}$. We see that the mentions to be placed to non-leaf edges are not improved with inverted index and fixed embeddings, but are improved with the fine-tuned, Edge-Bi-encoder, this is because the latter places a more lenient score for the leaf edges that do not always rank them before the non-leaf edges.

Table 4. Results on edge search and enrichment (vs. not using edge enrichment) for MM-S14-Disease, under the top-50 setting. Each setting has validation and testing results, separated by a slash (/) sign. “lf” and “nlf” mean leaf and non-leaf, resp.

Appendix 3: Qualitative examples

Examples of a non-leaf and a leaf concept placement, with prompt options, model predictions, and instruction-tuned Llama-2-7B’s explanations, are in Table 5.

Table 5. Examples of two mentions in the out-of-KB test set of MM-S14-Disease to enrich SNOMED CT 2014.09. The correct predictions are in bold. (Note: while the concept Chronic kidney disease in SNOMED CT ver 2017.03 is not in ver 2014.09, it is modified from Chronic renal impairment, ID 236425005, in the older ontology.)