KGEL: A novel end-to-end embedding learning framework for knowledge graph completion

Highlights

• Embedding models perform the link prediction task on the basis of individual triples.

• Gathering neighborhood information leads to richer node embeddings.

• A GCN based model is used to produce two embedding vectors of each node.

• A novel tensor factorization model predicts missing entities to derive new triples.

• The results prove that richer node embeddings lead to significant performance gain.

Abstract

Knowledge graphs (KGs) have recently become increasingly popular due to the broad range of essential applications in various downstream tasks including intelligent search, personalized recommendations, intelligent financial data analytics, etc. During an automated construction of a KG, the knowledge facts from multiple knowledge sources are automatically extracted in the form of triples, and these observed triples are used to derive new unobserved triples for KG completion (also known as link prediction). State-of-the-art link prediction methods are known to be primarily KG embedding models, among which tensor factorization models have recently drawn much attention due to their scalability and expressive feature embeddings, and hence, perform well for link prediction. However, these embedding models consider each KG triple individually and fail to capture the useful information present in the neighborhood of a node. To this end, we propose a novel end-to-end KG embedding learning framework that consists of an encoder of a dual weighted graph convolutional network, and a decoder of a novel fully expressive tensor factorization model. The proposed encoder extends weighted graph convolutional network to generate two rich and high quality embedding vectors for each node by aggregating information from the neighboring nodes. The proposed decoder has a flexible and powerful tensor representation form of the Tensor Train decomposition that takes benefit of the two representations of each node in its embedding space to accurately model the KG triples. We also derive a bound on the size of the embeddings for full expressivity and show that our proposed tensor factorization model is fully expressive. Additionally, we show the relationship of our tensor factorization model to previous tensor factorization models. The experimental results show the effectiveness of the proposed framework that consistently marks performance gains over several previous models on recent standard link prediction datasets.

Introduction

Expert and intelligent systems perform decision making on the basis of knowledge from multiple sources. An effective knowledge representation is important to integrate and organize knowledge in a form that can easily be explored and interpreted by these intelligent systems (Martinez et al., 2018, Nickel et al., 2015). Knowledge graphs (KGs) provide a semantically structured graphical representation of knowledge facts, where the factual information is represented as triples of the form $(e_s,r,e_o)$, where $e_s,e_o$ are the subject and object entities, represented by nodes, and $\mathit{r}$ is the relationship between entities, represented by an edge between the nodes. KGs provide an effective way to extract and integrate structural information of entities from diverse knowledge sources at large scale, link these entities together into flexible ontologies which can easily be extended with new information, and can effectively be illustrated that how the entities are semantically interconnected so that the intelligent systems can effectively explore and interpret knowledge for sufficient inference (Chen et al., 2020, Zhao et al., 2017). Various companies have created KGs for performance improvement of intelligent systems including search engines, conversational agents, data analytic systems, etc. For instance, eBay’s conversational agent (Pittman, Srivastava, Hewavitharana, Kale, & Mansour, 2017) uses KG to encode product details and customer shopping behavior to find the relevant products based on customer queries. Bloomberg (Edgar, 2019) uses KG to encode tweets and news reports to empower their intelligent data analytic system to detect emerging events that may affect stock values (Hogan et al., 2020).

KGs can also be integrated with various expert and intelligent systems to provide new insights for decision making. For example, a relational data based intelligent financial fraud detection system may not identify a potential fraudulent customer by features, however, a KG based system can identify the same customer (entity) as fraudulent if connected with another fraudulent customer (entity) through a relation such as phone number/email address/IP address which suggests that both customers are same. Similarly, KGs can be integrated in intelligent automatic driving systems to find similarity between locations (entities) by using information from the distance (relation) to guide a vehicle to the potential targeted location. Moreover, KGs can be used in intelligent investment advisers to identify similarity between features of a new and existing customers (entities) connected by some relation such as financial status to recommend same investment plan to the new customer. Furthermore, in medical expert systems, KGs can be used to identify a relation such as high temperature to connect all diseases with high temperature which can consequently advise to avoid some medicines that are forbidden at high temperature.

Several previous studies (Chen et al., 2018, Kertkeidkachorn and Ichise, 2018, Malik et al., 2020, Zhao et al., 2017) have attempted automated construction of KGs from various information sources. These automated KG construction approaches mainly perform automated extraction of triples, however, they do not focus on deriving new unobserved triples based on the extracted ones as done by Alobaidi, Malik, and Hussain (2018) for automated ontology generation. More precisely, it is not only important to automatically extract entities and relations, and link them into a semantically structured representation, but also to discover existing entities that have relations with other existing entities to derive new unobserved triples for improving and maximizing knowledge with the ultimate goal of KG completion to provide new insights for interpretation and sufficient inference. Thus, KG completion methods can be considered an integral part of the automated KG construction approaches and an important prerequisite for the development of more intelligent systems. KG completion methods predict new unobserved triples based on the existing ones for KG completion. In particular, link prediction is a KG completion task, where the object $(e_s,r,?)$ or subject entity $(?,r,e_o)$ is predicted to derive new triples.

For link prediction, several KG embedding learning approaches have been proposed and gained significant attention. These KG embedding models can be broadly classified as translational distance models (Bordes et al., 2013, Ji et al., 2015, Lin et al., 2015, Sun et al., 2018), tensor factorization models (Nickel et al., 2016, Trouillon et al., 2016, Yang et al., 2015), and deep learning models (Dettmers et al., 2018, Dong et al., 2014, Nguyen et al., 2018). Given a KG, these models first embed the components of a KG including entities and relations into continuous vector space, and define a scoring function on each triple to measure its plausibility of being a valid fact (Wang, Mao, Wang, & Guo, 2017).

Among KG embedding models, tensor factorization models have gained most popularity due to their simplicity, scalability, expressivity, and significant performance. The key idea in tensor factorization approaches is to represent the triples $(e_s,r,e_o)$ as a 3rd order binary tensor $\mathbf{Y}$, and consider link prediction as a binary tensor completion task based on the low rank factorization (embeddings) to infer a predicted tensor $\mathbf{P}$ that approximates the given tensor $\mathbf{Y}$ (Trouillon et al., 2016). To do so, several tensor factorization approaches decompose a given tensor $\mathbf{Y}$ into a product of embedding matrices, resulting in r-dimensional vector and vector/matrix representation of entities and relations respectively. For a given triple $(e_s,r,e_o)$, the subject entity $e_s$ and object entity $e_o$ are linked together through relation $\mathit{r}$. Thus, the score is recovered through a bi-linear product between the embeddings of $e_s,r$ and $e_o$ (Trouillon et al., 2016).

Tensor factorization models perform the link prediction task solely on the basis of individual triples. They ignore the knowledge graph connectivity structure, and thus fail to embed useful information from the local neighborhood in the node embedding space. Weighted Graph Convolutional Networks (WGCNs) (Liu et al., 2019, Shang et al., 2019) have been effective tools for aggregating the local information from the neighborhood of each node to generate richer node embeddings. WGCNs learn to assign different weights to the edges connecting the neighboring nodes and simply aggregates all neighboring nodes to update the embeddings of a given node. However, KGs are one of the various data structures where nodes often exhibit different behaviors simultaneously depending upon their appearance as a subject or object entity in different triples. For example (Lion, eat, Wildcat) and (Wildcat, eat, Rabbit) are two triples from the animal food chain. In this case, the node Wildcat has two neighbors (Lion and Rabbit). However, in the former triple, Wildcat appears as an object entity where it exhibits a different behavior (e.g. prey) than the behavior (e.g. hunter) it exhibits in the later triple as a subject entity. Despite being effective tools for gathering neighborhood information, WGCNs may fail to differentiate between the behaviors of a node as an object or subject of a specific triple.

Motivated by the aforementioned observations, we propose KGEL, a novel end-to-end KG embedding learning framework. KGEL consists of an encoder of a dual weighted graph convolutional network (which we call DualWGCN), and a decoder of a novel fully expressive tensor factorization model (which we call TensorT), based on the Tensor Train decomposition (Oseledets, 2011). The proposed DualWGCN contains a clustering layer, followed by a graph embedding layer. The clustering layer splits the neighborhood of each node into two clusters, based on the appearance of a given node as a subject or object with respect to its neighboring nodes in the corresponding triples. These two clusters are assigned to two similar but independent WGCN models in the graph embedding layer to produce two embedding vectors of each node, one captures the behavior as a subject node, and the second captures the behavior as an object node.

As a decoder, our proposed TensorT defines a scoring function that distinguishes between the subject and object embeddings of an entity to take benefit from the two embedding vectors produced by the encoder for each entity, in contrast to many tensor factorization models, which do not consider any difference between the embeddings of an entity (i.e. consider a single embedding vector for each entity) whether it appears as an object or subject in a specific triple. Since a KG contains various types of relation patterns (e.g. symmetric and asymmetric), a representative embedding model should be fully expressive to model all the relation patterns. Thus, we further derive a dimensionality bound that guarantees fully expressiveness of TensorT.

Our specific contributions are summarized as follows:

• A novel KG embedding learning framework consisting of (1) an encoder: a DualWGCN that extends WGCN to produce two high quality embedding vectors of each entity, one as an object and another as a subject of the triples, and (2) a decoder: a novel fully expressive tensor factorization model TensorT, which exploits those embedding vectors for predicting missing entities.

• Deriving a bound on the size of the embeddings for fully expressiveness and show that TensorT is fully expressive.

• Theoretical analysis to show how the two existing representative models DistMult and ComplEx can be considered as special cases of our TensorT.

• Evaluating the proposed framework on three recent standard link prediction datasets to demonstrate the relative improvements over several state-of-the-art link prediction models.

The rest of the paper is organized in the following sections. Related work and the proposed framework are discussed in Sections 2 Related work, 3 Methodology respectively, followed by the theoretical insights, experimental results, and concluding remarks in Sections 4, 5 and 6 respectively.