GRL: Knowledge graph completion with GAN-based reinforcement learning

Abstract

Knowledge graph completion intends to infer the entities that need to be queried through the entities and relations known in the knowledge graphs. It is used in many applications, such as question and answer systems, and searching engines. As the completion process can be represented as a Markov process, existing works would solve this problem with reinforcement learning. However, there are three issues blocking them from achieving high accuracy, which are reward sparsity, missing specific domain rules, and ignoring the generation of knowledge graphs. In this paper, we design a generative adversarial net (GAN)-based reinforcement learning model, named GRL, for knowledge graph completion. First, GRL employs the graph convolutional network to embed the knowledge graphs into the low-dimensional space. Second, GRL employs both GAN and long short-term memory (LSTM) to record trajectory sequences obtained by the agent from traversing the knowledge graph and generate new trajectory sequences if needed. At the same time, GRL applies domain-specific rules accordingly. Finally, GRL employs the deep deterministic policy gradient method to optimize both rewards and adversarial loss. The experiments show that GRL is able to both generate better policies and outperform traditional methods for several tasks.

Introduction

Knowledge graph is designed to represent the various entities or concepts and their relations existing in the real world. It constitutes a huge semantic network graph in which nodes represent entities or concepts, and edges represent attributes or relations. Knowledge graph has been used to represent a variety of large-scale knowledge bases. However, the real-world knowledge bases are usually incomplete [1], [2], one needs to complete it by inferring the missing ones with the help of existing information. After that, one can apply the completed knowledge base to real tasks, such as question and answer and searching engines. This knowledge base completion process is known as the knowledge graph completion problem [3], [4], [5].

Existing knowledge graph completion methods can be classified into three types: (1) Path ranking algorithm (PRA) based [6]. They predict the potential relations between entities by connecting existing paths of entities; (2) Representation learning based. They first map the entities and relations into the low-dimensional space, and then reason through vector operations, e.g., TransE [7]; and (3) Probability graph based [8] includes the Markov logic networks and their derivatives, e.g., reinforcement learning methods [9]. These methods have been shown to perform well in their respective areas of expertise.

However, most existing methods did not consider the fact that the knowledge graph can generate new sub-graphs (including nodes and edges). The previous reinforcement learning methods assume that the agent can find the target entity in the knowledge graph within limited time steps [10], [11], [12]. However, the target entity may be missing and thus requires completion. For example, the question “who is the current president of USA?” is formalized as the triple (“USA”, “president of”, ?). If there is a complete triplet (“USA”, “president of”, “Trump”) in the knowledge graph, the agent will find the target entity “Trump” easily. However, if the target entity is missing in the knowledge graph, how could the agent find the correct answer? Inspired by the graph generation techniques [13], we can generate the sub-graphs when the target entity is not found in the known knowledge graph, and let the agent select the correct edges and target entity. Knowledge graph completion is essentially a problem of generating missing parts based on the existing. Compared with the previous methods, we also inferred the missing parts with the help of existing ones in the knowledge graph, but we use the graph generation techniques to strengthen the generation ability and optimize the reward so that the agent can find the target entities or missing entities more intelligently. Further, in existing reinforcement learning based models, the sampling method faces large variance and relatively low learning efficiency, and thus faces the challenges of fragile convergence and sparse rewards.

Deep neural network has shown its great flexibility and power by learning to represent the world as a nested hierarchy of concepts, achieving excellent results in lots of applications [14]. It should be pointed out that when using deep neural networks, very little work directly uses a single neural network to encode the graph structure, but combines multiple different forms of neural networks to enhance the capabilities of the entire network architecture. Motivated by these, in this paper, we propose a new framework based on GAN [15] and reinforcement learning for knowledge graph completion, named GRL. We divide the knowledge graph completion problem into two parts: first, the target entity can be found within the limited time step from the original knowledge graph; second, the target entity cannot be found from the original knowledge graph while there are still time steps left, one can generate a new sub-graph as shown in Fig. 1. In other words, when the agent did not find an answer within the time steps, it can generate a new path at the right time, which refers to the time when the agent thinks that generating a new path or node is more conducive to find the target entity based on the rules. This mechanism is more intelligent than the previous work. As reinforcement learning method learns policies through interaction with the environment to guide the agent to walk in the knowledge graph [10], [11], it is a good fit for sequence decision-making problems. As a result, we define the knowledge graph completion problem as a Markov process, and explore the rules that can be introduced in both state transition process and rewards, which can better guide the walking path under the optimization of GAN. We employ long short-term memory (LSTM) [16] as a generator of GAN, which not only records historical trajectories but also generates new sub-graphs, and trains policy networks with GAN. Further, to better generate new sub-graphs, a graph neural network (GNN) [17] is used to embed the knowledge graph into low-dimensional vectors and parameterize the message passing process at each layer. Finally, we employ DDPG [18] to optimize the entire model.

To summarize, our main contributions are:

• We are the first to introduce the graph generation techniques to knowledge graph reasoning to help agents better find the target entity.

• We are the first to use GAN to optimize reward in the knowledge graph completion problem, in which the introduced rules are optimized to guide the agent’s walking path and combined with trajectory generation to train the policy network.

• DDPG is employed in the framework to solve the problems of sparse rewards and low sampling efficiency brought by REINFORCE in previous work. The experimental results show the efficiency and effectiveness of our model.