DynHEN: A heterogeneous network model for dynamic bipartite graph representation learning

Abstract

Learning node representations in graphs is a widespread problem in node classification and link prediction. Current research has focused on static heterogeneous, homogeneous networks, and dynamic homogeneous networks. However, many existing graphs, such as citation and social networks, are heterogeneous. Therefore, it is still a great challenge to obtain heterogeneous information in a dynamic heterogeneous graph to assist node representation learning. In this paper, we propose a Dynamic HEterogeneous Network (DynHEN) for user-item bipartite networks. It is a discrete dynamic graph neural network model that can be used directly for node representation learning by utilizing dynamic heterogeneous graphs. Specifically, DynHEN takes a bipartite graph at each time step as input, gets the corresponding embedding by capturing the deep heterogeneous information of the nodes while fusing the temporal information. To illustrate the effectiveness of heterogeneous information for graph representation learning, we compare with the current SOTA methods in the two type experiments of link prediction and node classification, and achieve outstanding results.

Introduction

Representation learning of nodes in graphs has recently attracted extensive research interest and gradually evolved into a crucial problem. This technique has been widely used in various fields such as biology [1], recommendation [2], [3], [4], road condition prediction [5], and semantic analysis [6]. The graphs that people encounter in real applications change over time and have various node types, called dynamic heterogeneous graphs. For example, the continuous emergence of scientific articles and novel techniques will increase both the complexity and scale of the scientific citation network over time. Therefore, articles and scholars are always in a state of change, so the node embedding of articles and scholars needs to be updated ceaselessly. Similarly, statistics of the consumption platform show that the commodities clicked by users will continuously change with time as well as their preferences. The commodities will always emerge, and users will also keep changing. Vector representations of user preferences and product characteristics should also be updated to reflect this change accurately. These scenes widely occur in the real world. Many studies use a bipartite graph to model the above data and analyze the embedding of entities and relations, such as [7], [8]. Bipartite network is a special kind of heterogeneous graph, Fig. 1 is the bipartite network of the MovieLens dataset.¹ The ubiquitous dynamic bipartite network makes the representation learning of nodes in it more and more significant. Meanwhile, the dynamics and structural heterogeneity of dynamic networks provide rich information for node embedding.

Dynamic graph neural network (DGNN) is a kind of neural network structure that encodes a dynamic network and can be considered a network where nodes and edges appear and/or disappear over time [9]. The most important part of the structure is aggregating each node’s neighbor information in the dynamic graph. DGNN is usually an elaborate combination of a recursive neural network (RNN) and a graph neural network (GNN) [10], where the GNN captures the structural information in the graph, while RNN extracts the temporal information. The DGNN can be roughly divided into the discrete version and the continuous version [9]. Discrete DGNNs usually use an independent GNN to process each snapshot, and then feed each GNN output to a time series component. At the same time, the continuous DGNNs are usually a newly created model that parameterizes the graph, and the node embeddings vary according to changes in nodes and edges. There are many discrete DGNNs, such as RgCNN [11], which uses GCN to model graph topology, and LSTM [12] (a kind of RNN) to model the time dependency, DyGGCN [13], a model organizes GGNN [14] and LSTM, and DySAT [15], which chooses graph attention networks (GAT) [16] as the GNN and substitute Self-Attention for RNN. Therefore, GNN is the core component of DGNN.

However, most of the discrete DGNNs for dynamic bipartite networks convert the bipartite networks into homogeneous graphs and use the idea of discrete DGNNs to learn node embeddings. For example, DySAT [15] makes no distinction between node types in the bipartite graph. Instead, it uses the graph attention network model to capture the graph’s structure and extends it to temporal graph snapshots of the dynamic bipartite graph. Simultaneously, EvolveGCN [17], as a discrete DGNN, also treats all nodes equally and uses RNN evolutionary GCN parameters to extract the dynamic information of graph sequence. Although these methods possess higher computing efficiency, they ignore the ample heterogeneous information of the data itself, resulting in node embedding inadequacy in the bipartite network. Plenty of studies [18], [19] have proved the importance of heterogeneous information to node embedding in graphs. Heterogeneous information can help enhance node embedding while bringing more interpretability, especially for bipartite networks. The neighbor nodes of any node in the bipartite network are other types of nodes, as shown in Fig. 1. Thus, user node embedding with aggregated neighbor information can be interpreted as user preference embedding for heterogeneous information in a bipartite graph. In contrast, item node embedding can represent a certain type of user preference. Based on this observation, we intuitively argue that the heterogeneous information in the bipartite network is meaningful for node representation in the graph.

In view of the above problems, we propose DynHEN, a node embedding technique for dynamic bipartite networks in dynamic heterogeneous graphs, which can fully capture heterogeneous information while learning node embeddings in dynamic graphs. DynHEN consists of three main parts: HGCN layer, heterogeneous GAT layer, and temporal self-attention layer. Heterogeneous graphs contain abundant structural and semantic information. Meta-path can preserve the semantic features of heterogeneous graphs to the greatest extent [20]. So we first design HGCN, which uses the meta-path to obtain the neighbor nodes of different types and depths and then uses different weights to capture information of the nodes. GAT [16] can adaptively assign weights to the neighborhood information of nodes and can capture the structure information of graphs. Based on this, we design heterogeneous GAT to capture the structure information in heterogeneous graphs. In addition, dynamic graphs usually have periodic patterns. Self-attention [21] can extract context from all past graph snapshots to adaptively assign interpretable weights to previous time steps, so we use it to capture the dynamic changes of node features.

Then, the process of DynHEN is as follows. First, DynHEN selects an appropriate time step to represent the dynamic bipartite network as multiple bipartite network snapshots with this time step interval. Then, each snapshot uses the heterogeneous graph convolutional neural network (HGCN) to learn different types of node embeddings. Next, heterogeneous GAT is used to extract heterogeneous information further. Finally, we use the temporal self-attention to learn the representations of different nodes under each time step. In this way, we can capture the heterogeneous and dynamic information of the dynamic bipartite network. We perform extensive experiments on two tasks: link prediction and node classification, and verify the effectiveness of our proposed method. Our contributions can be summarized as follows:

• We propose DynHEN, a heterogeneous graph neural network model for dynamic bipartite networks belonging to discrete DGNNs, to utilize the heterogeneous information in bipartite graphs effectively. The model can comprehensively capture heterogeneous information in dynamic bipartite networks.

• We organically combine temporal information to learn dynamic node heterogeneous embedding when extracting heterogeneous information.

• We conduct link prediction experiments on three real datasets with different sparsity, and do node classification experiments on two publicly available datasets, which proves the effectiveness of our model.

The rest of this paper is structured as follows. In Section 2, we introduce some work related to our proposed model. In Section 3, we introduce our model in detail. In Section 4, we describe the design of experiments and the analysis of results. Section 5 is the summary of this paper.