TigeCMN: On exploration of temporal interaction graph embedding via Coupled Memory Neural Networks

Abstract

With the increasing demand of mining rich knowledge in graph structured data, graph embedding has become one of the most popular research topics in both academic and industrial communities due to its powerful capability in learning effective representations. The majority of existing work overwhelmingly learn node embeddings in the context of static, plain or attributed, homogeneous graphs. However, many real-world applications frequently involve bipartite graphs with temporal and attributed interaction edges, named temporal interaction graphs. The temporal interactions usually imply different facets of interest and might even evolve over the time, thus putting forward huge challenges in learning effective node representations. Furthermore, most existing graph embedding models try to embed all the information of each node into a single vector representation, which is insufficient to characterize the node’s multifaceted properties. In this paper, we propose a novel framework named TigeCMN to learn node representations from a sequence of temporal interactions. Specifically, we devise two coupled memory networks to store and update node embeddings in the external matrices explicitly and dynamically, which forms deep matrix representations and thus could enhance the expressiveness of the node embeddings. Then, we generate node embedding from two parts: a static embedding that encodes its stationary properties and a dynamic embedding induced from memory matrix that models its temporal interaction patterns. We conduct extensive experiments on various real-world datasets covering the tasks of node classification, recommendation and visualization. The experimental results empirically demonstrate that TigeCMN can achieve significant gains compared with recent state-of-the-art baselines.

Introduction

Accompanied by the proliferation of graph-structured data, graph embedding has attracted a surge of research attention in recent years (Grover and Leskovec, 2016, Li, et al., 2020, Liu, et al., 2020, Perozzi et al., 2014, Tang, et al., 2015, Wang et al., 2016). By learning a low-dimensional vector representation for each node, graph embedding provides an effective way to mine the knowledge in graphs and benefits various downstream tasks such as node classification (Bhagat, Cormode, & Muthukrishnan, 2011), link prediction (Li, et al., 2019, Lü and Zhou, 2011) and graph clustering (Zhou, Cheng, & Yu, 2009), etc. Recent graph embedding methods, such as DeepWalk (Perozzi et al., 2014), LINE (Tang, et al., 2015) and node2vec (Grover & Leskovec, 2016), mainly focus on static graphs by exploring various contextual information. However, in reality, the complicated graph structure is not formed in one move, which is essentially a dynamic process driven by adding nodes and edges sequentially (Zuo, et al., 2018). The observed static graph structure is the accumulation of node neighborhood in certain time periods, thus these aforementioned approaches inevitably overlook the temporal interaction information among different nodes.

In general, graphs in many real-world applications are inherently temporal interaction graphs, and examples range from recommender systems (Xiang, et al., 2010, Ying et al., 2020), question answering systems (Hickl et al., 2006, Qu, et al., 2020) to search engines (Beeferman & Berger, 2000). Temporal interaction graph is a special form of bipartite graph, which involves two types of entities with edges annotated by chronological interaction events. Fig. 1 shows a simplified temporal interaction graph with four user nodes and four item nodes. The edges in the graph indicate the interactions between two heterogeneous nodes, and the attributes of an edge represent the content of the interaction, which could provide rich context information. For example, in public review websites like Yelp,¹ users and businesses naturally form a temporal interaction graph, where the edges indicate users’ comment behaviors on business entities, and the comment information may contain photos in addition to a plain text review.

Despite the temporal interaction graphs pervasively existing in various domains, the studies on embedding these graphs are still rather limited. The temporal and heterogeneous settings pose great challenges for most current graph embedding models, as they are specifically designed for static and homogeneous graphs. Following the pioneering work of DeepWalk (Perozzi et al., 2014), these methods typically perform random walks on the graph to generate a corpus of nodes, which are then fed into skip-gram model (Mikolov, Chen, Corrado, and Dean, 2013, Mikolov, Sutskever, Chen, Corrado, and Dean, 2013) to learn low-dimensional embedding vectors. Although these algorithms can be applied to the temporal interaction graphs by ignoring the temporal interaction information and node types, we argue that they may result in less powerful node representations due to the following reasons: (1) They fail to capture the temporal interaction patterns (e.g., user multifaceted preferences and item popularity trends). For example, a researcher may focus on one particular field like data mining algorithms in the beginning, thus mainly submitting his/her papers to the KDD and ICDM conferences. But when deep learning becomes more and more popular in the research community, he/she may shift his/her research interest to the deep learning techniques, thus his/her published papers may primarily appear in NeurIPS and ICLR conferences. Hence, it is important to capture its dynamic and multifaceted patterns. (2) They fail to differentiate the heterogeneous node types and cannot utilize the informative edge attributes. In particular, the interactions between heterogeneous nodes usually form complex influences and dependencies among them, thus they should be treated differently. Moreover, different from attributed graphs (Li, et al., 2017, Zhang, et al., 2018), the attributes here are associated with each edge instead of each node. The attributes on an edge could also be heterogeneous and involve categorical numbers, texts or even images. For instance, after finishing dining at a restaurant, users might submit some reviews as well as the food’s pictures in that restaurant to the Yelp website, which might reflect their personal preferences. Therefore, it is necessary for us to incorporate the informative edge attributes into node representations, because it is hard to capture the semantic meanings by only exploring the graph structure.

To tackle the aforementioned challenges, we propose a novel model named Temporal interaction graph embedding via Coupled Memory Neural Networks (abbreviated as TigeCMN). The illustrative comparison between traditional method and our proposed TigeCMN is shown in Fig. 2. Instead of performing random walks like DeepWalk (Perozzi et al., 2014) and node2vec  (Grover & Leskovec, 2016), we utilize the graph’s inherent temporal information and model nodes’ interactions according to their sequential orders. To capture node’s multifaceted properties and temporal interaction patterns, we leverage the recent advances in memory networks (Graves et al., 2014, Sukhbaatar et al., 2015, Weston et al., 2014), which introduce external matrices to dynamically store and update its obtained knowledge. The proposed TigeCMN framework consists of two major components: augmented memory matrices to store the interaction information and controller networks to manipulate the memories. Similar to the physical computers, the augmented memories allow writing and reading operations to erase old information and add new information, which increase its capability of tracing the dynamic knowledge (Zhang, Shi, King, & Yeung, 2017). Specifically, we introduce an external matrix for each node to maintain its temporal interaction information instead of simply compressing its multifaceted interactions into a single latent vector. When performing inference, the memory matrix will be attentively readout based on the attention mechanism (Vaswani, et al., 2017), which summarizes the temporal interaction patterns stored in the memory and forms a dynamic node embedding. Meanwhile, nodes’ IDs with one-hot format are transformed into dense vectors to represent its stationary properties. To capture its stationary and dynamic properties, the final node representations are the nonlinear transformation of the static and dynamic embeddings.

To summarize, the main contributions of this paper are as follows:

• We design a novel graph embedding algorithm based on memory networks in the scenario of temporal interaction graph. The proposed TigeCMN has the capability of handling heterogeneous node types, edge attributes and temporal node interaction sequences.

• Instead of compressing the node’s multifaceted interactions into a single latent vector, we utilize external memory matrices to store and manipulate node embeddings in a more explicit and dynamic manner, which enhances the expressiveness of the node representations.

• We perform comprehensive experiments on five datasets to verify the effectiveness of TigeCMN including node classification, recommendation and visualization tasks. The results empirically show its superiority compared to a range of state-of-the-art baselines.

The rest of this paper is organized as follows. In Section 2, we briefly review the related work on graph embedding methods and memory augmented neural networks. Section 3 introduces the proposed TigeCMN model. We present our experimental evaluation and comparison against state-of-the-art methods in Section 4. Finally, we draw our conclusions and discuss some potential future research directions in Section 5.