TigeCMN: On exploration of temporal interaction graph embedding via Coupled Memory Neural Networks
Introduction
Accompanied by the proliferation of graph-structured data, graph embedding has attracted a surge of research attention in recent years (Grover and Leskovec, 2016, Li, et al., 2020, Liu, et al., 2020, Perozzi et al., 2014, Tang, et al., 2015, Wang et al., 2016). By learning a low-dimensional vector representation for each node, graph embedding provides an effective way to mine the knowledge in graphs and benefits various downstream tasks such as node classification (Bhagat, Cormode, & Muthukrishnan, 2011), link prediction (Li, et al., 2019, Lü and Zhou, 2011) and graph clustering (Zhou, Cheng, & Yu, 2009), etc. Recent graph embedding methods, such as DeepWalk (Perozzi et al., 2014), LINE (Tang, et al., 2015) and node2vec (Grover & Leskovec, 2016), mainly focus on static graphs by exploring various contextual information. However, in reality, the complicated graph structure is not formed in one move, which is essentially a dynamic process driven by adding nodes and edges sequentially (Zuo, et al., 2018). The observed static graph structure is the accumulation of node neighborhood in certain time periods, thus these aforementioned approaches inevitably overlook the temporal interaction information among different nodes.
In general, graphs in many real-world applications are inherently temporal interaction graphs, and examples range from recommender systems (Xiang, et al., 2010, Ying et al., 2020), question answering systems (Hickl et al., 2006, Qu, et al., 2020) to search engines (Beeferman & Berger, 2000). Temporal interaction graph is a special form of bipartite graph, which involves two types of entities with edges annotated by chronological interaction events. Fig. 1 shows a simplified temporal interaction graph with four user nodes and four item nodes. The edges in the graph indicate the interactions between two heterogeneous nodes, and the attributes of an edge represent the content of the interaction, which could provide rich context information. For example, in public review websites like Yelp,1 users and businesses naturally form a temporal interaction graph, where the edges indicate users’ comment behaviors on business entities, and the comment information may contain photos in addition to a plain text review.
Despite the temporal interaction graphs pervasively existing in various domains, the studies on embedding these graphs are still rather limited. The temporal and heterogeneous settings pose great challenges for most current graph embedding models, as they are specifically designed for static and homogeneous graphs. Following the pioneering work of DeepWalk (Perozzi et al., 2014), these methods typically perform random walks on the graph to generate a corpus of nodes, which are then fed into skip-gram model (Mikolov, Chen, Corrado, and Dean, 2013, Mikolov, Sutskever, Chen, Corrado, and Dean, 2013) to learn low-dimensional embedding vectors. Although these algorithms can be applied to the temporal interaction graphs by ignoring the temporal interaction information and node types, we argue that they may result in less powerful node representations due to the following reasons: (1) They fail to capture the temporal interaction patterns (e.g., user multifaceted preferences and item popularity trends). For example, a researcher may focus on one particular field like data mining algorithms in the beginning, thus mainly submitting his/her papers to the KDD and ICDM conferences. But when deep learning becomes more and more popular in the research community, he/she may shift his/her research interest to the deep learning techniques, thus his/her published papers may primarily appear in NeurIPS and ICLR conferences. Hence, it is important to capture its dynamic and multifaceted patterns. (2) They fail to differentiate the heterogeneous node types and cannot utilize the informative edge attributes. In particular, the interactions between heterogeneous nodes usually form complex influences and dependencies among them, thus they should be treated differently. Moreover, different from attributed graphs (Li, et al., 2017, Zhang, et al., 2018), the attributes here are associated with each edge instead of each node. The attributes on an edge could also be heterogeneous and involve categorical numbers, texts or even images. For instance, after finishing dining at a restaurant, users might submit some reviews as well as the food’s pictures in that restaurant to the Yelp website, which might reflect their personal preferences. Therefore, it is necessary for us to incorporate the informative edge attributes into node representations, because it is hard to capture the semantic meanings by only exploring the graph structure.
To tackle the aforementioned challenges, we propose a novel model named Temporal interaction graph embedding via Coupled Memory Neural Networks (abbreviated as TigeCMN). The illustrative comparison between traditional method and our proposed TigeCMN is shown in Fig. 2. Instead of performing random walks like DeepWalk (Perozzi et al., 2014) and node2vec (Grover & Leskovec, 2016), we utilize the graph’s inherent temporal information and model nodes’ interactions according to their sequential orders. To capture node’s multifaceted properties and temporal interaction patterns, we leverage the recent advances in memory networks (Graves et al., 2014, Sukhbaatar et al., 2015, Weston et al., 2014), which introduce external matrices to dynamically store and update its obtained knowledge. The proposed TigeCMN framework consists of two major components: augmented memory matrices to store the interaction information and controller networks to manipulate the memories. Similar to the physical computers, the augmented memories allow writing and reading operations to erase old information and add new information, which increase its capability of tracing the dynamic knowledge (Zhang, Shi, King, & Yeung, 2017). Specifically, we introduce an external matrix for each node to maintain its temporal interaction information instead of simply compressing its multifaceted interactions into a single latent vector. When performing inference, the memory matrix will be attentively readout based on the attention mechanism (Vaswani, et al., 2017), which summarizes the temporal interaction patterns stored in the memory and forms a dynamic node embedding. Meanwhile, nodes’ IDs with one-hot format are transformed into dense vectors to represent its stationary properties. To capture its stationary and dynamic properties, the final node representations are the nonlinear transformation of the static and dynamic embeddings.
To summarize, the main contributions of this paper are as follows:
- •
We design a novel graph embedding algorithm based on memory networks in the scenario of temporal interaction graph. The proposed TigeCMN has the capability of handling heterogeneous node types, edge attributes and temporal node interaction sequences.
- •
Instead of compressing the node’s multifaceted interactions into a single latent vector, we utilize external memory matrices to store and manipulate node embeddings in a more explicit and dynamic manner, which enhances the expressiveness of the node representations.
- •
We perform comprehensive experiments on five datasets to verify the effectiveness of TigeCMN including node classification, recommendation and visualization tasks. The results empirically show its superiority compared to a range of state-of-the-art baselines.
The rest of this paper is organized as follows. In Section 2, we briefly review the related work on graph embedding methods and memory augmented neural networks. Section 3 introduces the proposed TigeCMN model. We present our experimental evaluation and comparison against state-of-the-art methods in Section 4. Finally, we draw our conclusions and discuss some potential future research directions in Section 5.
Section snippets
Related work
In the following, we briefly discuss the related work closest to our problem setting from three perspectives: static graph embedding, temporal graph embedding and memory neural networks.
The proposed model
In this section, we first define some related concepts, notations and then present our problem formulation. After that, we introduce the overall framework of our proposed Temporal interaction graph embedding via Coupled Memory Neural Networks (TigeCMN), and then more details are presented in the following.
Experiments
Similarly to many previous work (Gao et al., 2018, Grover and Leskovec, 2016, Perozzi et al., 2014, Zhang, et al., 2020, Zuo, et al., 2018), we employ the node embeddings learned by TigeCMN to address three representative applications: node classification, recommendation and visualization. Through our extensive experiments, we aim to answer the following research questions:
- RQ1:
How does TigeCMN perform compared with state-of-the-art graph embedding and recommendation methods?
- RQ2:
Is the memory-augmented
Conclusions
In this paper, we propose a novel framework named TigeCMN, which utilizes memory networks to express, store and manipulate node embeddings explicitly via the writing and reading operations. Through modeling node’s temporal interactions, our model has the advantage of capturing its dynamic patterns by tracking the information changes with a little higher space consumption. The generated deep matrix representation can not only represent the complicated relations between heterogeneous nodes but
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
We would like to thank the anonymous reviewers for their useful comments that help improve the quality of this work. We would also like to express our gratitude to the handling editors for their valuable comments and suggestions. This work is supported by the National Natural Science Foundation of China (Grant No. 61972349), the National Key Research and Development Program, China (Grant No. 2018YFB1403202, 2018YFC2002603, 2019YFF0302601) and the Alibaba-Zhejiang University Joint Institute of
References (77)
- et al.
Link prediction in complex networks: A survey
Physica A: Statistical Mechanics and Its Applications
(2011) - Baldi, P. (2012). Autoencoders, unsupervised learning, and deep architectures. In Proceedings of ICML workshop on...
- et al.
Agglomerative clustering of a search engine query log
- et al.
Laplacian eigenmaps for dimensionality reduction and data representation
Neural Computation
(2003) - et al.
Node classification in social networks
- et al.
Grarep: Learning graph representations with global structural information
- Cao, S., Lu, W., & Xu, Q. (2016). Deep neural networks for learning graph representations. In Proceedings of the...
- et al.
Heterogeneous network embedding via deep architectures
- et al.
Sequential recommendation with user memory networks
- et al.
Metapath2vec: Scalable representation learning for heterogeneous networks
Collaborative memory network for recommendation systems
BiNE: Bipartite network embedding
DynGEM: Deep embedding method for dynamic graphs
Neural turing machines
Hybrid computing using a neural network with dynamic external memory
Nature
Learning to transduce with unbounded memory
Node2vec: Scalable feature learning for networks
A survey of accuracy evaluation metrics of recommendation tasks
Journal of Machine Learning Research
Inductive representation learning on large graphs
Session-based recommendations with recurrent neural networks
Long short-term memory
Neural Computation
Node2bits: Compact time-and attribute-aware node representations for user stitching
Self-attentive sequential recommendation
Adam: A method for stochastic optimization
Semi-supervised classification with graph convolutional networks
Predicting dynamic embedding trajectory in temporal interaction networks
Attributed network embedding for learning in a dynamic environment
MV-GCN: Multi-view graph convolutional networks for link prediction
IEEE Access
Hierarchical bipartite graph neural networks: Towards large-scale E-commerce applications
Deep learning for community detection: Progress, challenges and opportunities
Visualizing data using t-SNE
Journal of Machine Learning Research
Cited by (6)
ConTIG: Continuous representation learning on temporal interaction graphs
2024, Neural NetworksForeSeer: Product Aspect Forecasting Using Temporal Graph Embedding
2023, International Conference on Information and Knowledge Management, ProceedingsKG2Lib: knowledge-graph-based convolutional network for third-party library recommendation
2023, Journal of SupercomputingSurvey of Graph Neural Network in Recommendation System
2022, Journal of Frontiers of Computer Science and TechnologyTemporal Knowledge Graph Entity Alignment via Representation Learning
2022, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)