Continuous-time dynamic graph learning based on spatio-temporal random walks

Abstract

The application of dynamic graph representation learning in the processing of dynamic systems, such as social networks and transportation networks, has increased in recent times due to its ability to efficiently integrate topological and temporal information into a compact representation. Continuous-time dynamic graphs (CTDGs) have received considerable attention due to their capacity to retain precise temporal information. Existing methods based on random walk techniques often use time-biased sampling to extract dynamic graph patterns, neglecting the topological properties of the graph. Additionally, previous anonymous walks do not share node identifiers, failing to fully leverage the correlations between network patterns, which play a crucial role in predicting future interactions. Consequently, this study focuses on methods related to CTDGs. This paper presents a novel continuous-time dynamic graph learning method based on spatio-temporal random walks, which makes three main contributions: (i) By considering temporal constraints and topological structures, our method extracts diverse expressive patterns from CTDGs; (ii) It introduces the hitting counts of the nodes at a certain position as the node’s relative identity. This approach fully leverages the correlation of network patterns, ensuring that the pattern structure remains consistent even after removing node identities; (iii) An attention mechanism is employed to aggregate walk encodings, allowing the importance of different walks to be distinguished. This facilitates a more precise delineation of the relationships and structural attributes between nodes, thereby enhancing the precision and expressive power of node representations. The proposed method demonstrates superior performance compared to the average strongest baseline, achieving gains of 2.72% and 2.46% in all transductive and inductive link prediction tasks, respectively. Additionally, it attains up to an 8.7% improvement on specific datasets. Furthermore, it exhibits the second best overall performance in dynamic node classification tasks.

Appendices

A Notations

See Table 7

Table 7 Summary of important notations

B Theoretical analysis of the combined sampling probabilities

1.1 B.1 Theoretical background

In dynamic graph learning, both temporal and spatial information are crucial for understanding the relationships between nodes. Temporal random walks emphasize the relevance of time adjacency, while spatial random walks focus on the connectivity between nodes. To leverage both types of information, it is essential to consider their combined effects.

Temporal sampling probability $P_t (a)$: The temporal sampling probability reflects the influence of nodes around a specific time t. The formula is given by:

$$\begin{aligned} P_t(a) = \frac{\exp(\alpha (t_a - t))}{\sum \limits _{a' \in \mathcal {G}_{u,t}} \exp(\alpha (t_{a'} - t))} \end{aligned}$$

(18)

The closer the timestamp $t_a$ of node a is to the current time t, the higher its probability. This design aims to prioritize nodes that are temporally adjacent, thereby capturing temporal dynamics effectively.

Spatial sampling probability $P_s (a)$: The spatial sampling probability considers the connectivity of nodes. The formula is:

$$\begin{aligned} P_s(a) = \frac{\exp(-\beta / d_a)}{\sum \limits _{a' \in \mathcal {G}_{u,t}} \exp(-\beta / d_{a'})} \end{aligned}$$

(19)

The higher the degree $d_a$ of node a, the greater its probability. Nodes with higher degrees are often in more critical positions in the network, so they should be given higher priority during sampling.

1.2 B.2 Theoretical analysis: necessity of averaging

In many practical scenarios, temporal and spatial information are interconnected. In dynamic networks, the connectivity of a node (spatial) may influence its interaction time (temporal) and vice versa. Thus, assuming equal contributions from $P_t (a)$ and $P_s (a)$ can effectively reflect the complex relationships present in real-world situations.

In certain situations, nodes that are close in time may not be connected spatially, and vice versa. Therefore, solely considering one factor might result in the loss of crucial information. By averaging, we can balance the effects of both, enhancing the model’s adaptability and generalization capability.

1.3 B.3 Mathematical derivation

To rigorously illustrate this theory, we can approach it from a probability theory perspective.

Assuming $P_t (a)$ and $P_s (a)$ are valid probability distributions, we have:

$$\begin{aligned} & \sum \limits _{a' \in \mathcal {G}_{u,t}} P_t (a) = 1 \end{aligned}$$

(20)

$$\begin{aligned} & \sum \limits _{a' \in \mathcal {G}_{u,t}} P_s (a) = 1 \end{aligned}$$

(21)

Defining $P_{\text{combined}}$ as:

$$\begin{aligned} P_{\text{combined}} = \frac{P_t (a) + P_s (a)}{2} \end{aligned}$$

(22)

We then have:

$$\begin{aligned} \sum \limits _{a' \in \mathcal {G}_{u,t}} P_{\text{combined}} = \frac{1+1}{2} = 1 \end{aligned}$$

(23)

This indicates that $P_{\text{combined}}$ is also a valid probability distribution.

Combining the temporal and spatial information through averaging $P_t (a)$ and $P_s (a)$ is both reasonable and necessary. This approach not only considers the unique contributions of both factors but also enhances the model’s performance in dynamic graph learning. In summary, taking into account the influences of time and space under different conditions allows us to more effectively uncover the diversity and complexity within dynamic systems.

C Time complexity analysis

In Algorithm 1, the time complexity of the outer loop is O(l), the middle loop is O(C), and the inner loop, which traverses neighbors, has a complexity of O(d), where d is the maximum degree of the nodes. Therefore, the overall time complexity is O(lCd).

D Experimental setting

1.1 D.1 Dataset source

Most of the used original dynamic graph datasets come from Origin Datasets, which can be downloaded here. For convenience, you can also directly download the processed data package from processed_data. Here are the UNTrade contains the food and agriculture trade between 181 nations for more than 30 years. The weight of each link indicates the total sum of normalized agriculture import or export values between two particular countries of these datasets:

• Wikipedia is a bipartite interaction graph that contains the edits on Wikipedia pages over a month. Nodes represent users and pages, and links denote the editing behaviors with timestamps. Each link is associated with a 172-dimensional Linguistic Inquiry and Word Count (LIWC) feature. This dataset additionally contains dynamic labels that indicate whether users are temporarily banned from editing.

• Reddit is bipartite and records the posts of users under subreddits during one month. Users and subreddits are nodes, and links are the timestamped posting requests. Each link has a 172-dimensional LIWC feature. This dataset also includes dynamic labels representing whether users are banned from posting.

• Enron records the email communications between employees of the ENRON energy corporation over three years.

• UCI is an online communication network, where nodes are university students and links are messages posted by students.

• Flights is a dynamic flight network that displays the development of air traffic during the COVID-19 pandemic. Airports are represented by nodes and the tracked flights are denoted as links. Each link is associated with a weight, indicating the number of flights between two airports in a day.

• MOOC is a bipartite interaction network of online sources, where nodes are students and course content units (e.g., videos and problem sets). Each link denotes a student’s access behavior to a specific content unit and is assigned with a 4-dimensional feature.

• LastFM is bipartite and consists of the information about which songs were listened to by which users over one month. Users and songs are nodes, and links denote the listening behaviors of users.

1.2 D.2 Baselines

• CTDNE [22] extends the static network embedding to dynamic graphs, where temporal random walks have been proposed with the skip-gram model to learn node representations.

• DyRep [18] introduces a recurrent architecture to update node states during each interaction. It also includes a temporal attention aggregation module to consider the structural information evolving over time in dynamic graphs.

• JODIE [20] uses two coupled recurrent neural networks to update the states of users and items. It introduces a projection operation to learn the future representation trajectories of each user/item.

• TGAT [13] computes node representations by aggregating features from each node’s temporal-topological neighbors through a self-attention mechanism. It also features a time encoding function to capture temporal patterns.

• TGN [14] maintains an evolving memory for each node, updating it when nodes are observed in interactions. This is achieved through message functions, a message aggregator, and a memory updater. An embedding module generates the temporal representation of nodes.

• CAWN [15] extracts multiple causal anonymous walks for each node, exploring the causal relationships in the network dynamics and generating relative node identities. It then encodes each walk using a recurrent neural network and aggregates these walks to obtain the final node representation.

• EdgeBank [37] is a purely memory-based method for transductive dynamic link prediction, with no trainable parameters. It stores observed interactions in memory cells and updates the memory through various strategies.

• GraphMixer [17] integrates a fixed time encoding function into an MLP-Mixer-based link encoder to learn temporal link relationships.

• NeurTWs [36] learns temporal node embeddings by combining contrastive learning and random walks with neighbor graphs. The focus is on optimizing node representations by contrasting positive and negative samples.

1.3 D.3 Implementation setails

Our code is available at STAW, where we provide detailed instructions for dataset preparation and model training. The searched ranges of hyperparameters and the related methods are shown in Table 8.

Table 8 Searched ranges of hyperparameters and the related methods

E Time encoding

In our model, time is modeled using a series of cosine functions with different frequencies. We do not directly apply the traditional Fourier transform; instead, we encode the timestamps through a linear transformation to indirectly capture the frequency features in the time series. This process can be viewed as a “Fourier-like”encoding of the time series, with the goal of modeling the periodic characteristics of time using sine and cosine basis functions at different frequencies.

Specifically, $\Delta t$ is $t' - t$, the trainable parameter matrix $\omega$ represents different frequency scales. Each frequency scale corresponds to a specific time period, and during the forward pass, the timestamps are transformed linearly and then passed through the cosine function (cos) to generate the corresponding encoding. This process simulates the decomposition of the time series similar to a Fourier transform, but here we directly generate the frequency features using pre-defined frequency scales (initialized via $1 / 10^{\frac{9k}{time\_dim}}$ $for \ k=0,1,2,...,time\_dim-1$ ).

The main motivation for using Fourier transforms or similar approaches is to capture the periodic characteristics of the time series, especially when the data contains multiple frequency components. This method allows the model to extract and represent periodic patterns at different time scales, providing better generalization when dealing with long-term and complex sequential data.