DTAFORMER: Directional Time Attention Transformer For Long-Term Series Forecasting

Abstract

This paper introduces the Directional Time Attention Transformer (DTAformer) model for long-term time series forecasting, addressing the inherent limitations of traditional Transformer-based models in capturing the sequential order. By establishing a causal graph, we identify the confounding relationships, which lead to the erroneous capture of spurious sequential temporal direction information in time series models. The Directional Time Attention, a key component of the model, leverages the front-door adjustment to eliminate the confounder from the causal relationship, ensuring accurate modeling of temporal direction in time series. Additionally, we further analyze the impact of different patching methods and loss functions on prediction performance. The model’s performance is evaluated on nine benchmark datasets, with the results demonstrating its superiority over the State-of-the-Art methods.

Appendix

In this Appendix, we provide related work, causality analysis and other experimental details.

1.1 Related Work

Loss Function: MSE The Mean Squared Error (MSE) has been a cornerstone in the domain of time series prediction and is commonly used as a loss function in various forecasting models. The MSE is defined as $\text {MSE} = \frac{1}{n}\sum _{i=1}^{n}(Y_i - \hat{Y}_i)^2$, where $Y_i$ represents the true value, $\hat{Y}_i$ denotes the predicted value, and $n$ is the number of observations.

The popularity of MSE in time series analysis can be attributed to its ability to emphasize larger errors due to its quadratic nature. This characteristic makes it particularly suitable for applications where large errors are more undesirable than smaller ones. Studies such as those by Hyndman and Koehler [30] have highlighted the effectiveness of MSE in capturing the variance of forecasting errors and providing a clear measure of prediction accuracy.

However, MSE is sensitive to outliers, as noted by Chai and Draxler [29]. In datasets with significant anomalies or noise, MSE might result in biased estimations, emphasizing the need for robust preprocessing steps.

1.2 Causality Analysis

In this formula, $Y$ is no longer affected by the confounder $U$, but directly affected by $X$ and $Z$. The original text is as follows: In the causal graph, it is not only the legitimate causal pathway extending from input variable $X$ through mediator $Z$ to outcome $Y$ that warrants consideration. Concomitantly, the “backdoor" path, delineated as $X \leftarrow U \rightarrow Z \rightarrow Y$, exerts an influence on $Y$ through the cofounder $U$. This introduces a spurious correlation between between $X$ and $Y$, thus confounding the relationship. Consequently, if one is to rely solely on the correlation $P(Y \mid X)$ for model training, without addressing the confounding effects, the true causal effect from $X$ to $Y$ remains obscured, irrespective of the quantity and quality of training data [17, 18]. To disentangle the confounded relationship between $X$ and $Y$, it is imperative to obstruct the path $X \leftarrow U \rightarrow Y$, thereby isolating the causal effect between $X$ and $Y$. However, in the context of time series analysis, the exact nature of the cofounder remains difficult to quantify. As an alternative, the front-door adjustment is employed, which does not demand specific information regarding the cofounder. Additionally, this approach offers a more intelligible means of understanding the mediator.

Therefore, instead of employing the likelihood $P(Y \mid X)$, we utilize the causal intervention $P(Y \mid \text {do}(X))$ as proposed by Pearl for time series forecasting [8]. This approach aims to elucidate the genuine causal relationship between $X$ and $Y$. The front-door adjustment method is applied to compute $P(Y \mid \text {do} (X))$ through the front-door path $X \rightarrow M \rightarrow Y$. This path is constructed from two partially causal effects: $P(M \mid \text {do}(X))$ and $P(Y \mid \text {do}(Z))$. Therefore, it follows that:

$$\begin{aligned} P(Y \mid \text {do}(X)) &= \sum _z P(Z = z \mid \text {do}(X)) P(Y \mid \text {do}(Z = z)). \end{aligned}$$

(12)

Similarly, to determine $P(Z = z \mid \text {do}(X))$, it is necessary to obstruct the backdoor path $X \leftarrow U \rightarrow Y \leftarrow Z$ between $X$ and $Z$. Notably, this backdoor path includes a collider ($U \rightarrow Y \leftarrow Z$). According to Pearl [8], the presence of a collider within a path implies that it obstructs the association between the influencing variables. Thus, this path is inherently blocked, leading to the conclusion that:

$$\begin{aligned} P(Z = z \mid \text {do}(X)) = P(Z = z \mid X). \end{aligned}$$

(13)

For $P(Y \mid \text {do}(Z))$, it is necessary to block the backdoor path $Z \leftarrow X \leftarrow U \rightarrow Y$ between $Z$ and $Y$. Given the unknown specifics regarding the cofounder $U$, this path must be blocked by intervening on $X$. Hence,

$$\begin{aligned} P(Y \mid \text {do}(Z = z)) &= \sum _x P(Y \mid Z = z, X = x)P(X = x). \end{aligned}$$

(14)

Ultimately, this leads to the following formulation:

$$\begin{aligned} P(Y \mid \text {do}(X)) &= \sum _{z} P(Z = z \mid X) \sum _{x} P(X = x) P(Y \mid Z = z, X = x). \end{aligned}$$

(15)

1.3 Experimental Details

Data Descriptions The datasets employed in this study are summarized in Table 4, which delineate their inherent characteristics. These datasets have been widely utilized in the domain of time series analysis, providing a robust benchmark for evaluating the performance of various models.

Table 4. Statistics of popular datasets for benchmark.

The datasets employed encompass a diverse array of domains, each contributing unique characteristics and challenges to time series analysis:

• ETT (Electricity Transformer Temperature): Consisting of both hourly-level (ETTh) and quarter-hourly-level (ETTm) datasets, it captures key parameters such as oil and load features of electricity transformers from July 2016 to July 2018.

• Traffic: This dataset includes hourly road occupancy rates from sensors on San Francisco freeways, recorded from 2015 to 2016.

• Electricity: Details the hourly electricity consumption patterns of 321 clients from 2012 to 2014.

• Exchange-Rate: Offers daily exchange rates for eight countries, spanning from 1990 to 2016.

• Weather: Comprises 21 diverse indicators, including air temperature and humidity, recorded every 10 minutes in Germany during 2020.

• ILI: Reflects weekly data on the proportion of patients with flu-like symptoms, reported by the U.S. Centers for Disease Control and Prevention from 2002 to 2021.

Table 5. Univariate long-term forecasting results with DTAformer. ETT datasets are used with prediction lengths $T \in \{96, 192, 336, 720\}$. The best results are in bold and the second best are underlined.

The comprehensive nature of these datasets, covering different intervals, features, and domains, provides a rigorous testing ground for time series analysis methodologies.

Univariate Long-term Forecasting Results Table 5 shows the univariate long-term time series forecasting results. Compared with other baseline methods, our DTAformer achieves most of the best results.

Attention Score Heatmap In this study, we employed both Self-attention and Directional Time Attention within the DTAformer framework to generate corresponding Attention Score Heatmaps, as depicted in Fig. 4. Compared to the Self-attention, the Directional Time Attention significantly reduces feature noise, concurrently enhancing the model’s ability to capture pronounced sequence direction features.

Analysis reveals that owing to the inherent mechanism of Directional Time Attention, all query $Q$ elements are constrained to capture relationships only with key $K$ elements at identical or preceding positions. Consequently, the attention scores de-emphasize the influence of potential future directional events within the input sequence. This contrasts with the Self-attention, which encompasses a broader range of interfering factors. As a result, Directional Time Attention more effectively discerns the authentic sequential and temporal relationships inherent in the input sequence.

Fig. 4.

Attention Score Heatmap for Q = 20 in the First Header. Figure 4(a) shows the Self-attention Score Heatmap; Fig. 4(b) shows the Directional Time Attention Score Heatmap.