InTrans: Fast Incremental Transformer for Time Series Data Prediction

Abstract

Predicting time-series data is useful in many applications, such as natural disaster prevention system, weather forecast, traffic control system, etc. Time-series forecasting has been extensively studied. Many existing forecasting models tend to perform well when predicting short sequence time-series. However, their performances greatly degrade when dealing with the long one. Recently, more dedicated research has been done for this direction, and Informer is currently the most efficient predicting model. The main drawback of Informer is the inability to incrementally learn. This paper proposes an incremental Transformer, called InTrans, to address the above bottleneck by reducing the training/predicting time of Informer. The time complexities of InTrans comparing to the Informer are: (1) O(S) vs O(L) for positional and temporal embedding, (2) $O((S+k-1)*k)$ vs $O(L*k)$ for value embedding, and (3) $O((S+k-1)*d_{dim})$ vs $O(L*d_{dim})$ for the computation of Query/Key/Value, where L is the length of the input; k is the kernel size; $d_{dim}$ is the number of dimensions; and S is the length of the non-overlapping part of the input that is usually significantly smaller than L. Therefore, InTrans could greatly improve both training and predicting speed over the state-of-the-art model, Informer. Extensive experiments have shown that InTrans is about 26% faster than Informer for both short sequence and long sequence time-series prediction.

S. Bou—Due to name change, Savong Bou is now known as Takehiko Hashimoto.

Appendices

Appendix

A Proof of Theorem 1

The inputs to the Eq. 2 are positional and dimensional information. The dimensional information is the same for all records in the training data. Assuming that we have two consecutive input i and $i+1$ of L records. The gap between the beginning of input i and $i+1$ is S records. The positional information of input i is $POS_i=[pos_{(i-1)S+1}, pos_{(i-1)S+2},..., pos_{(i-1)S+S}, pos_{(i-1)S+S+1}, ..., pos_{(i-1)S+L}]$, and the positional information of input $i+1$ is $POS_{i+1}=[pos_{(i-1)S+S+1}, pos_{(i-1)S+S+2}, ..., pos_{(i-1)S+L}, pos_{(i-1)S+L+1}, ..., pos_{(i-1)S+L+S}]$. We have

• $POS_i=POSn_i \oplus POSo_i$, and

• $POS_{i+1}=POSo_{i+1} \oplus POSn_{i+1}$, where

  • $POSn_i=[pos_{(i-1)S+1}, pos_{(i-1)S+2},..., pos_{(i-1)S+S}]$, and

  • $POSo_i=POSo_{i+1}=[pos_{(i-1)S+S+1}, ..., pos_{(i-1)S+L}]$

  • $POSn_{i+1}=[pos_{(i-1)S+L+1}, ..., pos_{(i-1)S+L+S}]$

PosEncoding(POS) represents encoding the POS by Eq. 2. We have:

• $Pn_{i}=PosEncoding(POSn_i)$, and

• $Po_{i}=PosEncoding(POSo_i)$, so

• $P_{i}=Pn_{i} \oplus Po_{i}$.

Because $POSo_i=POSo_{i+1}$, then we have:

• $Pn_{i+1}=PosEncoding(POSn_{i+1})$, therefore

• $P_{i+1}= Po_{i+1} \oplus Pn_{i+1}= Po_{i} \oplus Pn_{i+1}$.

Theorem 1 is proven.

B Proof of Theorem 2

The notations in Appendix A are also used in this Section. For relative position, the positional information of all input/output is the same, so $POS_1= POS_2=...= POS_i=POS_{i+1}=[pos_{1}, pos_{2},..., pos_{L}]$. Therefore, we have $P_1= P_2=...=P_i=P_{i+1}=PosEncoding(POS_1)$. Theorem 2 is proven.

C Proof of Theorem 3

The temporal embedding takes temporal information, such as weak, month, and holiday, of the records as a basis for embedding. The temporal information of the same record does not change wrt different training samples. Therefore, the temporal embedding of all records belongs to the overlapping part between input i and $i+1$ is the same. The proof is similar to that of absolute positional embedding in Appendix A. Thus, $To_{i+1}=To_{i}$, so Theorem 3 is proved.

D Proof of Theorem 4

The notations in Appendix A are also used in this Section. We need to prove that $Vr_{i} = Vr_{i+1}$. When computing the embedding value of input i, the input i is convolved by a kernel (e.g., width k). Since the default stride is set to one, the resulting embedding values of records $x_{(i-1)S+1}$ to $x_{(i-1)S+L-(k-1)}$ are fully convolved without including the padding values. Similarly, the embedding values of records $x_{(i-1)S+S+1}$ to $x_{(i-1)S+L-(k-1)}$ are fully convolved. Therefore, the embedding values of records $x_{(i-1)S+S+1}$ to $x_{(i-1)S+L-(k-1)}$ are the same for both input i and $i+1$. Therefore, $Vr_{i} = Vr_{i+1}$, which proves Theorem 4.

E Proof of Theorem 5

Query, Key, and Value are computed as in Eq. 7. Such multiplication does not change the value distribution from the original input embedding. Therefore, Query, Key, and Value can be incrementally computed in similar manner to that of input embedding, which is proved in Theorems 3, 1, and 4. Therefore, Theorem 5 is proven.

F Proof of Theorem 6

InTrans is implemented on top of Informer by adopting the incremental computation of the temporal/positional/value embeddings and Query/Key/Value of the training sample. To prove that InTrans has the same predicting accuracy as that of Informer, we have to prove that the temporal/positional/value embeddings and Query/Key/Value incrementally computed by InTrans is the same as the temporal/positional/value embeddings and Query/Key/Value computed by Informer. Theorems 1, 2, 3, 4, 5 have proved that the embedding of the input and the Query/Key/Value incrementally computed by InTrans are the same as those non-incrementally computed by Informer. Theorem 3.7 is proven.

G Proof of Theorem 7

Equations 4 and 5 suggests that positional and temporal embedding can be incrementally computed, which is proved in Theorems 1 and 3. For each input $i+1$, the positional embedding ($Po_i$) and the temporal embedding ($To_i$) corresponding to the overlapping part between inputs i and $i+1$ computed when embedding the input i can be reused for input $i+1$. Therefore, only the positional embedding ($Pn_{i+1}$) and the temporal embedding ($Tn_{i+1}$) corresponding to the non-overlapping part between input i and $i+1$ need to be computed when embedding input $i+1$. The size of the non-overlapping part between input i and $i+1$ is represented by S, so the time complexity to compute the positional and temporal embedding of each input is O(S).

Similarly, value embedding is done by using Conv1d. Equation 6 and Theorem 4 suggest that the value embedding of the overlapping part, excluding the value embedding of the last $k-1$ records, between input i and $i+1$ after computing value embedding of input i can be reused for embedding the input $i+1$. Therefore, the time complexity to compute the value embedding of each input is $O((S+k-1)*k)$.

Similar to value embedding, Eq. 6 and Theorem 4 suggest that the time complexity to compute Query or Key or Value is $O((S+k-1)*d_{dim})$. Theorem 7 is proved.