Detformer: Detect the Reliable Attention Index for Ultra-long Time Series Forecasting

Abstract

Long sequence time-series forecasting is a challenging task that involves all aspects of production and life. This requires establishing a model to efficiently predict the future by using temporal dependencies from the past. Although Transformer-based solutions deliver state-of-the-art forecasting performance, there are still two issues when focusing on the ultra-long time series: First, existing solutions take heuristic approaches for black-box sampling to reduce the quadratic time complexity of canonical self-attention that leads to numerical instability and loss of accuracy. Furthermore, attention-based models cannot be applied directly due to the lack of temporal modelling capability. To tackle these issues, we propose a stable and accurate model, named Detformer, which can achieve $\mathcal{O}(L\cdot logL)$ time complexity. Specially, we design a dual-feedback sparse attention mechanism to eliminate the poor numerical stability in heuristic sparse attention, and then propose a temporal dependency extraction mechanism that enables Detformer to carry out temporal modelling from the perspective of the attention index. We further propose a noise-eliminating algorithm that identifies reliable attention to improve the temporal modelling. Extensive experiments on four benchmark datasets demonstrate the effectiveness of our Detformer model and the efficiency of our dual-feedback attention mechanism.