Skip to main content
SpringerLink
Log in

A lightweight model using frequency, trend and temporal attention for long sequence time-series prediction

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Although deep learning makes great success in increasing the accuracy for long sequence time-series forecasting, its complex neural network structure, which comprises many different types of layers and each layer containing hundreds and thousands of neurons, challenges the computing and memory capability of embedded platforms. This paper proposes a lightweight and efficient neural network called TTFNet, which forecasts long time series using three types of features (i.e., the Trend, Temporal attention, and Frequency attention) extracted from raw time series. In TTFNet, we perform a pooling operation on the historical data in a recent time window to extract a general trend, use a multi-layer perceptron to discover the temporal correlation between data as temporal attention, and apply the fast Fourier transforms on data to obtain frequency information as frequency attention. Each feature is separately extracted from its neural network branch with an output result, and we weight the three results to generate the final prediction while optimal weights are learnt. Also, the three prediction results can run in parallel since they are independent from one another. The experimental results show that the proposed method reduces the memory overhead and runtime by 62% and 81% of the five counterpart methods on average while achieving comparable performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availibility

The datasets analyzed during the current study are available in https://github.com/zhouhaoyi/ETDataset, https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014 and https://www.ncei.noaa.gov/data/local-climatological-data/.

Notes

  1. https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014.

  2. https://github.com/zhouhaoyi/ETDataset.

  3. https://www.ncei.noaa.gov/data/local-climatological-data/archive/.

References

  1. Kart U, Lukežič A, Kristan M et al (2019) Object tracking by reconstruction with view-specific discriminative correlation filters. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 1339–1348

  2. Zhang P, Liu W, Wang D et al (2020) Non-rigid object tracking via deep multi-scale spatial-temporal discriminative saliency maps. Pattern Recognit 100:107130

    Article  Google Scholar 

  3. Kalajdjieski J, Korunoski M, Stojkoska BR, et al (2020) Smart city air pollution monitoring and prediction: a case study of skopje. In: ICT Innovations 2020. Machine Learning and Applications, pp 15–27

  4. Xie P, Li T, Liu J et al (2020) Urban flow prediction from spatiotemporal data using machine learning: a survey. Inf Fus 59:1–12

    Article  Google Scholar 

  5. Yang A-M, Han Y, Liu C-S et al (2021) D-tsvr recurrence prediction driven by medical big data in cancer. IEEE Trans Ind Inf 17(5):3508–3517

    Article  Google Scholar 

  6. Ren L, Liu Y, Huang D, Huang K, Yang C (2022) Mctan: a novel multichannel temporal attention-based network for industrial health indicator prediction. IEEE Trans Neural Netw Learn Syst 1–12

  7. Liu F, Xue S, Wu J, et al (2020) Deep learning for community detection: progress, challenges and opportunities. In: Proceedings of the twenty-ninth international joint conference on artificial intelligence, IJCAI-20, pp 4981–4987

  8. Bui T-C, Kim J, Kang T, et al (2021) Star: spatio-temporal prediction of air quality using a multimodal approach. In: Intelligent systems and applications, pp 389–406

  9. Bentsen LØ, Warakagoda ND, Stenbro R et al (2023) Spatio-temporal wind speed forecasting using graph networks and novel transformer architectures. Appl Energy 333:120565

    Article  Google Scholar 

  10. Wu M, Zhu C, Chen L (2020) Multi-task spatial-temporal graph attention network for taxi demand prediction. In: Proceedings of the 2020 5th international conference on mathematics and artificial intelligence. ICMAI 2020, pp 224–228

  11. An J, Guo L, Liu W et al (2021) Igagcn: information geometry and attention-based spatiotemporal graph convolutional networks for traffic flow prediction. Neural Netw 143:355–367

    Article  Google Scholar 

  12. Chaovalit P, Gangopadhyay A, Karabatis G, et al (2011) Discrete wavelet transform-based time series analysis and mining. ACM Comput Surv 43(2)

  13. Mohammadi HA, Ghofrani S, Nikseresht A (2023) Using empirical wavelet transform and high-order fuzzy cognitive maps for time series forecasting. Appl Soft Comput 109990

  14. Ong P, Zainuddin Z (2019) Optimizing wavelet neural networks using modified cuckoo search for multi-step ahead chaotic time series prediction. Appl Soft Comput 80:374–386

    Article  Google Scholar 

  15. Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673–2681

    Article  Google Scholar 

  16. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  17. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Proceedings of the 27th international conference on neural information processing systems- Volume 2. NIPS’14, pp 3104–3112

  18. Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. In: Proceedings of the 31st international conference on neural information processing systems NIPS’17, pp 6000–6010

  19. Zhou H, Zhang S, Peng J, et al (2021) Informer: beyond efficient transformer for long sequence time-series forecasting. In: The thirty-fifth AAAI conference on artificial intelligence, AAAI 2021, p

  20. Li S, Jin X, Xuan Y, et al (2019) Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In: Advances in neural information processing systems, vol. 32

  21. Lee-Thorp J, Ainslie J, Eckstein I, et al (2021) FNet: mixing tokens with fourier transforms, 2105–03824. arXiv:2105.03824 [cs.CL]

  22. Williams BM, Durvasula PK, Brown DE (1998) Urban freeway traffic flow prediction: application of seasonal autoregressive integrated moving average and exponential smoothing models. Transp Res Rec 1644(1):132–141

    Article  Google Scholar 

  23. Kumar SV, Vanajakshi L (2015) Short-term traffic flow prediction using seasonal arima model with limited input data. Eur Transp Res Rev 7(3):1–9

    Article  Google Scholar 

  24. Contreras-Reyes JE (2022) Rényi entropy and divergence for varfima processes based on characteristic and impulse response functions. Chaos Solitons Fractals 160:112268

    Article  MathSciNet  MATH  Google Scholar 

  25. Karevan Z, Suykens JAK (2020) Transductive LSTM for time-series prediction: an application to weather forecasting. Neural Netw 125:1–9

    Article  Google Scholar 

  26. Yang Y, Fan C, Xiong H (2022) A novel general-purpose hybrid model for time series forecasting. Appl Intell 52(2):2212–2223

    Article  Google Scholar 

  27. Khedhiri S (2022) Comparison of SARFIMA and LSTM methods to model and to forecast Canadian temperature. Region Stat 12(02):177–194

    Article  Google Scholar 

  28. Cho K, van Merriënboer B, Gulcehre C, et al (2014) Learning phrase representations using RNN encoder–decoder for statistical machine translation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1724–1734

  29. Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, conference track proceedings

  30. Li D, Han M, Wang J (2012) Chaotic time series prediction based on a novel robust echo state network. IEEE Trans Neural Netw Learn Syst 23(5):787–799

    Article  Google Scholar 

  31. González-Zapata AM, Tlelo-Cuautle E, Ovilla-Martinez B et al (2022) Optimizing echo state networks for enhancing large prediction horizons of chaotic time series. Mathematics 10(20):3886

    Article  Google Scholar 

  32. Zhang Z, Wang Y, Wang K (2013) Fault diagnosis and prognosis using wavelet packet decomposition, Fourier transform and artificial neural network. J Intell Manuf 24(6):1213–1227

    Article  Google Scholar 

  33. Mironovova M, Bíla J (2015) Fast fourier transform for feature extraction and neural network for classification of electrocardiogram signals. In: 2015 fourth international conference on future generation communication technology (FGCT), pp 1–6

  34. Lin S, Liu N, Nazemi M, et al. (2018) FFT-based deep learning deployment in embedded systems. In: 2018 design, automation test in Europe conference exhibition (DATE), pp 1045–1050

  35. Abtahi T, Shea C, Kulkarni A et al (2018) Accelerating convolutional neural network with FFT on embedded hardware. IEEE Trans.VLSI Syst 26(9):1737–1749

    Article  Google Scholar 

  36. Choromanski K, Likhosherstov V, Dohan D, et al (2020) Masked language modeling for proteins via linearly scalable long-context transformers, 2006–03555. arXiv:2006.03555 [cs.LG]

  37. Sak H, Senior A, Beaufays F (2014) Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition. Comput Sci 338–342

  38. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Proceedings of the 27th international conference on neural information processing systems - Volume 2. NIPS’14, pp 3104–3112

Download references

Acknowledgements

This work is funded in part by the National Natural Science Foundation of China (File No. 62072216) and the Science and Technology Development Fund, Macau SAR (File No. 0076/2022/A2 and 0008/2022/AGJ).

Author information

Authors and Affiliations

  1. School of Information and Electrical Engineering, Hebei University of Engineering, Handan, China

    Lingqiang Chen

  2. School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi, China

    Guanghui Li

  3. School of Information Technology, Deakin University, Melbourne, Australia

    Guangyan Huang

  4. School of Computer Science and Engineering, Macau University of Science and Technology, Macau, China

    Qinglin Zhao

  5. Hebei Key Laboratory of Security & Protection Information Sensing and Processing, Handan, China

    Lingqiang Chen

Authors
  1. Lingqiang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guanghui Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guangyan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qinglin Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guanghui Li.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, L., Li, G., Huang, G. et al. A lightweight model using frequency, trend and temporal attention for long sequence time-series prediction. Neural Comput & Applic 35, 21291–21307 (2023). https://doi.org/10.1007/s00521-023-08871-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08871-9

Keywords

Search

Navigation