Skip to main content

Time-Series Forecasting Through Contrastive Learning with a Two-Dimensional Self-attention Mechanism

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14448))

Included in the following conference series:

  • 1239 Accesses

Abstract

Contrastive learning methods have impressive capabilities in time-series representation; however, challenges in capturing contextual consistency and extracting features that meet the requirements of representation learning remain. To address these problems, this study proposed a time-series prediction contrastive learning model based on a two-dimensional self-attention mechanism. The main innovations of this model were as follows: First, long short-term memory (LSTM) adaptive pruning was used to form two subsequences with overlapping parts to provide robust context representation for each timestamp. Second, the model extracted sequence data features in both global and local dimensions. In the channel dimension, the model encoded sequence data using a combination of a self-attention mechanism and dilated convolution to extract key features for capturing long-term trends and periodic changes in data. In the spatial dimension, the model adopted a sliding-window self-attention mechanism to encode sequence data, thereby improving its perceptual ability for local features. Finally, the model introduced a self-correlation attention mechanism that converted the similarity calculation from the real domain to the frequency domain through a Fourier transform, better capturing the periodicity and trends in the data. The experimental results showed that the proposed model outperformed existing models in multiple time-series prediction tasks, demonstrating its effectiveness and feasibility in time-series prediction tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014.

  2. 2.

    https://github.com/liyaguang/ETDataset.

References

  1. Albawi, S., Mohammed, T.A., Al-Zawi, S.: Understanding of a convolutional neural network. In: International Conference on Engineering and Technology (ICET), pp. 1–6. IEEE (2017)

    Google Scholar 

  2. Angryk, R.A., et al.: Multivariate time series dataset for space weather data analytics. Sci. Data 7(1), 227 (2020)

    Article  Google Scholar 

  3. Beck, N., Katz, J.N.: Modeling dynamics in time-series-cross-section political economy data. Ann. Rev. Polit. Sci. 14, 331–352 (2011)

    Article  Google Scholar 

  4. Börjesson, L., Singull, M.: Forecasting financial time series through causal and dilated convolutional neural networks. Entropy 22(10), 1094 (2020)

    Article  MathSciNet  Google Scholar 

  5. Bose, A.J., Ling, H., Cao, Y.: Adversarial contrastive estimation. In: International Conference on Machine Learning (2018)

    Google Scholar 

  6. Bromley, J., Guyon, I., LeCun, Y., Säckinger, E., Shah, R.: Signature verification using a “siamese” time delay neural network. Adv. Neural Inf. Process. Syst. 6 (1993)

    Google Scholar 

  7. Chen, H., Wu, C., Du, B., Zhang, L.: Deep siamese multi-scale convolutional network for change detection in multi-temporal VHR images. In: 2019 10th International Workshop on the Analysis of Multitemporal Remote Sensing Images (MultiTemp), pp. 1–4. IEEE (2019)

    Google Scholar 

  8. Chen, M., Peng, H., Fu, J., Ling, H.: Autoformer: searching transformers for visual recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12270–12280 (2021)

    Google Scholar 

  9. Franceschi, J.Y., Dieuleveut, A., Jaggi, M.: Unsupervised scalable representation learning for multivariate time series. Adv. Neural Inf. Process. Syst. 32, 1–11 (2019)

    Google Scholar 

  10. Gasparin, A., Lukovic, S., Alippi, C.: Deep learning for time series forecasting: the electric load case. CAAI Trans. Intell. Technol. 7(1), 1–25 (2022)

    Article  Google Scholar 

  11. Graves, A., Graves, A.: Supervised Sequence Labelling. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-24797-2_2

    Book  MATH  Google Scholar 

  12. Kipf, T., Van der Pol, E., Welling, M.: Contrastive learning of structured world models. In: International Conference on Learning Representations (2019)

    Google Scholar 

  13. Li, S., et al.: Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Adv. Neural Inf. Process. Syst. 32, 1–11 (2019)

    Google Scholar 

  14. Liu, X., Guo, J., Wang, H., Zhang, F.: Prediction of stock market index based on ISSA-BP neural network. Expert Syst. Appl. 204, 117604 (2022)

    Article  Google Scholar 

  15. Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)

    Google Scholar 

  16. Ma, C., Wen, J., Bengio, Y.: Universal successor representations for transfer reinforcement learning. In: International Conference on Learning Representations (2018)

    Google Scholar 

  17. Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. In: Conference on Neural Information Processing Systems (2018)

    Google Scholar 

  18. Peng, Z., et al.: Conformer: local features coupling global representations for visual recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 367–376 (2021)

    Google Scholar 

  19. Sezer, O.B., Gudelek, M.U., Ozbayoglu, A.M.: Financial time series forecasting with deep learning: a systematic literature review: 2005–2019. Appl. Soft Comput. 90, 106181 (2020)

    Article  Google Scholar 

  20. Tonekaboni, S., Eytan, D., Goldenberg, A.: Unsupervised representation learning for time series with temporal neighborhood coding. In: International Conference on Learning Representations (2021)

    Google Scholar 

  21. Vaswani, A., et al.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30, 1–11 (2017)

    Google Scholar 

  22. Vlahogianni, E.I., Karlaftis, M.G.: Testing and comparing neural network and statistical approaches for predicting transportation time series. Transp. Res. Rec. 2399(1), 9–22 (2013)

    Article  Google Scholar 

  23. Wang, J., Zhou, F., Wen, S., Liu, X., Lin, Y.: Deep metric learning with angular loss. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2593–2601 (2017)

    Google Scholar 

  24. Wang, J., et al.: Learning fine-grained image similarity with deep ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1386–1393 (2014)

    Google Scholar 

  25. Woo, G., Liu, C., Sahoo, D., Kumar, A., Hoi, S.: Cost: contrastive learning of disentangled seasonal-trend representations for time series forecasting. In: International Conference on Learning Representations (2022)

    Google Scholar 

  26. Wu, Z., Xiong, Y., Yu, S.X., Lin, D.: Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733–3742 (2018)

    Google Scholar 

  27. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: International Conference on Learning Representations (2016)

    Google Scholar 

  28. Yu, Y., Si, X., Hu, C., Zhang, J.: A review of recurrent neural networks: LSTM cells and network architectures. Neural Comput. 31(7), 1235–1270 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  29. Yue, Z., et al.: Ts2vec: towards universal representation of time series. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 8980–8987 (2022)

    Google Scholar 

  30. Zaremba, W., Sutskever, I., Vinyals, O.: Recurrent neural network regularization. In: International Conference on Learning Representations (2014)

    Google Scholar 

  31. Zhang, D., Han, J., Zhang, Y.: Supervision by fusion: towards unsupervised learning of deep salient object detector. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4048–4056 (2017)

    Google Scholar 

  32. Zhang, F., Chen, G., Wang, H., Li, J., Zhang, C.: Multi-scale video super-resolution transformer with polynomial approximation. IEEE Trans. Circ. Syst. Video Technol. 33, 4496–4506 (2023). https://doi.org/10.1109/TCSVT.2023.3278131

    Article  Google Scholar 

  33. Zhang, F., Guo, T., Wang, H.: DFNET: decomposition fusion model for long sequence time-series forecasting. Knowl.-Based Syst. 277, 110794 (2023)

    Article  Google Scholar 

  34. Zhang, J., Nawata, K.: Multi-step prediction for influenza outbreak by an adjusted long short-term memory. Epidemiol. Infect. 146(7), 809–816 (2018)

    Article  Google Scholar 

  35. Zheng, X., Chen, X., Schürch, M., Mollaysa, A., Allam, A., Krauthammer, M.: SimTS: rethinking contrastive representation learning for time series forecasting. In: International Joint Conference on Artificial Intelligence (2023)

    Google Scholar 

  36. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)

    Google Scholar 

  37. Zhou, H., et al.: Informer: beyond efficient transformer for long sequence time-series forecasting. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 11106–11115 (2021)

    Google Scholar 

  38. Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., Jin, R.: Fedformer: frequency enhanced decomposed transformer for long-term series forecasting. In: International Conference on Machine Learning, pp. 27268–27286 (2022)

    Google Scholar 

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (62272281), the Special Funds for Taishan Scholars Project(tsqn202306274), and the Youth Innovation Technology Project of Higher School in Shandong Province (2019KJN042).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Fan Zhang .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jiang, L., Zhang, F., Zhang, M., Zhang, C. (2024). Time-Series Forecasting Through Contrastive Learning with a Two-Dimensional Self-attention Mechanism. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Lecture Notes in Computer Science, vol 14448. Springer, Singapore. https://doi.org/10.1007/978-981-99-8082-6_12

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8082-6_12

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8081-9

  • Online ISBN: 978-981-99-8082-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics