Abstract
Contrastive learning methods have impressive capabilities in time-series representation; however, challenges in capturing contextual consistency and extracting features that meet the requirements of representation learning remain. To address these problems, this study proposed a time-series prediction contrastive learning model based on a two-dimensional self-attention mechanism. The main innovations of this model were as follows: First, long short-term memory (LSTM) adaptive pruning was used to form two subsequences with overlapping parts to provide robust context representation for each timestamp. Second, the model extracted sequence data features in both global and local dimensions. In the channel dimension, the model encoded sequence data using a combination of a self-attention mechanism and dilated convolution to extract key features for capturing long-term trends and periodic changes in data. In the spatial dimension, the model adopted a sliding-window self-attention mechanism to encode sequence data, thereby improving its perceptual ability for local features. Finally, the model introduced a self-correlation attention mechanism that converted the similarity calculation from the real domain to the frequency domain through a Fourier transform, better capturing the periodicity and trends in the data. The experimental results showed that the proposed model outperformed existing models in multiple time-series prediction tasks, demonstrating its effectiveness and feasibility in time-series prediction tasks.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Albawi, S., Mohammed, T.A., Al-Zawi, S.: Understanding of a convolutional neural network. In: International Conference on Engineering and Technology (ICET), pp. 1–6. IEEE (2017)
Angryk, R.A., et al.: Multivariate time series dataset for space weather data analytics. Sci. Data 7(1), 227 (2020)
Beck, N., Katz, J.N.: Modeling dynamics in time-series-cross-section political economy data. Ann. Rev. Polit. Sci. 14, 331–352 (2011)
Börjesson, L., Singull, M.: Forecasting financial time series through causal and dilated convolutional neural networks. Entropy 22(10), 1094 (2020)
Bose, A.J., Ling, H., Cao, Y.: Adversarial contrastive estimation. In: International Conference on Machine Learning (2018)
Bromley, J., Guyon, I., LeCun, Y., Säckinger, E., Shah, R.: Signature verification using a “siamese” time delay neural network. Adv. Neural Inf. Process. Syst. 6 (1993)
Chen, H., Wu, C., Du, B., Zhang, L.: Deep siamese multi-scale convolutional network for change detection in multi-temporal VHR images. In: 2019 10th International Workshop on the Analysis of Multitemporal Remote Sensing Images (MultiTemp), pp. 1–4. IEEE (2019)
Chen, M., Peng, H., Fu, J., Ling, H.: Autoformer: searching transformers for visual recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12270–12280 (2021)
Franceschi, J.Y., Dieuleveut, A., Jaggi, M.: Unsupervised scalable representation learning for multivariate time series. Adv. Neural Inf. Process. Syst. 32, 1–11 (2019)
Gasparin, A., Lukovic, S., Alippi, C.: Deep learning for time series forecasting: the electric load case. CAAI Trans. Intell. Technol. 7(1), 1–25 (2022)
Graves, A., Graves, A.: Supervised Sequence Labelling. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-24797-2_2
Kipf, T., Van der Pol, E., Welling, M.: Contrastive learning of structured world models. In: International Conference on Learning Representations (2019)
Li, S., et al.: Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Adv. Neural Inf. Process. Syst. 32, 1–11 (2019)
Liu, X., Guo, J., Wang, H., Zhang, F.: Prediction of stock market index based on ISSA-BP neural network. Expert Syst. Appl. 204, 117604 (2022)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
Ma, C., Wen, J., Bengio, Y.: Universal successor representations for transfer reinforcement learning. In: International Conference on Learning Representations (2018)
Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. In: Conference on Neural Information Processing Systems (2018)
Peng, Z., et al.: Conformer: local features coupling global representations for visual recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 367–376 (2021)
Sezer, O.B., Gudelek, M.U., Ozbayoglu, A.M.: Financial time series forecasting with deep learning: a systematic literature review: 2005–2019. Appl. Soft Comput. 90, 106181 (2020)
Tonekaboni, S., Eytan, D., Goldenberg, A.: Unsupervised representation learning for time series with temporal neighborhood coding. In: International Conference on Learning Representations (2021)
Vaswani, A., et al.: Attention is all you need. Adv. Neural Inf. Process. Syst. 30, 1–11 (2017)
Vlahogianni, E.I., Karlaftis, M.G.: Testing and comparing neural network and statistical approaches for predicting transportation time series. Transp. Res. Rec. 2399(1), 9–22 (2013)
Wang, J., Zhou, F., Wen, S., Liu, X., Lin, Y.: Deep metric learning with angular loss. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2593–2601 (2017)
Wang, J., et al.: Learning fine-grained image similarity with deep ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1386–1393 (2014)
Woo, G., Liu, C., Sahoo, D., Kumar, A., Hoi, S.: Cost: contrastive learning of disentangled seasonal-trend representations for time series forecasting. In: International Conference on Learning Representations (2022)
Wu, Z., Xiong, Y., Yu, S.X., Lin, D.: Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733–3742 (2018)
Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: International Conference on Learning Representations (2016)
Yu, Y., Si, X., Hu, C., Zhang, J.: A review of recurrent neural networks: LSTM cells and network architectures. Neural Comput. 31(7), 1235–1270 (2019)
Yue, Z., et al.: Ts2vec: towards universal representation of time series. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 8980–8987 (2022)
Zaremba, W., Sutskever, I., Vinyals, O.: Recurrent neural network regularization. In: International Conference on Learning Representations (2014)
Zhang, D., Han, J., Zhang, Y.: Supervision by fusion: towards unsupervised learning of deep salient object detector. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4048–4056 (2017)
Zhang, F., Chen, G., Wang, H., Li, J., Zhang, C.: Multi-scale video super-resolution transformer with polynomial approximation. IEEE Trans. Circ. Syst. Video Technol. 33, 4496–4506 (2023). https://doi.org/10.1109/TCSVT.2023.3278131
Zhang, F., Guo, T., Wang, H.: DFNET: decomposition fusion model for long sequence time-series forecasting. Knowl.-Based Syst. 277, 110794 (2023)
Zhang, J., Nawata, K.: Multi-step prediction for influenza outbreak by an adjusted long short-term memory. Epidemiol. Infect. 146(7), 809–816 (2018)
Zheng, X., Chen, X., Schürch, M., Mollaysa, A., Allam, A., Krauthammer, M.: SimTS: rethinking contrastive representation learning for time series forecasting. In: International Joint Conference on Artificial Intelligence (2023)
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)
Zhou, H., et al.: Informer: beyond efficient transformer for long sequence time-series forecasting. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 11106–11115 (2021)
Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., Jin, R.: Fedformer: frequency enhanced decomposed transformer for long-term series forecasting. In: International Conference on Machine Learning, pp. 27268–27286 (2022)
Acknowledgements
This work is supported by the National Natural Science Foundation of China (62272281), the Special Funds for Taishan Scholars Project(tsqn202306274), and the Youth Innovation Technology Project of Higher School in Shandong Province (2019KJN042).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Jiang, L., Zhang, F., Zhang, M., Zhang, C. (2024). Time-Series Forecasting Through Contrastive Learning with a Two-Dimensional Self-attention Mechanism. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Lecture Notes in Computer Science, vol 14448. Springer, Singapore. https://doi.org/10.1007/978-981-99-8082-6_12
Download citation
DOI: https://doi.org/10.1007/978-981-99-8082-6_12
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8081-9
Online ISBN: 978-981-99-8082-6
eBook Packages: Computer ScienceComputer Science (R0)