Skip to main content
SpringerLink
Log in

ALAE: self-attention reconstruction network for multivariate time series anomaly identification

  • Data analytics and machine learning
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Multivariate time series from the real world has great application values, where its accurate anomaly identification has become an important research topic. Although existing methods have achieved promising performance to some extent, there still exist some limitations, such as the temporal dependency of long sequences and complex relationships among multiple features. To address these problems, a self-attention-based unsupervised multivariate reconstruction network called ALAE is proposed in this paper. ALAE first establishes a LSTM autoencoder framework to capture the temporal dependencies of long sequences. Then, a multi-head weighted self-attention mechanism is proposed to explore the complex relationships between multiple features in high-dimensional data. This attention mechanism also resolves the issue of insufficient information caused by the fixed vector between the encoder and decoder. Finally, a more robust strategy is used to calculate the anomaly score. Abundant experiments conducted on six public multivariate time series datasets, including financial risks, social governance, and NASA, show that ALAE performs better than seven baseline algorithms. Noteworthy, ablation experiments reveal the validity of integrating the autoencoder with the attention mechanism.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availibility

Enquiries about data availability should be directed to the authors.

References

  • Ahmed CM, Palleti VR, Mathur AP (2017) Wadi: a water distribution testbed for research in the design of secure cyber physical systems. In: Proceedings of the 3rd international workshop on cyber-physical systems for smart water networks, pp 25–28

  • Amarbayasgalan T, Pham VH, Theera-Umpon N, Ryu KH (2020) Unsupervised anomaly detection approach for time-series in multi-domains using deep reconstruction error. Symmetry 12(8):1251

    Article  Google Scholar 

  • Angiulli F, Pizzuti C (2002) Fast outlier detection in high dimensional spaces. In: European conference on principles of data mining and knowledge discovery, pp 15–27. Springer

  • Audibert J, Michiardi P, Guyard F, Marti S, Zuluaga MA (2020) Usad: unsupervised anomaly detection on multivariate time series. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp 3395–3404

  • Basu S, Meckesheimer M (2007) Automatic outlier detection for time series: an application to sensor data. Knowl Inf Syst 11:137–154

    Article  Google Scholar 

  • Bhojanapalli S, Yun C, Rawat AS, Reddi S, Kumar S (2020). Low-rank bottleneck in multi-head attention models. In: International conference on machine learning, pp 864–873. PMLR

  • Blazquez-Garcia A, Conde A, Mori U, Lozano JA (2021) A review on outlier/anomaly detection in time series data. ACM Comput Surv (CSUR) 54(3):1–33

    Article  Google Scholar 

  • Braei M, Wagner S (2020). Anomaly detection in univariate time-series: A survey on the state-of-the-art. arXiv preprint arXiv:2004.00433

  • Candanedo Luis M, Véronique F (2016) Accurate occupancy detection of an office room from light, temperature, humidity and co2 measurements using statistical learning models. Energy Build 112:28–39

    Article  Google Scholar 

  • Cheng D, Wang X, Zhang Y, Zhang L (2020) Graph neural network for fraud detection via spatial-temporal attention. IEEE Trans Knowl Data Eng 34(8):3800–3813

    Article  Google Scholar 

  • Cheng W, Ma T, Wang X, Wang G (2022) Anomaly detection for internet of things time series data using generative adversarial networks with attention mechanism in smart agriculture. Front Plant Sci, 13

  • Deng A, Hooi B (2021) Graph neural network-based anomaly detection in multivariate time series. Proc AAAI Conf Artif Intell 35(5):4027–4035

    Google Scholar 

  • Dou S, Yang K, Poor HV (2019) Pc 2 a: predicting collective contextual anomalies via lstm with deep generative model. IEEE Internet Things J 6(6):9645–9655

    Article  Google Scholar 

  • Guo Y, Ji T, Wang Q, Lixing Yu, Min G, Li P (2020) Unsupervised anomaly detection in iot systems for smart cities. IEEE Trans Netw Sci Eng 7(4):2231–2242

    Article  Google Scholar 

  • Gupta M, Gao J, Aggarwal C, Han J (2013) Outlier detection for temporal data: a survey. IEEE Trans Knowl Data Eng 26(9):2250–2267

    Article  Google Scholar 

  • Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  • Hsieh RJ, Chou J, Ho CH (2019) Unsupervised online anomaly detection on multivariate sensing time series data for smart manufacturing. In: 2019 IEEE 12th conference on service-oriented computing and applications (SOCA), pp 90–97. IEEE

  • Hundman K, Constantinou V, Laporte C, Colwell I, Soderstrom T (2018) Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp 387–395

  • Karim F, Majumdar S, Darabi H, Harford S (2019) Multivariate lstm-fcns for time series classification. Neural Netw 116:237–245

    Article  Google Scholar 

  • Kwon D, Kim H, Kim J, Suh SC, Kim I, Kim KJ (2019) A survey of deep learning-based network anomaly detection. Clust Comput 22:949–961

    Article  Google Scholar 

  • Li D, Chen D, Jin B, Shi L, Goh J, Ng S-K (2019b) Mad-gan: multivariate anomaly detection for time series data with generative adversarial networks. In: Artificial neural networks and machine learning–ICANN 2019: text and time series: 28th international conference on artificial neural networks, Munich, Germany, September 17–19, 2019, Proceedings, Part IV

  • Li D, Chen D, Jin B, Shi L, Goh J, Ng S-K(2019a) Mad-gan: multivariate anomaly detection for time series data with generative adversarial networks. In: International conference on artificial neural networks, pp 703–716. Springer

  • Li Y, Zhu Z, Kong D, Han H, Zhao Y (2019) Ea-lstm: evolutionary attention-based lstm for time series prediction. Knowl-Based Syst 181:104785

    Article  Google Scholar 

  • Li Z, Xu W, Huang A, SM (2012) Dimensionality reduction for anomaly detection in electrocardiography: a manifold approach. In: 2012 ninth international conference on wearable and implantable body sensor networks, pp 161–165. IEEE

  • Lin S, Clark R, Birke R, Schönborn S, Trigoni N, Roberts S (2020) Anomaly detection for time series using vae-lstm hybrid model. In: ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp 4322–4326. IEEE

  • Liu X, Liu H, Guo Q, Zhang C (2020) Adaptive wavelet transform model for time series data prediction. Soft Comput 24:5877–5884

    Article  Google Scholar 

  • Mahesh B (2020) Machine learning algorithms-a review. Int J Sci Res (IJSR). 9:381–386

    Google Scholar 

  • Malhotra P, Ramakrishnan A, Anand G, Vig L, Agarwal P, Shroff G (2016). Lstm-based encoder-decoder for multi-sensor anomaly detection. arXiv preprint arXiv:1607.00148

  • Malhotra P, Vig L, Shroff G, Agarwal P, et al. (2015) Long short term memory networks for anomaly detection in time series. In: ESANN, vol 2015, p 89

  • Mathur AP, Tippenhauer NO (2016) Swat: a water treatment testbed for research and training on ics security. In: 2016 international workshop on cyber-physical systems for smart water networks (CySWater), pp 31–36. IEEE

  • Mehrang S, Helander E, Pavel M, Chieh A, Korhonen I (2015) Outlier detection in weight time series of connected scales. In: 2015 IEEE international conference on bioinformatics and biomedicine (BIBM), pp 1489–1496. IEEE

  • Niu Z, Ke Yu, Xiaofei W (2020) Lstm-based vae-gan for time-series anomaly detection. Sensors 20(13):3738

    Article  Google Scholar 

  • Ostad-Ali-Askari K, Shayannejad M, Ghorbanizadeh-Kharazi H (2017) Artificial neural network for modeling nitrate pollution of groundwater in marginal area of zayandeh-rood river, isfahan, iran. KSCE J Civ Eng 21:134–140

    Article  Google Scholar 

  • Pang G, Shen C, Cao L, Van Den Hengel A (2021) Deep learning for anomaly detection: a review. ACM Comput Surv (CSUR) 54(2):1–38

    Article  Google Scholar 

  • Park D, Hoshi Y, Kemp CC (2018) A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder. IEEE Robot Autom Lett 3(3):1544–1551

    Article  Google Scholar 

  • Park D, Hoshi Y, Kemp CC (2018) A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder. IEEE Robot Autom Lett 3(3):1544–1551

    Article  Google Scholar 

  • Pincombe B (2005) Anomaly detection in time series of graphs using arma processes. Asor Bull 24(4):2

    Google Scholar 

  • Pumsirirat A, Liu Y (2018) Credit card fraud detection using deep learning based on auto-encoder and restricted boltzmann machine. Int J Adv Comput Sci Appl, 9(1)

  • Reddy A, Ordway-West M, Lee M, Dugan M, Whitney J, Kahana R, Ford B, Muedsam J, Henslee A, Rao M (2017) Using gaussian mixture models to detect outliers in seasonal univariate network traffic. In: 2017 IEEE security and privacy workshops (SPW), pp 229–234. IEEE

  • Sakurada M, Yairi T (2014) Anomaly detection using autoencoders with nonlinear dimensionality reduction. In: Proceedings of the MLSDA 2014 2nd workshop on machine learning for sensory data analysis, pp 4–11

  • Shazeer N, Lan Z, Cheng Y, Ding N, Le H (2020) Talking-heads attention. arXiv preprint arXiv:2003.02436

  • Shyu ML, Chen SC, Sarinnapakorn K, Chang L (2003) A novel anomaly detection scheme based on principal component classifier. Technical report, Miami Univ Coral Gables Fl Dept of Electrical and Computer Engineering

  • Su Y, Zhao Y, Niu C, Liu R, Sun W, Pei D (2019) Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp 2828–2837

  • Vaswani A (2017) Attention is all you need. Adv Neural Inf Process Syst 30:1

    Google Scholar 

  • Wang Y, Xin D, Zhihui L, Duan Q, Jie W (2022) Improved lstm-based time-series anomaly detection in rail transit operation environments. IEEE Trans Ind Inf 18(12):9027–9036

    Article  Google Scholar 

  • Xu B, Huawei S, Bingjie S, Rong A, Qi C, Xueqi C (2021) Towards consumer loan fraud detection: graph neural networks with role-constrained conditional random field. Proc AAAI Conf Artif Intell 35(5):4537–4545

    Google Scholar 

  • Yeh IC, Lien CH (2009) The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Exp Syst Appl 36(2):2473–2480

    Article  Google Scholar 

  • Yu X, Xianfei Y, Qingji T, Chun S, Zhihan L (2022) An edge computing based anomaly detection method in IOT industrial sustainability. Appl Soft Comput 128:109486

  • Zhang Y, Chen Y, Wang J, Pan Z (2021) Unsupervised deep anomaly detection for multi-sensor time-series signals. IEEE Trans Knowl Data Eng Zong B, Song Q, Martin RM, Wei C, Cristian L, Daeki C, Haifeng C (2018) Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In: International conference on learning representations

  • Zong B, Song Q, Martin RM, Wei C, Cristian L, Daeki C, Haifeng C (2018) Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In: International conference on learning representations

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant No. 62072274), the Shandong Provincial Transfer and Transformation Project of Scientific and Technological Achievements (Grant No.2021LYXZ021), and the Special Funds of Taishan Scholars Project of Shandong Province(No.tstp20221137).

Funding

The authors have not disclosed any funding.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, 250014, China

    Kai Jiang, Hui Liu & Yuxiu Lin

  2. Shandong Key Laboratory of Blockchain Finance, Jinan, 250014, China

    Kai Jiang

  3. Key Laboratory of Digital Media Technology of Shandong Province, Jinan, 250014, China

    Hui Liu & Yuxiu Lin

  4. Institute of Information Technology, Shandong Academy of Agricultural Sciences, Jinan, 250000, China

    Huaijun Ruan & Jia Zhao

Authors
  1. Kai Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hui Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huaijun Ruan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jia Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuxiu Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hui Liu.

Ethics declarations

Conflict of interest

The author declares that he has no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, K., Liu, H., Ruan, H. et al. ALAE: self-attention reconstruction network for multivariate time series anomaly identification. Soft Comput 27, 10509–10519 (2023). https://doi.org/10.1007/s00500-023-08467-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-023-08467-4

Keywords

Search

Navigation