Abstract
In a large-scale cloud environment, many key performance indicators (KPIs) of entities are monitored in real time. These multivariate time series consist of high-dimensional, high-noise, random and time-dependent data. As a common method implemented in artificial intelligence for IT operations (AIOps), time series anomaly detection has been widely studied and applied. However, the existing detection methods cannot fully consider the influence of multiple factors and cannot quickly and accurately detect anomalies in multivariate KPIs of entities. Concurrently, fine-grained root cause locations cannot be determined for detected anomalies and often require abundant normal data that are difficult to obtain for model training. To solve these problems, we propose a long short-term memory (LSTM)-based semisupervised variational autoencoder (VAE) anomaly detection strategy called LR-SemiVAE. First, LR-SemiVAE uses VAE to perform feature dimension reduction and reconstruction of multivariate time series data and judges whether the entity is abnormal by calculating the reconstruction probability score. Second, by introducing an LSTM network into the VAE encoder and decoder, the model can fully learn the time dependence of multivariate time series. Then, LR-SemiVAE predicts the data labels by introducing a classifier to reduce the dependence on the original labeled data during model training. Finally, by proposing a new evidence lower bound (ELBO) loss function calculation method, LR-SemiVAE pays attention to the normal pattern and ignores the abnormal pattern during training to reduce the time cost of removing random anomaly and noise data. However, due to the limitations of LSTM in learning the long-term dependence of time series data, based on LR-SemiVAE, we propose a transformer-based semisupervised VAE anomaly detection and location strategy called RT-SemiVAE for cluster systems with complex service dependencies. This method learns the long-term dependence of multivariate time series by introducing a parallel multihead attention mechanism transformer, while LSTM is used to capture short-term dependence, and the introduction of parallel computing also markedly reduces model training time. After RT-SemiVAE detects entity anomalies, it traces the root entities according to the obtained service dependence graph and locates the root causes at the indicator level. We verify the strategies by using public data sets and constructing a system prototype. Experimental results show that compared with existing baseline methods, the LR-SemiVAE and RT-SemiVAE strategies can detect anomalies more quickly and accurately and perform fine-grained and accurate localization of the root causes of anomalies.
Similar content being viewed by others
References
Borghesi A et al (2019) A semisupervised autoencoder-based approach for anomaly detection in high performance computing systems. Eng Appl Artif Intell 85:634–644
Notaro P, Cardoso J, Gerndt M (2021) A survey of AIOps methods for failure management. ACM Trans on Intell Sys and Tech (TIST) 12.6:1–45
He S et al (2021) A survey on automated log analysis for reliability engineering. ACM Comp Surveys (CSUR) 54.6:1–37
Yadav RB, Kumar PS, Dhavale SV (2020) A survey on log anomaly detection using deep learning. 2020 8th International Conference on Reliability, Infocom Technologies and Optimi zation (Trends and Future directions)(ICRITO). IEEE
Blázquez-García A et al (2021) A review on outlier/anomaly detection in time series data. ACM Comp Surveys (CSUR) 54.3:1–33
Kingma DP, Welling M (2014) Auto-encoding variational bayes. In: ICLR
Goodfellow I et al (2014) Generative adversarial nets. Advances in neural information processing systems. vol 27
Hundman K et al (2018) Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining
Li D et al (2019) MAD-GAN: multivariate anomaly detection for time series data with generative adversarial networks. International conference on artificial neural networks. Springer, Cham
Park D, Hoshi Y, Kemp CC (2018) A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder. IEEE Robotics and Automation Lett 3.3:1544–1551
Lin S et al (2020) Anomaly detection for time series using vae-lstm hybrid model. ICASSP 2020-2020 IEEE international conference on acoustics speech and signal processing. (ICASSP) IEEE
Niu Z, Yu K, Wu X (2020) LSTM-Based VAE-GAN for time-series anomaly detection. Sensors 20.13:3738
Razavi-Far R et al (2018) Information fusion and semi-supervised deep learning scheme for diagnosing gear faults in induction machine systems. IEEE Trans on Industrial Elect 66.8:6331–6342
Xu Haowen et al (2018) Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. Proceedings of the 2018 world wide web conference
Lindemann B‘ et al (2021) A survey on anomaly detection for technical systems using LSTM networks. Comp in Industry 131:103498
Ergen T, Kozat SS (2019) Unsupervised anomaly detection with LSTM neural networks. IEEE Trans on Neural Networks and Learning Sys 31.8:3127–3141
Zhou X et al (2020) Variational LSTM enhanced anomaly detection for industrial big data. IEEE Trans on Industrial Informatics 17.5:3469–3477
Huang F et al (2018) Multimodal network embedding via attention based multi-view variational autoencoder. Proceedings of the 2018 ACM on international conference on multimedia retrieval
Lin S et al (2020) Anomaly detection for time series using vae-lstm hybrid model. ICASSP 2020-2020 IEEE international conferenc on acoustics, speech and signal processing (ICASSP). IEEE
Maleki S, Maleki S, Jennings NR (2021) Unsupervised anomaly detection with LSTM autoencoders using statistical data-filtering. Applied Soft Computing 108:107443
Geiger A et al (2020) TadGAN: time series anomaly detection using generative adversarial networks. 2020 IEEE international conference on big data (Big Data). IEEE
Bashar MA, Nayak R (2020) TANoGAN: time series anomaly detection with generative adversarial networks. 2020 IEEE symposium series on computational intelligence (SSCI). IEEE
Vaswani A et al (2017) Attention is all you need. Advances in neural information processing systems. vol 30
Phongwattana T, Chan JH (2019) Development of biomedical corpus enlargement platform using BERT for bio-entity recognition. International conference on neural information processing. Springer Cham
He J et al (2019) HSI-BERT: hyperspectral image classification using the bidirectional encoder representation from transformers. IEEE Trans on Geoscience and Remote Sensing 58.1:165–178
Ziyu Z, Wang Q (2019) R-transformer network based on position and self-attention mechanism for aspect-level sentiment classification. IEEE Access 7:127754–127764
Bian J et al (2019) A novel and efficient CVAE-GAN-based approach with informative manifold for semi-supervised anomaly detection. IEEE Access 7:88903–88916
Das A et al (2020) An End-to-End Approach for Benchmarking Time-Series Models Using Autoencoders. Proceedings of the Global AI Congress 2019. Springer, Singapore
Zhang S et al (2020) Semi-supervised bearing fault diagnosis and classification using variational autoencoder-based deep generative models. IEEE Sensors J 21.5:6476–6486
Song M, Zhang C, Haihong E (2018) An auto scaling system for API gateway based on Kubernetes. 2018 IEEE 9th international conference on software engineering and service science (ICSESS). IEEE
Chang C-C et al (2017) A kubernetes-based monitoring platform for dynamic cloud resource provisioning. GLOBECOM 2017-2017 IEEE global communications conference. IEEE
Lee J-W et al (2019) Collaborative distillation for top-N recommendation. 2019 IEEE international conference on data mining (ICDM). IEEE
Wu Z et al (2020) A comprehensive survey on graph neural networks. IEEE Trans on Neural Networks and Learning Systems 32.1:4–24
Acknowledgements
This work is supported in part by the National Key Research and Development Project of China under Grant 2017YFC1602005 and 2018YFB1404404, the Natural Science Foundation of China under Grant 62162003 and 61762008, and the Innovation Project of Guangxi Graduate Education under Grant YCSW2022075.
Author information
Authors and Affiliations
Contributions
Ningjiang Chen: Conceptualization, Methodology, Resources, Supervision, Funding acquisition. Huan Tu: Software, Investigation, Resources, Writing - review & editing, Visualization. Xiaoyan Duan: Validation, Formal analysis, Writing original draft. Liangqing Hu: Data collection and curation. Chengxiang Guo: Data processing and analysis.
Corresponding author
Ethics declarations
Competing interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Chen, N., Tu, H., Duan, X. et al. Semisupervised anomaly detection of multivariate time series based on a variational autoencoder. Appl Intell 53, 6074–6098 (2023). https://doi.org/10.1007/s10489-022-03829-1
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-022-03829-1