Abstract
Sensors in complex industrial systems generate multivariate time series data, frequently leading to diverse abnormal patterns that pose challenges for detection. The existing multivariate abnormal detection methods may encounter difficulties when applied to datasets with low dimensions or sparse relationships between variables. To address these issues, this study proposes a two-stage adversarial Transformer-based anomaly detection method. On the one hand, an autoregressive temporal convolutional network component is embedded before the multi-head attention module to capture features encompassing long-term and local information. Besides, this component utilizes a trainable neural network instead of the vanilla Transformer’s absolute position encoding, resulting in enhanced position information. On the other hand, the proposed two-stage adversarial learning strategy allows the model to effectively learn intricate multivariate data patterns via constraining latent space, thereby enhancing anomaly detection performance. Our method achieves F1 scores of 0.9679, 0.7947, and 0.6452 on a real-world dataset and two public industrial sensor datasets, demonstrating superior overall anomaly detection performance compared to recent advanced works.










Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability
The public dataset SKAB is available at https://www.kaggle.com/datasets/yuriykatser/skoltech-anomaly-benchmark-skab. The public dataset NAB is available at https://www.kaggle.com/datasets/boltzmannbrain/nab. The real-world dataset SAT is not available for privacy protection.
References
Gültekin O, Cinar E, Özkan K, Yazıcı A (2022) Multisensory data fusion-based deep learning approach for fault diagnosis of an industrial autonomous transfer vehicle. Expert Syst Appl 200:117055
Amini N, Zhu Q (2022) Fault detection and diagnosis with a novel source-aware autoencoder and deep residual neural network. Neurocomputing 488:618–633
Audibert J, Michiardi P, Guyard F, Marti S, Zuluaga MA (2020) USAD: unsupervised anomaly detection on multivariate time series. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. pp 3395–3404
Li G, Jung JJ (2023) Deep learning for anomaly detection in multivariate time series: approaches, applications, and challenges. Information Fusion 91:93–102
Zhou Y, Liang X, Zhang W, Zhang L, Song X (2021) VAE-based deep SVDD for anomaly detection. Neurocomputing 453:131–140
Harrou F, Dairi A, Taghezouit B, Sun Y (2019) An unsupervised monitoring procedure for detecting anomalies in photovoltaic systems using a one-class support vector machine. Sol Energy 179:48–58
Sarmadi H, Karamodin A (2020) A novel anomaly detection method based on adaptive Mahalanobis-squared distance and one-class kNN rule for structural health monitoring under environmental effects. Mech Syst Signal Process 140:106495
Imrana Y, Xiang Y, Ali L, Abdul-Rauf Z (2021) A bidirectional LSTM deep learning approach for intrusion detection. Expert Syst Appl 185:115524
Paula Monteiro R, Lozada MC, Mendieta DRC, Loja RVS, Bastos Filho CJA (2022) A hybrid prototype selection-based deep learning approach for anomaly detection in industrial machines. Expert Systems with Applications 117528
Deng H, Qian G, Luo D, Lv X, Liu H, Li H (2023) MRS-Net: an image inpainting algorithm with multi-scale residual attention fusion. Appl Intell 53(7):7497–7511
He P, Wang L, Cui Y, Wang R, Wu D (2023) Unsupervised feature learning based on autoencoder for epileptic seizures prediction. Applied Intelligence 1–19
Bang J, Park J, Park J (2023) GACaps-HTC: graph attention capsule network for hierarchical text classification. Applied Intelligence 1–18
Li L, Yan J, Wang H, Jin Y (2020) Anomaly detection of time series with smoothness-inducing sequential variational auto-encoder. IEEE Trans Neural Netw Learn Syst 32(3):1177–1191
Maleki S, Maleki S, Jennings NR (2021) Unsupervised anomaly detection with LSTM autoencoders using statistical data-filtering. Appl Soft Comput 108:107443
Thill M, Konen W, Wang H, Bäck T (2021) Temporal convolutional autoencoder for unsupervised anomaly detection in time series. Appl Soft Comput 112:107751
Chen S, Jin G, Ma X (2022) Detection and analysis of real-time anomalies in large-scale complex system. Measurement 184:109929
Jiang B, Chen S, Wang B, Luo B (2022) MGLNN: semi-supervised learning via multiple graph cooperative learning neural networks. Neural Netw 153:204–214
Wu Y, Dai HN, Tang H (2021) Graph neural networks for anomaly detection in industrial internet of things. IEEE Internet Things J 9(12):9214–9231
He Z, Chen P, Li X, Wang Y, Yu G, Chen C, Li X, Zheng Z (2023) A spatiotemporal deep learning approach for unsupervised anomaly detection in cloud systems. IEEE Trans Neural Netw Learn Syst 34(4):1705–1719
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30:1–11
Wu X, Tang B, Zhao M, Wang J, Guo Y (2023) STR transformer: a cross-domain transformer for scene text recognition. Appl Intell 53(3):3444–3458
Wu H, Xu J, Wang J, Long M (2021) Autoformer: decomposition transformers with auto-correlation for long-term series forecasting. Adv Neural Inf Process Syst 34:22419–22430
Zerveas G, Jayaraman S, Patel D, Bhamidipaty A, Eickhoff C (2021) A transformer-based framework for multivariate time series representation learning. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. pp 2114–2124
Makhzani A, Shlens J, Jaitly N, Goodfellow I, Frey B (2015) Adversarial autoencoders. arXiv:1511.05644
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2020) Generative adversarial networks. Commun ACM 63(11):139–144
Jang K, Hong S, Kim M, Na J, Moon I (2021) Adversarial autoencoder based feature learning for fault detection in industrial processes. IEEE Trans Industr Inf 18(2):827–834
Zhou X, Hu Y, Liang W, Ma J, Jin Q (2020) Variational LSTM enhanced anomaly detection for industrial big data. IEEE Trans Industr Inf 17(5):3469–3477
Benidis K, Rangapuram SS, Flunkert V, Wang Y, Maddix D, Turkmen C, Gasthaus J, Bohlke-Schneider M, Salinas D, Stella L et al (2022) Deep learning for time series forecasting: tutorial and literature survey. ACM Comput Surv 55(6):1–36
Xing Y, Zhu J, Li Y, Huang J, Song J (2023) An improved spatial temporal graph convolutional network for robust skeleton-based action recognition. Appl Intell 53(4):4592–4608
Cheng Y, Xu Y, Zhong H, Liu Y (2020) Leveraging semisupervised hierarchical stacking temporal convolutional network for anomaly detection in IoT communication. IEEE Internet Things J 8(1):144–155
Zhan J, Wu C, Ma X, Yang C, Miao Q, Wang S (2022) Abnormal vibration detection of wind turbine based on temporal convolution network and multivariate coefficient of variation. Mech Syst Signal Process 174:109082
Zhou H, Zhang S, Peng J, Zhang S, Li J, Xiong H, Zhang W (2021) Informer: Beyond efficient transformer for long sequence time-series forecasting. In: Proceedings of the AAAI conference on artificial intelligence, vol 35. pp 11106–11115
Wang X, Pi D, Zhang X, Liu H, Guo C (2022) Variational transformer-based anomaly detection approach for multivariate time series. Measurement 191:110791
Cai L, Janowicz K, Mai G, Yan B, Zhu R (2020) Traffic transformer: capturing the continuity and periodicity of time series for traffic forecasting. Trans GIS 24(3):736–755
Lara-Benítez P, Carranza-García M, Riquelme JC (2021) An experimental review on deep learning architectures for time series forecasting. Int J Neural Syst 31(03):2130001
Li S, Jin X, Xuan Y, Zhou X, Chen W, Wang YX, Yan X (2019) Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems 32
Chen RQ, Shi GH, Zhao WL, Liang CH (2021) A joint model for it operation series prediction and anomaly detection. Neurocomputing 448:130–139
Shao H, Xiao Z, Yao S, Sun D, Zhang A, Liu S, Wang T, Li J, Abdelzaher T (2021) ControlVAE: tuning, analytical properties, and performance analysis. IEEE Trans Pattern Anal Mach Intell 44(12):9285–9297
Creswell A, White T, Dumoulin V, Arulkumaran K, Sengupta B, Bharath AA (2018) Generative adversarial networks: an overview. IEEE Signal Process Mag 35(1):53–65
Wang X, Du Y, Lin S, Cui P, Shen Y, Yang Y (2020) advAE: a self-adversarial variational autoencoder with Gaussian anomaly prior knowledge for anomaly detection. Knowl-Based Syst 190:105187
Huang Y, Tang Y, VanZwieten J (2021) Prognostics with variational autoencoder by generative adversarial learning. IEEE Trans Industr Electron 69(1):856–867
Larsen ABL, Sønderby SK, Larochelle H, Winther O (2016) Autoencoding beyond pixels using a learned similarity metric. In: International conference on machine learning, PMLR. pp 1558–1566
Katser ID, Kozitsin VO (2020) Skoltech Anomaly Benchmark (SKAB). Kaggle. https://doi.org/10.34740/KAGGLE/DSV/1693952
Ahmad S, Lavin A, Purdy S, Agha Z (2017) Unsupervised real-time anomaly detection for streaming data. Neurocomputing 262:134–147
Yin C, Zhang S, Wang J, Xiong NN (2020) Anomaly detection based on convolutional recurrent autoencoder for IoT time series. IEEE Trans Syst Man Cybern Syst 52(1):112–122
Breunig MM, Kriegel HP, Ng RT, Sander J (2000) LOF: identifying density based local outliers. In: Proceedings of the 2000 ACM SIGMOD international conference on management of data. pp 93–104
Zhao Y, Nasrullah Z, Hryniewicki MK, Li Z (2019) LSCP: locally selective combination in parallel outlier ensembles. In: Proceedings of the 2019 SIAM international conference on data mining. SIAM, pp 585–593
Li Z, Xiang Z, Gong W, Wang H (2022) Unified model for collective and point anomaly detection using stacked temporal convolution networks. Appl Intell 52(3):3118–3131
Homayouni H, Ghosh S, Ray I, Gondalia S, Duggan J, Kahn MG (2020) An autocorrelation-based LSTM-autoencoder for anomaly detection on time-series data. In: 2020 IEEE international conference on big data (Big Data). IEEE, pp 5068–5077
Su Y, Zhao Y, Niu C, Liu R, Sun W, Pei D (2019) Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. pp 2828–2837
Zhao H, Wang Y, Duan J, Huang C, Cao D, Tong Y, Xu B, Bai J, Tong J, Zhang Q (2020) Multivariate time-series anomaly detection via graph attention network. In: 2020 IEEE international conference on data mining (ICDM). IEEE, pp 841–850
Deng A, Hooi B (2021) Graph neural network-based anomaly detection in multivariate time series. In: Proceedings of the AAAI conference on artificial intelligence, vol 35. pp 4027–4035
Woolson RF (2007) Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials 1–3
Podobnik B, Stanley HE (2008) Detrended cross-correlation analysis: a new method for analyzing two nonstationary time series. Phys Rev Lett 100(8):084102
Author information
Authors and Affiliations
Contributions
Junfu Chen carried out the method design, participated in the coding for the experiments, and drafted the manuscript. Dechang Pi provided the GPU service and polished the manuscript. Xixuan Wang participated in reproducing comparison methods.
Corresponding author
Ethics declarations
Ethical and informed consent for data used
Informed consent for publication of this paper and data usage was obtained from the Nanjing University of Aeronautics and Astronautics and all authors.
Conflict of interest
The authors declare that no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Chen, J., Pi, D. & Wang, X. A two-stage adversarial Transformer based approach for multivariate industrial time series anomaly detection. Appl Intell 54, 4210–4229 (2024). https://doi.org/10.1007/s10489-024-05395-0
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-024-05395-0