Skip to main content
SpringerLink
Log in

Probabilistic autoencoder with multi-scale feature extraction for multivariate time series anomaly detection

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Effectively detecting anomalies for multivariate time series is of great importance for the modern industrial system. Recently, reconstruction-based deep learning methods have been widely used in time series anomaly detection. However, the rich local and global characteristics of time series may not be well captured by methods that compress and reconstruct time series through a single-scale neural network. In addition, under the influence of a complex environment, small fluctuations occur during the normal operation of the system, which will also bring challenges for those methods to reconstruct exact values. In this paper, we propose an unsupervised multivariate time series anomaly detection method based on a probabilistic autoencoder with multi-scale feature extraction (PAMFE). The multiple parallel dilated convolutions with different dilation factors and feature fusion module enable PAMFE to capture overall and detailed information of time series to identify various types of anomalies better. Furthermore, considering the normal fluctuation of data, we reconstruct the expected distribution of input and calculate the anomaly score based on the probability that the input belongs to the distribution. Extensive experiments on four publicly real-world datasets demonstrate that PAMFE outperforms state-of-the-art methods in F1-Score. Moreover, we investigate the contributions of the major components of PAMFE, and the experimental results show that they all contribute to performance improvement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Algorithm 2
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Liu H, Zheng C, Li D et al (2021) EDMF: efficient deep matrix factorization with review feature learning for industrial recommender system. IEEE Trans Indust Inform 18(7):4361–4371. https://doi.org/10.1109/TII.2021.3128240

    Article  Google Scholar 

  2. Liu H, Zheng C, Li D et al (2022) Multi-perspective social recommendation method with graph representation learning. Neurocomputing 468:469–481. https://doi.org/10.1016/j.neucom.2021.10.050https://doi.org/10.1016/j.neucom.2021.10.050

    Article  Google Scholar 

  3. Li Z, Liu H, Zhang Z et al (2021) Learning knowledge graph embedding with heterogeneous relation attention networks. IEEE Trans Neural Netw Learn Syst, https://doi.org/10.1109/TNNLS.2021.3055147https://doi.org/10.1109/TNNLS.2021.3055147

  4. Chen Z, Chen D, Zhang X et al (2022) Learning graph structures with transformer for multivariate time-series anomaly detection in IoT. IEEE Internet Things J 9(12):9179–9189. https://doi.org/10.1109/JIOT.2021.3100509

    Article  Google Scholar 

  5. Cai Z, Zheng X (2018) A private and efficient mechanism for data uploading in smart cyber-physical systems. IEEE Trans Netw Sci Eng 7(2):766–775. https://doi.org/10.1109/TNSE.2018.2830307

    Article  MathSciNet  Google Scholar 

  6. Blázquez-García A, Conde A, Mori U et al (2021) A review on outlier/anomaly detection in time series data. ACM Comput Surveys (CSUR) 54(3):1–33. https://doi.org/10.1145/3444690

    Article  Google Scholar 

  7. Xu H, Chen W, Zhao N et al (2018) Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In: Proceedings of the 2018 world wide web conference, pp 187–196, https://doi.org/10.1145/3178876.3185996

  8. Zhou B, Liu S, Hooi B et al (2019) Beatgan: anomalous rhythm detection using adversarially generated time series. In: Proceedings of the twenty-eighth international joint conference on artificial intelligence, pp 4433–4439, https://doi.org/10.24963/ijcai.2019/616

  9. Pang G, Shen C, Cao L et al (2021) Deep learning for anomaly detection: a review. ACM Computing Surveys (CSUR) 54(2):1–38. https://doi.org/10.1145/3439950

    Article  Google Scholar 

  10. Park D, Hoshi Y, Kemp C C (2018) A multimodal anomaly detector for robot-assisted feeding using an lstm-based variational autoencoder. IEEE Robot Autom Lett 3(3):1544–1551. https://doi.org/10.1109/LRA.2018.2801475

    Article  Google Scholar 

  11. Audibert J, Michiardi P, Guyard F et al (2020) Usad: unsupervised anomaly detection on multivariate time series. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp 3395–3404. https://doi.org/10.1145/3394486.3403392

  12. Liang H, Song L, Wang J et al (2021) Robust unsupervised anomaly detection via multi-time scale dcgans with forgetting mechanism for industrial multivariate time series. Neurocomputing 423:444–462. https://doi.org/10.1016/j.neucom.2020.10.084

    Article  Google Scholar 

  13. Jiang J, Yasakethu L (2013) Anomaly detection via one class svm for protection of scada systems. In: 2013 international conference on cyber-enabled distributed computing and knowledge discovery. IEEE, pp 82–88, https://doi.org/10.1109/CyberC.2013.22

  14. Liu FT, Ting KM, Zhou ZH (2008) Isolation forest. In: 2008 eighth IEEE international conference on data mining. IEEE, pp 413–422, https://doi.org/10.1109/ICDM.2008.17

  15. Liu H, Liu T, Zhang Z et al (2022a) ARHPE: asymmetric relation-aware representation learning for head pose estimation in industrial human–computer interaction. https://doi.org/10.1109/TII.2022.3143605, vol 18, pp 7107–7117

  16. Liu H, Liu T, Chen Y et al (2022b) EHPE: skeleton cues-based gaussian coordinate encoding for efficient human pose estimation. IEEE Trans Multi:1–12. https://doi.org/10.1109/TMM.2022.3197364https://doi.org/10.1109/TMM.2022.3197364

  17. Wu J, Zeng W, Yan F (2018) Hierarchical temporal memory method for time-series-based anomaly detection. Neurocomputing 273:535–546. https://doi.org/10.1016/j.neucom.2017.08.026

    Article  Google Scholar 

  18. Hundman K, Constantinou V, Laporte C et al (2018) Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp 387–395, https://doi.org/10.1145/3219819.3219845

  19. Lai G, Chang WC, Yang Y et al (2018) Modeling long-and short-term temporal patterns with deep neural networks. In: The 41st international ACM SIGIR conference on research & development in information retrieval, pp 95–104, https://doi.org/10.1145/3209978.3210006

  20. Malhotra P, Ramakrishnan A, Anand G (2016) Lstm-based encoder-decoder for multi-sensor anomaly detection. In: Anomaly detection workshop at 33rd ICML

  21. Zong B, Song Q, Min MR et al (2018) Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In: International conference on learning representations, pp 1–19

  22. Thill M, Konen W, Wang H et al (2021) Temporal convolutional autoencoder for unsupervised anomaly detection in time series. Appl Soft Comput 112:107,751. https://doi.org/10.1016/j.asoc.2021.107751https://doi.org/10.1016/j.asoc.2021.107751

    Article  Google Scholar 

  23. oodfellow I, Pouget-Abadie J, Mirza M et al (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  24. Tuli S, Casale G, Jennings N R (2022) TranAD: deep transformer networks for Anomaly Detection in multivariate time series data. Proc VLDB Endow 15(6):1201—1214. https://doi.org/10.14778/3514061.3514067https://doi.org/10.14778/3514061.3514067

    Article  Google Scholar 

  25. Su Y, Zhao Y, Niu C et al (2019) Robust anomaly detection for multivariate time series through stochastic recurrent neural network. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp 2828–2837, https://doi.org/10.1145/3292500.3330672

  26. Malhotra P, Vig L, Shroff G et al (2015) Long short term memory networks for anomaly detection in time series. In: European symposium on artificial neural networks, pp 89–94

  27. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. Adv Neural Inform Process Syst:30

  28. Li T, Comer ML, Delp EJ et al (2020) Anomaly scoring for prediction-based anomaly detection in time series. In: 2020 IEEE aerospace conference. IEEE, pp 1–7, https://doi.org/10.1109/AERO47225.2020.9172442

  29. Li X, Luan Y, Chen L (2022) Semi-supervised GAN with similarity constraint for mode diversity. Appl Intell, pp 1–14, https://doi.org/10.1007/s10489-022-03771-2

  30. He Q, Zheng Y, Zhang C et al (2020) Mtad-tf: multivariate time series anomaly detection using the combination of temporal pattern and feature pattern. Complexity 2020:1–9. https://doi.org/10.1155/2020/8846608

    Google Scholar 

  31. Wang X, Han J, Li B et al (2021) Automatic ICD-10 coding based on multi-head attention mechanism and gated residual network. In: 2021 IEEE international conference on bioinformatics and biomedicine (BIBM), pp 536–543, https://doi.org/10.1109/BIBM52615.2021.9669625

  32. Vincent P, Larochelle H, Lajoie I et al (2010) Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J Mach Learn Res 11(12):3371–3408

    MathSciNet  MATH  Google Scholar 

  33. Tao Q, Liu F, Li Y et al (2019) Air pollution forecasting using a deep learning model based on 1d convnets and bidirectional gru. IEEE Access 7:76,690–76,698. https://doi.org/10.1109/ACCESS.2019.2921578

    Article  Google Scholar 

  34. Siffer A, Fouque PA, Termier A et al (2017) Anomaly detection in streams with extreme value theory. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp 1067–1075, https://doi.org/10.1145/3097983.3098144

  35. Pot A, Porkert J, Keijzer M (2019) The bidirectional in bilingual: cognitive, social and linguistic effects of and on third-age language learning. Behavior Sci 9(9):98. https://doi.org/10.3390/bs9090098

    Article  Google Scholar 

  36. Sun X, Gao Y, Sutcliffe R et al (2019) Word representation learning based on bidirectional grus with drop loss for sentiment classification. IEEE Trans Syst, Man, Cybern: Syst 51(7):4532–4542. https://doi.org/10.1109/TSMC.2019.2940097

    Article  Google Scholar 

  37. Liu G (2019) Bidirectional lstm with attention mechanism and convolutional layer for text classification. Neurocomputing 337:325–338. https://doi.org/10.1016/j.neucom.2019.01.078

    Article  Google Scholar 

  38. Schuster M, Paliwal K K (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673–2681. https://doi.org/10.1109/78.650093

    Article  Google Scholar 

  39. Sun Y, Xue B, Zhang M et al (2020) Automatically designing CNN architectures using the genetic algorithm for image classification. IEEE Trans Cybern 50(9):3840–3854. https://doi.org/10.1109/TCYB.2020.2983860https://doi.org/10.1109/TCYB.2020.2983860

    Article  Google Scholar 

  40. Munir M, Siddiqui S A, Dengel A et al (2018) Deepant: a deep learning approach for unsupervised anomaly detection in time series. IEEE Access 7:1991–2005. https://doi.org/10.1109/ACCESS.2018.2886457https://doi.org/10.1109/ACCESS.2018.2886457

    Article  Google Scholar 

  41. Ratul M A R, Mozaffari M H, Lee W S et al (2020) Skin lesions classification using deep learning based on dilated convolution. BioRxiv p 860700, https://doi.org/10.1101/860700

  42. Murphy KP (2022) Probabilistic machine learning: an introduction. MIT Press, USA

    MATH  Google Scholar 

  43. Rootzén H, Segers J, Wadsworth J L (2018) Multivariate generalized Pareto distributions: parametrizations, representations, and properties. J Multivariate Anal 165:117–131. https://doi.org/10.1016/j.jmva.2017.12.003https://doi.org/10.1016/j.jmva.2017.12.003

    Article  MathSciNet  MATH  Google Scholar 

  44. Mathur AP, Tippenhauer NO (2016) Swat: a water treatment testbed for research and training on ics security. In: 2016 international workshop on cyber-physical systems for smart water networks. IEEE, pp 31–36, https://doi.org/10.1109/CySWater.2016.7469060https://doi.org/10.1109/CySWater.2016.7469060

  45. Ahmed CM, Palleti VR, Mathur AP (2017) Wadi: a water distribution testbed for research in the design of secure cyber physical systems. In: Proceedings of the 3rd international workshop on cyber-physical systems for smart water networks, pp 25–28, https://doi.org/10.1145/3055366.3055375

  46. Abdulaal A, Liu Z, Lancewicki T (2021) Practical approach to asynchronous multivariate time series anomaly detection and localization. In: Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp 2485–2494, https://doi.org/10.1145/3447548.3467174

  47. Abadi M, Agarwal A, Barham P et al (2015) TensorFlow: large-scale machine learning on heterogeneous systems. Available at: http://tensorflow.org

  48. Chollet F et al (2015) Keras. Available at: https://github.com/fchollet/keras

  49. Nothaft FA, Massie M, Danford T et al (2015) Rethinking data-intensive science using scalable analytics systems. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp 631–646, https://doi.org/10.1145/2723372.2742787https://doi.org/10.1145/2723372.2742787

  50. Fang W, Zhang F, Sheng V S et al (2018) A method for improving CNN-based image recognition using DCGAN. Comput, Materials Continua 57(1):167–178. https://doi.org/10.32604/cmc.2018.02356https://doi.org/10.32604/cmc.2018.02356

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank their colleagues from the machine learning group for discussions on this paper.

Author information

Authors and Affiliations

  1. School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, 100876, China

    Guangyao Zhang, Xin Gao, Bing Xue, Shiyuan Fu, Jiahao Yu, Zijian Huang & Xu Huang

  2. Production and Operation Department, PipeChina West to East Gas Pipeline Company, Shanghai, 200122, China

    Lei Wang

Authors
  1. Guangyao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bing Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shiyuan Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jiahao Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zijian Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xu Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Guangyao Zhang: Methodology, Software, Writing - Original Draft, Writing - Review & Editing. Xin Gao: Conceptualization, Methodology, Supervision, Writing - Original Draft, Writing - Review & Editing. Lei Wang: Conceptualization, Resources. Bing Xue: Software, Validation. Shiyuan Fu: Software, Validation. Jiahao Yu: Software, Writing - Review & Editing. Zijian Huang: Writing - Review & Editing. Xu Huang: Writing - Review & Editing.

Corresponding author

Correspondence to Xin Gao.

Ethics declarations

Competing Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Data Availabilty

The datasets supporting the results of this article are SMD, SWaT, WADI and PSM public datasets, and the authors confirm that the datasets are indicated in the reference list.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, G., Gao, X., Wang, L. et al. Probabilistic autoencoder with multi-scale feature extraction for multivariate time series anomaly detection. Appl Intell 53, 15855–15872 (2023). https://doi.org/10.1007/s10489-022-04324-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04324-3

Keywords

Search

Navigation