Skip to main content
SpringerLink
Log in

ADET: anomaly detection in time series with linear time

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Time series data is ubiquitous in financial, biomedical, and other areas. Anomaly detection in time series has been widely researched in these areas. However, most existing algorithms suffer from “curse of dimension” and may lose some information in the process of feature extraction. In this paper, we propose two new data structures named interval table (ITable) and extend interval table (EITable) for time series representation to capture more original information. We also proposed ADET: a novel Anomaly Detection algorithm based on EITable, which only needs linear time to detect meaningful anomalies. Extensive experiments on eleven data sets of UCR Repository, MIT-BIH datasets, and the BIDMC database show that ADET has overall good performance in terms of AUC-ROC and outperforms other algorithms in time complexity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Ahmed M, Mahmood A, Rafiqul IM (2016) A survey of anomaly detection techniques in financial domain. Future Gener Comput Syst 55:278–288. https://doi.org/10.1016/j.future.2015.01.001

    Article  Google Scholar 

  2. Atkinson PM, Jeganathan C, Dash J, Atzberger C (2012) Inter-comparison of four models for smoothing satellite sensor time-series data to estimate vegetation phenology. Remote Sens Environ 123:400–417. https://doi.org/10.1016/j.rse.2012.04.001

    Article  Google Scholar 

  3. Baim DS, Colucci WS, Monrad ES, Smith HS, Wright RF, Lanoue AS, Gauthier DF, Ransil BJ, Grossman W, Braunwald E (1986) Survival of patients with severe congestive heart failure treated with oral milrinone. J Am Coll Cardiol 7(3):661–670. https://doi.org/10.1016/S0735-1097(86)80478-8

    Article  Google Scholar 

  4. Basu S, Meckesheimer M (2007) Automatic outlier detection for time series: an application to sensor data. Knowl Inf Syst 11:137–154. https://doi.org/10.1007/s10115-006-0026-6

    Article  Google Scholar 

  5. Breunig MM, Kriegel HP, Ng RT, Sander J (2000) LOF: identifying density-based local outliers. SIGMOD Rec 29(2):93–104. https://doi.org/10.1145/335191.335388

    Article  Google Scholar 

  6. Chakraborti A, Patriarca M, Santhanam MS (2007) Financial time-series analysis: a brief overview. In: Chatterjee A, Chakrabarti BK (eds) Econophysics of markets and business networks. New economic windows. Springer, Milano. https://doi.org/10.1007/978-88-470-0665-2_4

    Chapter  Google Scholar 

  7. Chauhan S, Vig L (2015) Anomaly detection in ECG time signals via deep long short-term memory networks. In: 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA). pp 1–7. https://doi.org/10.1109/DSAA.2015.4487273

  8. Chen Y, Keogh E, Hu B, Begum N, Bagnall A, Mueen A, Batista G (2015) The UCR time series classification archive. http://www.cs.ucr.edu/eamonn/time_series_data/. Retrieved on 24 Sep 2019

  9. Chen J, Sathe S, Aggarwa C, Turaga D (2017) Outlier detection with autoencoder ensembles. In: Proceedings of the 2017 SIAM international conference on data mining, Society for Industrial and Applied Mathematics, pp 90–98

  10. Chen RQ, Shi GH, Zhao WL, Liang CH (2019) Sequential VAE-LSTM for anomaly detection on time series. arXiv preprint arXiv:1910.03818

  11. Esra G, Eksi Z, Murat Ç (2012) WebECG: a novel ECG simulator based on matlab web figure. Adv Eng Softw 45(1):167–174. https://doi.org/10.1016/j.advengsoft.2011.09.005

    Article  Google Scholar 

  12. Goldberger AL, Amaral LAN, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng C, Stanley HE (2000) Physiobank, physiotoolkit, and physionet components of a new research resource for complex physiologic signals. Circulation 101(23):215–220

    Article  Google Scholar 

  13. Guo C, Li H, Pan D (2010) An improved piecewise aggregate approximation based on statistical features for time series mining. In: KSEM 2010: knowledge science, engineering and management, pp 234–244. https://doi.org/10.1007/978-3-642-15280-1_23

  14. Haemwaan S, Ratanamahatana CA (2015) Robust and accurate anomaly detection in ECG artifacts using time series motif discovery. Comput Math Methods Med 2015:1–20. https://doi.org/10.1155/2015/453214

    Article  MathSciNet  Google Scholar 

  15. Kampouraki A, Manis G, Nikou C (2009) Heartbeat time series classification with support vector machines. IEEE Trans Inf Technol Biomed 13(4):512–518. https://doi.org/10.1109/TITB.2008.2003323

    Article  Google Scholar 

  16. Keogh E, Kasetty S (2002) On the need for time series data mining benchmarks: a survey and empirical demonstration. In: Proceedings of the 8th ACM SIGKDD international conference on knowledge discovery and data mining, July 23–26, 2002, Edmonton, Alberta, Canada, pp 102–111

  17. Keogh E, Chakrabarti K, Pazzani M, Mehrotra S (2001) Locally adaptive dimensionality reduction for indexing large time series databases. Int Conf Manag Data 30(2):151–162. https://doi.org/10.1145/375663.375680

    Article  MATH  Google Scholar 

  18. Koski A, Juhola M, Meriste M (1995) Syntactic recognition of ECG signals by attributed finite automata. Pattern Recogn 28(12):1927–1940. https://doi.org/10.1016/0031-3203(95)00052-6

    Article  Google Scholar 

  19. Latecki LJ, Lazarevic A, Pokrajac D (2007) Outlier detection with kernel density functions. In: Proceedings of the international conference on machine learning and data mining in pattern recognition, pp 61–75

  20. Li G, Bräysy O, Jiang L, Wu Z, Wang Y (2013) Finding time series discord based on bit representation clustering. Knowl Based Syst 54(4):243–254. https://doi.org/10.1016/j.knosys.2013.09.015

    Article  Google Scholar 

  21. Li D, Chen D, Shi L, Jin B, Goh J, Ng S (2019) MAD-GAN: multivariate anomaly detection for time series data with generative adversarial networks. In: Artificial neural networks and machine learning – ICANN 2019: text and time series, pp 703–716. https://doi.org/10.1007/978-3-030-30490-4_56

  22. Lin J, Keogh EJ, Lonardi S, Chiu BY (2003) A symbolic representation of time series, with implications for streaming algorithms, pp 2–11. https://doi.org/10.1145/882082.882086

  23. Liu FT, Ting KM, Zhou ZH (2008) Isolation forest. In 2008 Eighth IEEE international conference on data mining, IEEE, pp 413–422

  24. Lkhagva B, Suzuki Y, Kawagoe K (2006) New time series data representation ESAX for financial applications. p 115. https://doi.org/10.1109/ICDEW.2006.99

  25. Malhotra P, Vig L, Shroff G, Agarwal P (2015) Long short term memory networks for anomaly detection in time series. In: European symposium on artificial neural networks, vol 89. Presses universitaires de Louvain, pp 89–94

  26. Moody GB, Mark RG (2001) The impact of the MIT-BIH arrhythmia database. IEEE Eng Med Biol Mag 20(3):45–50. https://doi.org/10.1109/51.932724

    Article  Google Scholar 

  27. Moonesignhe HDK, Pang-ning T (2006) Outlier detection using random walks. In: 2006 18th IEEE international conference on tools with artificial intelligence (ICTAI'06), pp 532–539. https://doi.org/10.1109/ICTAI.2006.94

  28. Ocak H (2009) Automatic detection of epileptic seizures in EEG using discrete wavelet transform and approximate entropy. Expert Syst Appl 36(2):2027–2036. https://doi.org/10.1016/j.eswa.2007.12.065

    Article  Google Scholar 

  29. Ratanamahatana CA, Lin J, Gunopulos D, Keogh E, Vlachos M, Das G (2010) Mining time series data. In: Maimon O, Rokach L (eds) Data mining and knowledge discovery handbook. Springer, Berlin, pp 1049–1077. https://doi.org/10.1007/0-387-25465-x_36

    Chapter  Google Scholar 

  30. Ren H, Liu M, Li Z, Pedrycz W (2017) A piecewise aggregate pattern representation approach for anomaly detection in time series. Knowl Based Syst 135:29–39. https://doi.org/10.1016/j.knosys.2017.07.021

    Article  Google Scholar 

  31. Ren H, Liu M, Liao X, Liang L, Ye Z, Li Z (2018) Anomaly detection in time series based on interval sets. IEEJ Trans Electr Electron Eng 13(5):757–762. https://doi.org/10.1002/tee.22626

    Article  Google Scholar 

  32. Schlegl T, Seebock P, Waldstein SM, Schmidterfurth U, Langs G (2017) Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. Int Conf Inf Process. https://doi.org/10.1007/978-3-319-59050-9_12

    Article  Google Scholar 

  33. Sun Y, Li J, Liu J, Sun B, Chow CWK (2014) An improvement of symbolic aggregate approximation distance measure for time series. Neurocomputing 138:189–198. https://doi.org/10.1016/j.neucom.2014.01.045

    Article  Google Scholar 

  34. Tang B, He H (2017) A local density-based approach for outlier detection. Neurocomputing 241:171–180

    Article  Google Scholar 

  35. Tran KP, Du Nguyen H, Thomassey S (2019) Anomaly detection using long short term memory networks and its applications in supply chain management. IFAC Pap OnLine 52(13):2408–2412

    Article  Google Scholar 

  36. Virani N, Jha DK, Ray A, Phoha S (2019) Sequential hypothesis tests for streaming data via symbolic time-series analysis. Eng Appl Artif Intell 81:234–246. https://doi.org/10.1016/j.engappai.2019.02.015

    Article  Google Scholar 

  37. Xuan PT, Anh DT (2018) An efficient hash-based method for time series motif discovery. Multidiscipl Trends Artif Intell. https://doi.org/10.1007/978-3-030-03014-8_17

    Article  Google Scholar 

  38. Yahyaoui H, Aldaihani R (2019) A novel trend based SAX reduction technique for time series. Expert Syst Appl 130:113–123. https://doi.org/10.1016/j.eswa.2019.04.026

    Article  Google Scholar 

  39. Zenati H, Romain M, Foo C, Lecouat B, Chandrasekhar V (2018) Adversarially learned anomaly detection. Int Conf Data Min. https://doi.org/10.1109/ICDM.2018.00088

    Article  Google Scholar 

  40. Zong B, Song Q, Min MR, Cheng W, Lumezanu C, Cho D, Chen H (2018) Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In: International conference on learning representations

Download references

Acknowledgements

This study was supported by the Shenzhen Research Council (Grant No. GJHZ20180928155209705).

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, China

    Chunkai Zhang, Wei Zuo, Ao Yin, Xuan Wang & Chuanyi Liu

Authors
  1. Chunkai Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ao Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xuan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chuanyi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chuanyi Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (RAR 285 kb)

Supplementary file2 (RAR 49 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, C., Zuo, W., Yin, A. et al. ADET: anomaly detection in time series with linear time. Int. J. Mach. Learn. & Cyber. 12, 271–280 (2021). https://doi.org/10.1007/s13042-020-01171-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-020-01171-x

Keywords

Search

Navigation