Skip to main content
Springer Nature Link
Log in

An anomaly detection approach for multiple monitoring data series based on latent correlation probabilistic model

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Condition monitoring systems are widely used to monitor the working condition of equipment, generating a vast amount and variety of monitoring data in the process. The main task of surveillance focuses on detecting anomalies in these routinely collected monitoring data, intended to help detect possible faults in the equipment. However, with the rapid increase in the volume of monitoring data, it is a nontrivial task to scan all the monitoring data to detect anomalies. In this paper, we propose an approach called latent correlation-based anomaly detection (LCAD) that efficiently and effectively detects potential anomalies from a large number of correlative isomerous monitoring data series. Instead of focusing on one or more isomorphic monitoring data series, LCAD identifies anomalies by modeling the latent correlation among multiple correlative isomerous monitoring data series, using a probabilistic distribution model called the latent correlation probabilistic model, which helps to detect anomalies according to their relations with the model. Experimental results on real-world data sets show that when dealing with a large number of correlative isomerous monitoring data series, LCAD yields better performances than existing anomaly detection approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. KOMTRAX: http://www.komatsuamerica.com/komtrax

  2. IEM: http://www.sanygroup.com/group/en-us/

  3. The response time delay is a physical property, determined by the manufacturing of the equipment. The response time delay can be shortened by improving the manufacturing, which is beyond the scope of this paper. In this paper, we make the reasonable assumption that most sensors of the same type have the same response time.

References

  1. Aggarwal CC, Philip SY (2008) Outlier detection with uncertain data. In: SDM. SIAM, pp 483–493

  2. Atzori L, Iera A, Morabito G (2010) The internet of things: A survey. Comput Netw 54(15):2787–2805

    Article  MATH  Google Scholar 

  3. Bilmes JA, et al. (1998) A gentle tutorial of the em algorithm and its application to parameter estimation for gaussian mixture and hidden markov models. Int Comput Sci Inst 4(510):126

    Google Scholar 

  4. Chan PK, Mahoney MV (2005) Modeling multiple time series for anomaly detection. In: Data Mining, Fifth IEEE International Conference on, pages 8–pp. IEEE

  5. Das K, Schneider J (2007) Detecting anomalous records in categorical datasets. In: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 220–229. ACM

  6. Dutton WH (2013) The internet of things. Available at SSRN

  7. Fawcett T (2006) An introduction to roc analysis. Pattern Recogn Lett 27(8):861–874

    Article  MathSciNet  Google Scholar 

  8. Fujimaki R, Nakata T, Tsukahara H, Sato A, Yamanishi K (2009) Mining abnormal patterns from heterogeneous time-series with irrelevant features for fault event detection. Stat Anal Data Min 2(1):1–17

    Article  MathSciNet  MATH  Google Scholar 

  9. Ghoting A, Parthasarathy S, Otey ME (2008) Fast mining of distance-based outliers in high-dimensional datasets. Data Min Knowl Disc 16(3):349–364

    Article  MathSciNet  Google Scholar 

  10. Heard NA, Weston DJ, Platanioti K, Hand DJ, et al. (2010) Bayesian anomaly detection methods for social networks. Ann Appl Stat 4(2):645–662

    Article  MathSciNet  MATH  Google Scholar 

  11. Hecht-Nielsen R (1989) Theory of the backpropagation neural network. In: Neural Networks, 1989. IJCNN., International Joint Conference on, pp 593–605. IEEE

  12. Hu X, Hu S, Zhang X, Zhang H, Luo L (2014) Anomaly detection based on local nearest neighbor distance descriptor in crowded scenes. Sci World J:2014

  13. Izakian H, Pedrycz W (2013) Anomaly detection in time series data using a fuzzy c-means clustering. In: IFSA World Congress and NAFIPS Annual Meeting (IFSA/NAFIPS), 2013 Joint, pp 1513–1518. IEEE

  14. Janakiram D, Adi Mallikarjuna Reddy V, Phani Kumar AVU (2006) Outlier detection in wireless sensor networks using bayesian belief networks. In: Communication System Software and Middleware, 2006. Comsware 2006. First International Conference on, pp 1–6. IEEE

  15. Keogh E, Chakrabarti K, Pazzani M, Mehrotra S (2001) Locally adaptive dimensionality reduction for indexing large time series databases. ACM SIGMOD Rec 30(2):151–162

    Article  Google Scholar 

  16. Keogh E, Kasetty S (2003) On the need for time series data mining benchmarks: a survey and empirical demonstration. Data Min Knowl Disc 7(4):349–371

    Article  MathSciNet  Google Scholar 

  17. Kou Y, Lu C-T, Chen D (2006) Spatial weighted outlier detection. In: SDM, pp 614–618. SIAM

  18. Liu D, Lung C-H, Lambadaris I, Seddigh N (2013) Network traffic anomaly detection using clustering techniques and performance comparison. In: Electrical and Computer Engineering (CCECE), 2013 26th Annual IEEE Canadian Conference on, pp 1–4. IEEE

  19. Ma J, Perkins S (2003) Online novelty detection on temporal sequences. In: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, pp 613–618. ACM

  20. Mabu S, Chen C, Lu N, Shimada K, Hirasawa K (2011) An intrusion-detection model based on fuzzy class-association-rule mining using genetic network programming. IEEE Trans Syst Man Cybern Part C Appl Rev 41(1):130–139

    Article  Google Scholar 

  21. Mahoney MV, Chan PK (2002) Learning nonstationary models of normal network traffic for detecting novel attacks. In: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp 376–385. ACM

  22. Marx ML, Larsen RJ (2006) Introduction to mathematical statistics and its applications. Pearson/Prentice Hall

  23. Mascaro S, Nicholso AE, Korb KB (2014) Anomaly detection in vessel tracks using bayesian networks. Int J Approx Reason 55(1):84–98

    Article  Google Scholar 

  24. Moshtaghi M, Havens TC, Bezdek JC, Park L, Leckie C, Rajasegarar S, Keller JM, Palaniswami M (2011) Clustering ellipses for anomaly detection. Pattern Recogn 44(1):55–69

    Article  MATH  Google Scholar 

  25. Otey ME, Ghoting A, Parthasarathy S (2006) Fast distributed outlier detection in mixed-attribute data sets. Data Min Knowl Disc 12(2-3):203–228

    Article  MathSciNet  Google Scholar 

  26. Papadimitriou S, Sun J, Faloutsos C (2005) Streaming pattern discovery in multiple time-series. In: Proceedings of the 31st international conference on Very large data bases, pp 697–708. VLDB Endowment

  27. Peterson LE (2009) K-nearest neighbor. Scholarpedia 4(2):1883

    Article  Google Scholar 

  28. Pokrajac D, Lazarevic A, Latecki LJ (2007) Incremental local outlier detection for data streams. In: Computational Intelligence and Data Mining, 2007. CIDM 2007. IEEE Symposium on, pp 504–515. IEEE

  29. Povinelli RJ, Johnson MT, Lindgren AC, Ye J (2004) Time series classification using gaussian mixture models of reconstructed phase spaces. IEEE Trans Knowl Data Eng 16(6):779–783

    Article  Google Scholar 

  30. Qin M, Hwang K (2004) Frequent episode rules for internet anomaly detection. In: Network Computing and Applications, 2004.(NCA 2004). Proceedings. Third IEEE International Symposium on, pp 161–168. IEEE

  31. Schlechtingen M, Santos IF (2011) Comparative analysis of neural network and regression based condition monitoring approaches for wind turbine fault detection. Mech Syst Signal Process 25(5):1849–1875

    Article  Google Scholar 

  32. Sheikhan M, Jadidi Z (2014) Flow-based anomaly detection in high-speed links using modified gsa-optimized neural network. Neural Comput Appl 24(3-4):599–611

    Article  Google Scholar 

  33. Sotiris VA, Tse PW, Pecht MG (2010) Anomaly detection through a bayesian support vector machine. IEEE Trans Reliab 59(2):277–286

    Article  Google Scholar 

  34. Su M-Y (2011) Real-time anomaly detection systems for denial-of-service attacks by weighted k-nearest-neighbor classifiers. Expert Syst Appl 38(4):3492–3498

    Article  Google Scholar 

  35. Tandon G, Chan PK (2007) Weighting versus pruning in rule validation for detecting network and host anomalies. In: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 697–706. ACM

  36. Thatte G, Mitra U, Heidemann J (2011) Parametric methods for anomaly detection in aggregate traffic. IEEE/ACM Trans Networking (TON) 19(2):512–525

    Article  Google Scholar 

  37. Tsai C-F, Lin C-Y (2010) A triangle area based nearest neighbors approach to intrusion detection. Pattern Recogn 43(1):222–229

    Article  MathSciNet  MATH  Google Scholar 

  38. Turner R, Fernandes E (2013) Optimizing back propagation parameters for anomaly detection. Eur J Eng Innov 10(1):10–19

    Google Scholar 

  39. Wang X, Mueen A, Ding H, Trajcevski G, Scheuermann P, Keogh E (2013) Experimental comparison of representation methods and distance measures for time series data. Data Min Knowl Disc 26(2):275–309

    Article  MathSciNet  Google Scholar 

  40. Wu M, Jermaine C (2006) Outlier detection by sampling with accuracy guarantees. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 767–772. ACM

  41. Yamanishi K, Takeuchi J-I, Williams G, Milne P (2004) On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms. Data Min Knowl Disc 8(3):275–300

    Article  MathSciNet  Google Scholar 

  42. Yi B-K, Faloutsos C (2000) Fast time sequence indexing for arbitrary lp norms. In: proceedings of the 26st Intl Conference on Very Large Databases. VLDB

  43. Yu JX, Qian W, Lu H, Zhou A (2006) Finding centric local outliers in categorical/numerical spaces. Knowl Inf Syst 9(3):309–338

    Article  Google Scholar 

  44. Yu R, He X, Liu Y (2014) Glad: group anomaly detection in social media analysis. In: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 372–381. ACM

  45. Zhang C, Weng N, Chang J, Zhou A (2009) Detecting abnormal trend evolution over multiple data streams. In: Advances in Data and Web Management, pp 285–296. Springer

  46. Zhang J, Wang H (2006) Detecting outlying subspaces for high-dimensional data: the new task, algorithms, and performance. Knowl Inf Syst 10(3):333–355

    Article  Google Scholar 

Download references

Author information

Author notes

  1. Jianwei Ding, Yingbo Liu, Li Zhang & Jianmin Wang

    Present address: East Main Building, School of Software, Tsinghua University, 100084, Beijing, China

Authors and Affiliations

  1. Department of Computer Science and Technology, Tsinghua University, Beijing, China

    Jianwei Ding

  2. Institute of Information System & Engineering, School of Software, Tsinghua University, Beijing, China

    Jianwei Ding, Yingbo Liu, Li Zhang & Jianmin Wang

  3. General Research Institute of SANY Group, Changsha, China

    Yonghong Liu

Authors
  1. Jianwei Ding
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Yingbo Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Li Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Jianmin Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Yonghong Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Jianwei Ding.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ding, J., Liu, Y., Zhang, L. et al. An anomaly detection approach for multiple monitoring data series based on latent correlation probabilistic model. Appl Intell 44, 340–361 (2016). https://doi.org/10.1007/s10489-015-0713-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-015-0713-7

Keywords