Abstract
Anomaly detection is very important in the field of operation and maintenance (O&M). However, in O&M, we find that direct use of the existing anomaly detection algorithms often causes a large number of false positives, and the detection results are not stable. Nothing a data characteristics in O&M: Many anomalies are often anomalous time periods formed by continuous anomaly points, we propose a novel concept “Scores Sequence” and a method based on learning of Scores Sequence. Our method has less false positives, can detect anomaly timely, and the detection result of our method is very stable. Through comparative experiments with many algorithms and practical industrial application, it proves that our method has good performance and is very suitable for the anomaly detection in O&M.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Liu, D., Zhao, Y., Xu, H., et al.: Opprentice: towards practical and automatic anomaly detection through machine learning. In: Internet Measurement Conference, pp. 211–224. ACM (2015)
Breunig, M.M., Kriegel, H.-P., Ng, R.T., Sander, J.: LOF: identifying density-based local outliers. In: Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data (SIGMOD), pp. 93–104 (2000)
Knorr, E., Ng, R.: Algorithms for mining distance-based outliers in large data sets. In: VLDB Conference (1998)
Knorr, E., Ng, R., Tucakov, V.: Distance-based outliers: algorithms and applications. VLDB J. 8(3–4), 237–253 (2000)
Barnett, V., Lewis, T.: Outliers in Statistical Data. Wiley, Chichester (1994)
Tao, Y., Xiao, X., Zhou, S.: Mining distance-based outliers from large databases in any metric space. In: SIGKDD Conference, pp. 394–403 (2006)
Angiulli, F., Fassetti, F.: Detecting distance-based outliers in streams of data. In: CIKM Conference, pp. 811–820 (2007)
Kontaki, M., Gounaris, A., Papadopoulos, A.N., et al.: Continuous monitoring of distance-based outliers over data streams. In: IEEE International Conference on Data Engineering, pp. 135–146. IEEE (2011)
Zhu, Y., Shasha, D.: Statstream: statistical monitoring of thousands of data streams in real time. In: VLDB Conference, pp. 358–369 (2002)
Liu, F.T., Kai, M.T., Zhou, Z.H.: Isolation-based anomaly detection. ACM Trans. Knowl. Discov. Data 6(1), 1–39 (2012)
Gupta, C., Bansal, M., Chuang, T.C., et al.: Astro: a predictive model for anomaly detection and feedback-based scheduling on Hadoop. In: IEEE International Conference on Big Data, pp. 854–862 (2014)
Gu, X., Wang, H.: Online anomaly prediction for robust cluster systems. In: IEEE International Conference on Data Engineering, pp. 1000–1011. IEEE Computer Society (2009)
Gu, X., Papadimitriou, S., Yu, P.S., et al.: Toward predictive failure management for distributed stream processing systems. In: The International Conference on Distributed Computing Systems, pp. 825–832. IEEE (2002)
Vanerio, J., Casas, P.: Ensemble-learning approaches for network security and anomaly detection. In: The Workshop, pp. 1–6 (2017)
Aggarwal, C.C.: Outlier ensembles: position paper. ACM SIGKDD Explor. Newslett. 14(2), 49–58 (2013)
Gupta, M., Gao, J., Aggarwal, C., et al.: Outlier detection for temporal data: a survey. IEEE Trans. Knowl. Data Eng. 26(9), 2250–2267 (2014)
Rousseeuw, P., Van Driessen, K.: A fast algorithm for the minimum covariance determinant estimator. Technometrics 41(3), 212–223 (1999)
Yang, D., Rundensteiner, E.A., Ward, M.O.: Neighbor based pattern detection for windows over streaming data. In: Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology (EDBT), pp. 529–540 (2009)
Dasgupta, D., Nino, F.: A comparison of negative and positive selection algorithms in novel pattern detection. In: Proceedings of the 2000 IEEE International Conference on Systems, Man, and Cybernetics, vol. 1, pp. 117–125 (2000)
Liaw, A., Wiener, M.: Classification and regression with random forest. R News 23(23), 18–22 (2002)
Luan, T., Fan, L., Shahabi, C.: Distance-based outlier detection in data streams. VLDB Endow. 9, 1089–1100 (2016)
UCI datasets: http://archive.ics.uci.edu/ml
Acknowledgments
This work is supported by the National Key Research and Development Program (No. 2016YFB1000703).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, D., Shi, S., Zhang, Y., Wang, H., Luo, J. (2018). An Anomaly Detection Method Based on Learning of “Scores Sequence”. In: Zhou, Q., Miao, Q., Wang, H., Xie, W., Wang, Y., Lu, Z. (eds) Data Science. ICPCSEE 2018. Communications in Computer and Information Science, vol 902. Springer, Singapore. https://doi.org/10.1007/978-981-13-2206-8_25
Download citation
DOI: https://doi.org/10.1007/978-981-13-2206-8_25
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-2205-1
Online ISBN: 978-981-13-2206-8
eBook Packages: Computer ScienceComputer Science (R0)