Abstract
In the complex and variable SDDC (Software Defined Data Center) environment, in order to ensure that applications and services are undisturbed, it needs to closely monitor various KPI (Key Performance Indicators, such as CPU utilization, number of online users, request response delays, etc.) streams of resources and services. However, the frequent launch and update of applications, which produces new KPIs or change the data characteristics of KPIs monitored, so that the original anomaly detection model may become unavailable. So we propose ADT-SHL (Anomaly Detection through Transferring the Shared-hidden-layers model). It first clusters historical KPIs by similarity, and then all KPIs in each cluster are through Shared-hidden-layers method to train VAE (Variational Auto-Encoder) anomaly detection model so as to reconstruct KPIs with corresponding characteristics. When a new KPI is generated, ADT-SHL classify it into similar cluster, and finally model of the cluster is transferred and fine-tuned to detect the new KPI anomalies. This process is rapid, accurate and without manual tuning or labeling for new KPI. and its F-scores range 0.69 to 0.959 for the studied KPIs from two cloud vendors and others, ADT-SHL is only lower a state-of-art supervised method under all labelings by 3.05% and greatly outperforming a state-of-art unsupervised method by 67.46%, and compared with them, ADT-SHL reduce model training time by 94% on average.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Software Defined Data Center. https://www.vmware.com/cn/solutions/software-defined-datacenter.html. Accessed 23 Jun 2018
Bu, J., Liu, Y., Zhang, S., et al.: Rapid deployment of anomaly detection models for large number of emerging KPI streams. In: 2018 IEEE 36th International Performance Computing and Communications Conference (IPCCC), pp. 1–8. IEEE. Orlando (2018)
Chen, Y., Mahajan, R., Sridharan, B., et al.: A provider-side view of web search response time. In: ACM SIGCOMM Computer Communication Review, vol. 43, no. 4, pp. 243–254. ACM, New York (2013)
Choffnes, D.R., Bustamante, F.E., Ge, Z.: Crowdsourcing service-level network event monitoring. ACM SIGCOMM Comput. Commun. Rev. 41(4), 387–398 (2011)
Liu, D., Zhao, Y., Xu, H., et al.: Opprentice: towards practical and automatic anomaly detection through machine learning. In: Proceedings of the 2015 Internet Measurement Conference, pp. 211–224. ACM, Tokyo (2015)
Xu, H., Chen, W., Zhao, N., et al.: Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in web applications. In: Proceedings of the 2018 World Wide Web Conference on World Wide Web. International World Wide Web Conferences Steering Committee, Lyon, France, pp. 187–196 (2018)
Li, Z., Chen, W., Pei, D.: Robust and unsupervised KPI anomaly detection based on conditional variational autoencoder. In: 2018 IEEE 37th International Performance Computing and Communications Conference (IPCCC), pp. 1–9. IEEE, Orlando (2018)
Li, Z., Zhao, Y., Liu, R., et al.: Robust and rapid clustering of kpis for large-scale anomaly detection. In: 2018 IEEE/ACM 26th International Symposium on Quality of Service (IWQoS), pp. 1–10. IEEE, Banff (2018)
Csurka, G.: Domain adaptation for visual applications: a comprehensive survey. arXiv preprint arXiv:1702.05374 (2017)
Petitjean, F., Forestier, G., Webb, G.I., et al.: Dynamic time warping averaging of time series allows faster and more accurate classification. In: 2014 IEEE International Conference on Data Mining, pp. 470–479. IEEE, Shenzhen (2014)
Zhang, Y.L., Li, L., Zhou, J., et al.: Anomaly detection with partially observed anomalies. In: Companion of the Web Conference 2018 on The Web Conference 2018. International World Wide Web Conferences Steering Committee, Lyon, France, pp. 639–646 (2018)
Diederik, P., Welling, M.: Auto-encoding variational Bayes. In: Proceedings of the International Conference on Learning Representations (2013)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on International Conference on Machine Learning, Lille, France, vol. 37, pp. 448–456 (2015)
Alibaba Cluster Data. https://github.com/alibaba/clusterdata/tree/master/cluster-trace-v2018. Accessed 10 May 2019
Baidu Public Data. https://github.com/baidu/Curve. Accessed 10 Apr 2019
Yahoo Public Data. https://webscope.sandbox.yahoo.com/. Accessed 15 Jun 2019
Acknowledgment
This work was is supported by the National Natural Science Foundation of China (61762008), the Natural Science Foundation Project of Guangxi (2017GXNSFAA198141) and Key R&D project of Guangxi (No. Guike AB17195014).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Duan, X., Chen, N., Xie, Y. (2019). Intelligent Detection of Large-Scale KPI Streams Anomaly Based on Transfer Learning. In: Jin, H., Lin, X., Cheng, X., Shi, X., Xiao, N., Huang, Y. (eds) Big Data. BigData 2019. Communications in Computer and Information Science, vol 1120. Springer, Singapore. https://doi.org/10.1007/978-981-15-1899-7_26
Download citation
DOI: https://doi.org/10.1007/978-981-15-1899-7_26
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-1898-0
Online ISBN: 978-981-15-1899-7
eBook Packages: Computer ScienceComputer Science (R0)
