Abstract
In warehouse-scale cloud datacenters, co-locating online services and offline batch jobs is an efficient approach to improving datacenter utilization. In this paper, we perform a deep analysis on the released Alibaba workload dataset, from the perspective of anomaly analysis and diagnosis. we first performed raw data preprocessing, including data supplementing, filtering, correlation and aggregation, and generating the container-level, batch-level and server-level resource usage data finally. Then based on the summary data, we illustrate the overall cluster usage distribution of online container services and batch jobs. Obviously, there are several abnormal nodes in the co-located cluster, and we explore the causes of anomalies from three aspects: (1) unbalanced co-located workloads distribution; (2) skew co-located workload resource utilization; (3) system failures or job instance failures. In addition, we also give some cases of abnormal nodes, which show that frequent system failures and unbalanced workload distribution have a great impact on abnormal nodes, the skew co-located workload resource utilization and frequent instance failures are the causes of abnormalities, too.
This work is supported by National Key Research and Development Plan of China No. 2017YFB1001602.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
If the requested memory of one container is greater than 0.9, all the requested memory of containers may be exceed the machine memory, which is obviously unreasonable.
- 2.
abnormal nodes are the nodes which are few and different in the cluster.
- 3.
Here, the workload distribution means the number of workloads on nodes.
- 4.
The number of recording interval is 143.
References
Maximizing CPU resource utilization on alibaba’s servers (2018). https://102.alibaba.com/detail/?id=61
Zhang, Z., Li, C., Tao, Y., Yang, R., Tang, H., Xu, J.: Fuxi: a fault-tolerant resource management and job scheduling system at internet scale. In: Proceedings of the VLDB Endowment (2014)
Lu, C., Ye, K., Xu, G., Xu, C., Bai, T.: Imbalance in the cloud: an analysis on alibaba cluster trace. In: IEEE International Conference on Big Data (Big Data) (2017)
Cheng, Y., Chai, Z., Anwar, A.: Characterizing co-located datacenter workloads: an alibaba case study. https://arxiv.org/abs/1808.02919 (2018)
Liu, Q., Yu, Z.: The elasticity and plasticity in semi-containerized co-locating cloud workload: a view from alibaba trace. In: Proceedings of ACM Symposium on Cloud Computing (SOCC) (2018)
Alibaba trace (2017). https://github.com/alibaba/clusterdata
Liu, F.T., Ting, K.M., Zhou, Z.H.: Isolation forests. In: Proceedings of International Conference on Data Mining (2008)
Wikipedia. k-means clustering (2018). https://en.wikipedia.org/wiki/K-means_clustering
Google cluster workload traces. https://github.com/google/cluster-data
Reiss, C., Tumanov, A., Ganger, G.R., Katz, R.H., Kozuch, M.A.: Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In: The Third ACM Symposium on Cloud Computing(SoCC) (2012)
Zhang, Q., Hellerstein, J.L., Boutaba, R.: Characterizing task usage shapes in google compute clusters. In: Large Scale Distributed Systems and Middleware Workshop(LADIS) (2011)
Liu, Z., Cho, S.: Characterizing machines and workloads on a google cluster. In: 41st International Conference on Parallel Processing Workshops (2012)
Di, S., Kondo, D., Cirne, W.: Characterization and comparison of cloud versus grid workloads. In: IEEE International Conference on Cluster Computing(CLUSTER) (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Ren, R., Li, J., Wang, L., Yin, Y., Cao, Z. (2020). Anomaly Analysis and Diagnosis for Co-located Datacenter Workloads in the Alibaba Cluster. In: Gao, W., Zhan, J., Fox, G., Lu, X., Stanzione, D. (eds) Benchmarking, Measuring, and Optimizing. Bench 2019. Lecture Notes in Computer Science(), vol 12093. Springer, Cham. https://doi.org/10.1007/978-3-030-49556-5_26
Download citation
DOI: https://doi.org/10.1007/978-3-030-49556-5_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-49555-8
Online ISBN: 978-3-030-49556-5
eBook Packages: Computer ScienceComputer Science (R0)