A new fuzzy MLE-clustering approach based on object-to-group probabilistic distance measure: from anomaly detection to multi-fault classification in datacenter computational nodes

Abstract

Datacenters are expanding in size and complexity to the point where anomaly detection and infrastructure monitoring become critical challenges. One potential strategy for dealing with the reliability of computational nodes in a datacenter is to identify cluster nodes or virtual machines exhibiting anomalous behavior. Throughout this paper, we introduce a novel clustering approach for analyzing cluster node behavior while running various workloads in a system based on resource usage details (CPU utilization, network events, etc.). The new clustering technique aims at boosting the efficiency of fuzzy clustering algorithms based on the maximum likelihood estimation (MLE) scheme. We propose the use of a recently developed object-to-group distance since it does not involve the computation of distances among all pairs of objects to assign the objects to the most appropriate group. The experimental findings under realistic settings demonstrate that the newly implemented algorithm outperforms many similar algorithms that have been used frequently in such tasks.

Appendix: p-Friedman test

The Friedman test is an extension of the Wilcoxon Signed Rank Test and the non-parametric equivalent of the 1-factor analysis of variance with repeated measures (Hubert and Arabie 1985). The Friedman test assumes the null hypothesis that k dependent variables belong to the same population. For the position parameter of a sample i by $M_{i}$, we denote the null hypothesis by $H_{0}$ and the alternative hypothesis by $H_{a}$ by the following:

$$\begin{aligned}&H_{0} : M_{1}= M_{2}=\ldots = M_{k} \\&H_{a} : M_{i} \ne M_{j} \text { for at least one } \left( i,j \right) \\&\qquad \text {pair of the Friedman test.} \end{aligned}$$

Given the Friedman null hypothesis, the expected summed ranks of each group are equal to $\frac{n(k + 1)}{2}$. The Friedman test statistic is expressed as follows:

$$\begin{aligned} Q=\frac{12}{nk(k+1)}\sum _{j=1}^{k}\left\{ R_{j} -\frac{n(k+1)}{2} \right\} ^{2} \end{aligned}$$

(15)

where $R_{i}$ is the sum of the ranks for the sample i.