DStab: estimating clustering quality by distance stability

Abstract

Most commonly, stability analyses are performed using an external validation measure. For example, the Jaccard index is one of the indexes of choice for stability measurement. The index is wrapped around a resampling method to sense the model’s stability. Other methods use classifiers to look for stable partitions instead. In these cases, a resampling method is also used with an external index, an error measure driven by a classifier, and a clustering algorithm aiming to find stable clustering model configurations. Contrary to previous stability-based methods, we propose a novel validation procedure consisting of an internal validation index within a resampling strategy. We propose an index based on the distance between cluster centroids coupled with a twofold cross-validation resampling approach. Moreover, we use a threshold based on a null hypothesis to detect meaningful clustering partitions. As part of our experimental study, we have selected the K-means algorithm because of its simplicity but primarily for its instability compared to other algorithms, such as Hierarchical methods. Finally, we compare our approach with several known validation indexes and discuss the results. Our findings show that our method cannot only find meaningful clustering partitions but is also helpful as an unsupervised data analysis tool.

Appendix

Here we include Tables 4 and 5, where we summarize Figs. 7,8 9,  10 and 11. In Table 4, we present clusters estimation for artificial datasets, and in Table 5, we estimate the number of groups on real datasets. Tables have three columns, the first with the dataset name, the second with the number of clusters estimation, and the third with a p-value. We analyze the relation between DStab and the reference for each k to determine candidates. In our analyses, we view DStab as an observed distribution and the reference as a Null Hypothesis. We assume that the observed data and the null come from the same distribution but have different locations. Hence, we can use a non-parametric test to estimate if DStab distribution median is smaller than the reference. In this case, a smaller median means better stability-results presented in Tables 4 and 5 show the number of clusters k with the lowest p-values after performing a Mann–Whitney U test. Fewer solutions mean that candidates have a p-value above the 0.05 threshold, and a hyphen indicates no candidate is below the 0.05 threshold. Also, solutions inside a parenthesis mean equivalent candidates. For example, Meter D dataset has three identical solutions (2-3-4), we report a single p-value for the identical solutions.

Fig. 7

We display the individual DStab results on 8 artificial datasets. The blue plot is the observed DStab score, and the orange plot is DStab reference. “Appendix” discusses candidate selection and Table 2 summarizes validation results for DStab and other methods (color figure online)

Fig. 8

Artificial data visualization. Panel a displays the Unbalance dataset, and Panel b the Gauss dataset

Fig. 9

We display the individual DStab results on 8 UCI datasets. The blue plot is the observed DStab score, and the orange plot is DStab reference. “Appendix” discusses candidate selection and Table 3 summarizes validation results for DStab and other methods (color figure online)

Fig. 10

We display the individual DStab results on 4 Genomic datasets. The blue plot is the observed DStab score, and the orange plot is DStab reference. “Appendix” discusses candidate selection and Table 3 summarizes validation results for DStab and other methods (color figure online)

Fig. 11

Real data discriminant components visualization. Panel a displays the Meter D dataset’s first and second components, and panel b shows second and third components. Panel c depicts Leukemia’s dataset first and second components. Panel d displays the Fetal Health dataset’s first and second components

Table 4 We perform a Mann–Whitney U test between DStab and DStab Ref., assuming that significant results have a lower median value than the reference

Table 5 We perform a Mann–Whitney U test between DStab and DStab Ref., assuming that significant results have a lower median value than the reference