Stability of a hierarchical clustering

https://doi.org/10.1016/0031-3203(80)90042-4Get rights and content

Abstract

Clustering algorithms have the annoying habit of finding clusters even when the data are generated randomly. Verifying that potential clusterings are real in some objective sense is receiving more attention as the number of new clustering algorithms and their applications grow. We consider one aspect of this question and study the stability of a hierarchical structure with a variation on a measure of stability proposed in the literature.(1,2)

Our measure of stability is appropriate for proximity matrices whose entries are on an ordinal scale. We randomly split the data set, cluster the two halves, and compare the two hierarchical clusterings with the clustering achieved on the entire data set. Two stability statistics, based on the Goodman-Kruskal rank correlation coefficient, are defined. The distributions of these statistics are estimated with Monte Carlo techniques for two clustering methods (single-link and complete-link) and under two conditions (randomly selected proximity matrices and proximity matrices with good hierarchical structure). The stability measures are applied to some real data sets.

References (26)

  • R. Dubes et al.

    Clustering techniques: the user's dilemma

    Pattern Recognition

    (1976)
  • R. Dubes et al.

    Validity studies in clustering methodologies

    Pattern Recognition

    (1979)
  • J.S. Strauss

    Classification by cluster analysis

  • J.S. Strauss et al.

    The use of clustering techniques for the classification of psychiatric patients

    Br. J. Psychiat.

    (1973)
  • M.R. Anderberg

    Cluster Analysis for Applications

    (1973)
  • B. Everitt

    Cluster Analysis

    (1974)
  • J.A. Hartigan

    Clustering Algorithms

    (1975)
  • R.M. Cormack

    A review of classification

    Jl R. statist. Soc.

    (1971)
  • F.B. Baker et al.

    A graph-theoretic approach to goodness-of-fit in complete-link hierarchical clustering

    J. Am. statist. Ass.

    (1976)
  • S. Fillenbaum et al.

    Structures in the Subjective Lexicon

    (1971)
  • L. Hubert

    Approximate evaluation techniques for the single-link and complete-link hierarchical clustering procedures

    J. Am. statist. Ass.

    (1974)
  • R.F. Ling

    On the theory and construction of k-clusters

    Comput. J.

    (1972)
  • R.F. Ling

    Probability theory of cluster analysis

    J. Am. statist. Ass.

    (1973)

    • Cited by (0)

    1

    Stephen P. Smith was born in Cincinnati, Ohio on September 4, 1955. He received the B.S. and M.S. degrees in Computer Science from Michigan State University, East Lansing in 1977 and 1979, respectively.

    ∗

    Research supported by NSF grant ENG 76-11936A01.

    View full text
    Copyright © 1980 Published by Elsevier B.V.