Smoothing dissimilarities to cluster binary data

Abstract

Cluster analysis attempts to group data objects into homogeneous clusters on the basis of the pairwise dissimilarities among the objects. When the data contain noise, we might consider performing a smoothing operation, either on the data themselves or on the dissimilarities, before implementing the clustering algorithm. Possible benefits to such pre-smoothing are discussed in the context of binary data. We suggest a method for cluster analysis of binary data based on “smoothed” dissimilarities. The smoothing method presented borrows ideas from shrinkage estimation of cell probabilities. Some simulation results are given showing that improvement in the accuracy of the clustering result is obtained via smoothing, especially in the case in which the observed data contain substantial noise. The method is illustrated with an example involving binary test item response data.

Introduction

Cluster analysis is the statistical technique of separating objects, or observations, into homogeneous groups on the basis of (typically multivariate) data for several variables. We often picture the variables as continuous, but there is a substantial literature about clustering objects based on binary data (e.g., Everitt et al. (2001) and Kaufman and Rousseeuw (1990)).

When the data contain some type of noise (whether measurement error or merely unexplained variability), it is intuitive that smoothing the data, when done properly, may better recapture the underlying process generating the data. Often individual data values contain substantial noise and, thus, are less trustworthy to reflect the process we hope to understand. Smoothing methods attempt to reduce this noise by balancing the information in individual data points with information in the data set as a whole, or by shrinking data values toward some assumed structural model. Cluster analysis itself may be viewed as a type of smoothing, in the sense of being a technique to obtain a less complex structure from noisy data. However, standard clustering methods can be sensitive to outliers that could exist when we directly cluster observed data. Therefore clustering a smoothed version of the data may be preferable to clustering the observed (unsmoothed) data.

In certain situations the idea of smoothing is natural. For example, Hitchcock et al. (2007) showed that a shrinkage method of smoothing could aid in the clustering of functional data (data arising as curves). With binary data, the idea of “smoothing” seems less natural than with functional data, but the concept of shrinkage will be an important one in the methods discussed here.

A common method for clustering binary data objects is to define pairwise dissimilarities among the objects, each of which is typically a function of the number of matches (or mismatches) among the $P$ binary variables measured on the pair of objects. A “match” occurs when, for a certain variable, both objects share the same value (both 0 or both 1). For any pair of objects a 2×2 table of matches and mismatches may be constructed. Our smoothing method will fundamentally use this table.

In Section 2 we will formally define the dissimilarities for a set of binary data and introduce a clustering method based on a smoothed version of this collection of dissimilarities. Section 3 describes a simulation study to determine the effect of this smoothing method on the accuracy of the cluster analysis. In Section 4, we apply the method to a real data set involving test item responses, and Section 5 is a conclusion.