Signature test as statistical testing in clustering

Abstract

We propose a new statistical test denoted by signature testing (Sigtest) with the application in clustering and image classification. Sigtest relies on probabilistic validation of empirical distribution function of data. We implement Sigtest to estimate the number of clusters in hierarchical and partitional clustering. In addition, we propose a new adaptive estimation of the vocabulary size in image classification. Simulation results on both real and synthetic data confirm superiority of Sigtest over existing statistical tests in both hierarchical and partitional clustering as it estimates the number of clusters more accurately. Sigtest also shows advantageous in terms of adjusted rand index and variation of information. In addition, using Sigtest for adaptive choice of vocabulary size in bag of visual words improves the efficiency of the SVM classifier as well as reducing the time complexity of the overall algorithm.

Appendices

Appendix 1: Folded normal distribution

Let $\theta$ be a sample of standard Gaussian distribution, $\theta \sim \mathcal {N}(0,1)$, with density function $\phi (\theta )$ as follows:

$$\begin{aligned} \phi (\theta ) = \frac{1}{\sqrt{2\pi }}e^{-\theta ^{2}/2}, \quad \theta \in R \end{aligned}$$

(11)

therefore, the distribution function of $\theta$ can be given as follows:

$$\begin{aligned} \varPhi (\theta ) = \int _{-\infty }^{\theta } \phi (v) \hbox {d}v = \int _{-\infty }^{\theta } \frac{1}{\sqrt{2\pi }}e^{-v^{2}/2} \hbox {d}v, \theta \in R \end{aligned}$$

(12)

We let v be a sample of Gaussian distribution with mean $\mu$ and standard deviation $\sigma$, $v \sim \mathcal {N}(\mu ,\sigma )$. Therefore, random variable V can be written as $V = \mu + \sigma \varTheta$. It follows that $W = |V| = |\mu + \sigma \varTheta |$ is a random variable with a folded normal distribution, where all of the negative values of V are folded to the positive region of the distribution.

Consequently, cdf of the sorted sample w, where $w\in [0,\infty )$, will be as follows:

$$\begin{aligned} F_{a}(w)= & {} P(W\le w) = P(|V| \le w) = P(|\mu +\sigma \varTheta |\le w) \nonumber \\= & {} P(-w \le \mu +\sigma \varTheta \le w) \nonumber \\= & {} P\left( \frac{-w-\mu }{\sigma } \le w \le \frac{w - \mu }{\sigma }\right) \nonumber \\= & {} \varPhi \left( \frac{w-\mu }{\sigma }\right) - \varPhi \left( \frac{-w-\mu }{\sigma }\right) \nonumber \\ \end{aligned}$$

(13)

Since $\varPhi (-\theta )=1-\varPhi (\theta )$, we will have:

$$\begin{aligned} F_{a}(w)= & {} \varPhi \left( \frac{w-\mu }{\sigma }\right) - \varPhi \left( \frac{-w-\mu }{\sigma }\right) \nonumber \\= & {} \varPhi \left( \frac{w-\mu }{\sigma }\right) + \varPhi \left( \frac{w+\mu }{\sigma }\right) - 1 \nonumber \\= & {} \int _{0}^{w}\frac{1}{\sigma \sqrt{(}2\pi )} \left\{ \exp {\left[ -\frac{1}{2}\left( \frac{v+\mu }{\sigma }\right) ^{2}\right] }\right. \nonumber \\&\left. +\exp {\left[ -\frac{1}{2}\left( \frac{v-\mu }{\sigma }\right) ^{2}\right] }\right\} \hbox {d}v\nonumber \\ \end{aligned}$$

(14)

In case of Gaussian mixture models, the above cdf will be calculated as follows [7]:

$$\begin{aligned} F_{a}(z)=\sum _{j=1}^{K}\pi _{j} F_{aj}(z) \end{aligned}$$

(15)

where $F_{aj}(z)$ is the Gaussian cdf of the jth component and $\pi _{j}$ is the mixing factor of that component in the mixture.

Fig. 8

Estimated $\alpha$ and T parameters using Genetic algorithm for different distances between clusters

Appendix 2: Estimation of $\alpha$ and T

For each combination of $\alpha$ and T, similarity score $A(\alpha , T)$ can be calculate using (9):

$$\begin{aligned} A(\alpha , T) = {\left\{ \begin{array}{ll} 1 \quad Sigtest_{score}(\alpha ) < T (H_{0})\\ 0 \quad Sigtest_{score}(\alpha ) \ge T (H_{1}) \end{array}\right. } \end{aligned}$$

(16)

Consider $A(\alpha , T)$ for two different scenarios: (1) $A_{0}(\alpha , T)$, where ecdf of the data is related to a single cluster and Dcdf is a single Gaussian as well, (2) $A_{1}(\alpha , T)$, where ecdf of the data is relate to two overlapped clusters and Dcdf is a single Gaussian. A proper combination of $\alpha$ and T should suggest to split the two clusters, and should not split the single cluster.

The following minimization problem can be used to estimate the proper $\alpha$ and T:

$$\begin{aligned}{}[\hat{\alpha }, \hat{T}] = \min _{\alpha , T}[A_{1}(\alpha , T) - A_{0}(\alpha , T)] \end{aligned}$$

(17)

where $0<T\le 1$ and $0<\alpha \le \sqrt{\frac{1}{1-p_{c}}}$ [i.e., $p_{c} = 0.99$ in (6)] are possible constraints for the minimization problem. To solve the (17), we have employed genetic algorithm (GA) with a population size of 100. Figure 8 shows the result of GA simulations for estimating $\alpha$ and T for different distances between the center of clusters. Consequently, to be in a safe range for the optimum behavior of Sigtest, we set the averaged values of 1.72 and 0.53 for $\alpha$ and T, respectively. In the case of Gaussian mixture models, the T value will be adaptively calculated based on the assumed mixing factor $\pi _{j}$ in (15). As a result, the T value in a mixture model will be give as follows:

$$\begin{aligned} T = 0.53 \max _{j}{\pi _{j}} \end{aligned}$$

(18)