Deep soft clustering: simultaneous deep embedding and soft-partition clustering

Abstract

Traditional clustering methods are not very effective when dealing with high-dimensional and huge datasets. Even if there are some traditional dimensionality reduction methods such as Principal components analysis (PCA), Linear discriminant analysis (LDA) and T-distributed stochastic neighbor embedding (T-SNE), they still can not significantly improve the effect of the clustering algorithm in this scenario. Recent studies have combined Non-linear dimensionality reduction achieved by deep neural networks with hard-partition clustering, and have achieved reliability results, but these methods can not update the parameters of dimensionality reduction and clustering at the same time. We found that soft-partition clustering can be well combined with deep embedding, and the membership of Fuzzy c-means (FCM) can solve the problem that gradient descent can not be implemented because the assignment process of the hard-partition clustering algorithm is discrete, so that the algorithm can update the parameters of deep neural network (DNN) and cluster centroids at the same time. We build an continuous objective function that combine the soft-partition clustering with deep embedding, so that the learning representations can be cluster-friendly. The experimental results show that our proposed method of simultaneously optimizing the parameters of deep dimensionality reduction and clustering is better than the method with separate optimization.

Appendices

Appendix A: Evaluation measures

We used the standard unsupervised measures to evaluate the performance of clustering algorithm: the clustering accuracy (ACC). For all algorithms we set the number of ground-truth categories as the number of clusters, then evaluate performance with ACC:

$$ ACC(C,S) = \mathop {\max }\limits_{\psi } \frac{1}{N}\sum\limits_{i = 1}^{N} {\chi \left\{ {s_{i} = \psi (c_{i} )} \right\}} $$

where $s_{i}$ is the groun-truth classes,$c_{i}$ is the obtained cluster, Where $\chi$ is the indicator function: $\chi \{ true\} = 1$ and $\chi \{ false\} = 0$;$\psi$ ranges over all possible one-to-one mapping between the obtained clusters and groud-truth class and this match is based on the Hungarian algorithm (Kuhn 1955).

In addition, we used Normalized Mutual Information (NMI) which is an information-thoretic measure that based on the mutual information:

$$ NMI(\Omega ,C) = \frac{I(\Omega ;C)}{{(H(\Omega ) + H(C))/2}} $$

where $I$ represents mutual information, $H$ is entropy

$$ \begin{gathered} I(\Omega ,C) = \sum\limits_{k} {\sum\limits_{j} {P(w_{k} \cap c_{j} )\log \frac{{P(w_{k} \cap c_{j} )}}{{P(w_{k} )P(c_{j} )}}} } \hfill \\ =\sum\limits_{k} {\sum\limits_{j} {\frac{{\left| {w_{k} \cap c_{j} } \right|}}{N}\log \frac{{N\left| {w_{k} \cap c_{j} } \right|}}{{\left| {w_{k} } \right|\left| {c_{j} } \right|}}} } \hfill \\ \end{gathered} $$

where $P(w_{k} ),P(c_{j} ),P(w_{k} \cap c_{j} )$ can be regarded as the probability that the data instance belongs to the cluster $w_{k}$, belongs to the ground-truth class $c_{j}$, and belongs to both at the same time. The second equivalent formula is derived from the maximum likelihood estimate of the probability.

$$ \begin{gathered} H(\Omega ) = - \sum\limits_{k} {P(w_{k} )\log P(w_{k} )} \hfill \\ = - \sum\limits_{k} {\frac{{\left| {w_{k} } \right|}}{N}\log \frac{{\left| {w_{k} } \right|}}{N}} \hfill \\ \end{gathered} $$

Appendix B: The boundary significance of the weighted index $m$

For the FCM algorithm where m belongs to the interval $[1,\infty )$, there are the following situations:

When $m = 1$, FCM algorithm degenerates into KM algorithm.

When $m \to 1^{ + }$, FCM degenerates to KM algorithm with probability 1.

When $m \to + \infty$, FCM loses its division characteristics, and $U = [\mu_{ik} ] = [\frac{1}{c}]$.

Proof

• 1. $m \to 1^{ + }$ $\because d_{ik} = \left\| {x_{k} - v_{i} } \right\|_{A}^{2} \, \therefore {\text{d}}_{k} \ge 0$

When d = 0, we can get.

$$ \mu_{ik} = \left\{ {\begin{array}{*{20}c} {1,{\text{ d}}_{ik} = 0 = \min_{j} \left\{ {d_{jk} } \right\}} \\ {0,{\text{ other}}s} \\ \end{array} } \right.;{ 1} \le i \le c, \, 1 \le k \le n $$

when $\forall d_{ik} \ne 0;1 \le i \le c,$ let $d^{(k)}_{\min } = \mathop {\min }\limits_{1 \le j \le c} \left\{ {d_{jk} } \right\} \ne 0,$ we can get

$$ \mu_{ik} = \frac{{\left( {\frac{1}{{d_{jk} }}} \right)^{{\frac{1}{m - 1}}} }}{{\sum\limits_{p = 1}^{c} {\left( {\frac{1}{{d_{pk} }}} \right)^{{\frac{1}{m - 1}}} } }} = \frac{{\left( {\frac{{d^{(k)}_{\min } }}{{d_{ik} }}} \right)^{{\frac{1}{m - 1}}} }}{{\sum\limits_{p = 1}^{c} {\left( {\frac{{d^{(k)}_{\min } }}{{d_{pk} }}} \right)^{{\frac{1}{m - 1}}} } }} $$

$$ \because d_{\min }^{(k)} = \mathop {\min }\limits_{1 \le j \le c} \left\{ {d_{jk} } \right\}\therefore 0 < \frac{{d^{(k)}_{\min } }}{{d_{ik} }} \le 1 $$

When $d_{ik} = d_{\min }^{(k)}$,$\frac{{d_{\min }^{(k)} }}{{d_{ik} }} = 1$,we can get

$$ \mathop {\lim }\limits_{{m\mathop{\longrightarrow}\limits^{ + }1}} \left( {\frac{{d_{\min }^{(k)} }}{{d_{ik} }}} \right)^{{\frac{1}{m - 1}}} = \mathop {\lim }\limits_{{m\mathop{\longrightarrow}\limits^{ + }1}} 1^{{\frac{1}{m - 1}}} = 1^{ + \infty } = 1 $$

When $d_{ik} \ne d_{\min }^{(k)}$,$\frac{{d_{\min }^{(k)} }}{{d_{ik} }} < 1$,we can get

$$ \mathop {\lim }\limits_{{m\mathop{\longrightarrow}\limits^{ + }1}} \left( {\frac{{d_{\min }^{(k)} }}{{d_{ik} }}} \right)^{{\frac{1}{m - 1}}} = \left( {\frac{{d_{\min }^{(k)} }}{{d_{ik} }}} \right)^{ + \infty } = 0 $$

Obviously, when $m = 1$, the FCM algorithm degenerates from softening to hardening, which is equivalent to K-means.

• 2. $m \to + \infty$ $\because d_{ik} = \left\| {x_{k} - v_{i} } \right\|_{A}^{2} \, \therefore {\text{d}}_{k} \ge 0$

We do not consider the special case of large $\exists d_{ik} = 0$,so $\forall d_{ik} \ne 0;1 \le i \le c$,let $d_{\max }^{(k)} = \mathop {\max }\limits_{1 \le j \le c} \left\{ {d_{jk} } \right\} \ne 0$,we can get

$$ \mu_{ik} = \frac{{\left( {\frac{1}{{d_{jk} }}} \right)^{{\frac{1}{m - 1}}} }}{{\sum\limits_{p = 1}^{c} {\left( {\frac{1}{{d_{pk} }}} \right)^{{\frac{1}{m - 1}}} } }} = \frac{{\left( {\frac{{d^{(k)}_{\max } }}{{d_{ik} }}} \right)^{{\frac{1}{m - 1}}} }}{{\sum\limits_{p = 1}^{c} {\left( {\frac{{d^{(k)}_{\max } }}{{d_{pk} }}} \right)^{{\frac{1}{m - 1}}} } }} $$

$\because d_{\max }^{(k)} = \mathop {\max }\limits_{1 \le j \le c} \left\{ {d_{jk} } \right\},1 \le \frac{{d^{(k)}_{\max } }}{{d_{ik} }} < + \infty$,

$$ \therefore \mathop {\lim }\limits_{m \to + \infty } \left( {\frac{{d^{(k)}_{\max } }}{{d_{ik} }}} \right)^{{\frac{1}{m - 1}}} = \left( {\frac{{d^{(k)}_{\max } }}{{d_{ik} }}} \right)^{ + 0} = 1 $$

$$ \therefore \mathop {\lim }\limits_{m \to + \infty } \sum\limits_{p = 1}^{c} {\left( {\frac{{d^{(k)}_{\max } }}{{d_{pk} }}} \right)^{{\frac{1}{m - 1}}} = \sum\limits_{p = 1}^{c} {\mathop {\lim }\limits_{m \to + \infty } } } \left( {\frac{{d^{(k)}_{\max } }}{{d_{ik} }}} \right)^{{\frac{1}{m - 1}}} = \sum\limits_{p = 1}^{c} 1 = c $$

So, we can get

$$ \begin{gathered} \mathop {\lim }\limits_{m \to + \infty } \mu_{ik} = \mathop {\lim }\limits_{m \to + \infty } \frac{{\left( {\frac{1}{{d_{jk} }}} \right)^{{\frac{1}{m - 1}}} }}{{\sum\limits_{p = 1}^{c} {\left( {\frac{1}{{d_{pk} }}} \right)^{{\frac{1}{m - 1}}} } }} = \frac{{\left( {\frac{{d^{(k)}_{\min } }}{{d_{ik} }}} \right)^{{\frac{1}{m - 1}}} }}{{\sum\limits_{p = 1}^{c} {\left( {\frac{{d^{(k)}_{\min } }}{{d_{pk} }}} \right)^{{\frac{1}{m - 1}}} } }} \hfill \\ \, = \frac{{\mathop {\lim }\limits_{m \to + \infty } \left( {\frac{{d^{(k)}_{\max } }}{{d_{ik} }}} \right)^{{\frac{1}{m - 1}}} }}{{\mathop {\lim }\limits_{m \to + \infty } \sum\limits_{p = 1}^{c} {\left( {\frac{{d^{(k)}_{\max } }}{{d_{pk} }}} \right)^{{\frac{1}{m - 1}}} } }}=\frac{{1}}{c};1 \le i \le c,1 \le k \le n \hfill \\ \end{gathered} $$

Obviously, when $m \to + \infty$, the membership value in the FCM algorithm is all $\frac{1}{c}$, The probability that the data instance belongs to each cluster is equal, and the algorithm will not work.