Deep embedded self-organizing maps for joint representation learning and topology-preserving clustering

Abstract

A recent research area in unsupervised learning is the combination of representation learning with deep neural networks and data clustering. The success of deep learning for supervised tasks is widely established. However, recent research has demonstrated how neural networks are able to learn representations to improve clustering in their intermediate feature space, using specific regularizations. By considering representation learning and clustering as a joint task, models learn clustering-friendly spaces and outperform two-stage approaches where dimensionality reduction and clustering are performed separately. Recently, this idea has been extended to topology-preserving clustering models, known as self-organizing maps (SOM). This work is a thorough study on the deep embedded self-organizing map (DESOM), a model composed of an autoencoder and a SOM layer, training jointly the code vectors and network weights to learn SOM-friendly representations. In other words, SOM induces a form a regularization to improve the quality of quantization and topology in latent space. After detailing the architecture, loss and training algorithm, we study hyperparameters with a series of experiments. Different SOM-based models are evaluated in terms of clustering, visualization and classification on benchmark datasets. We study benefits and trade-offs of joint representation learning and self-organization. DESOM achieves competitive results, requires no pretraining and produces topologically organized visualizations.

Appendices

Appendix 1: performance evaluation

Quantitative evaluation of self-organizing maps is not as straightforward as for supervised classification tasks. To assess and compare the performance of models, we implemented and evaluated a collection of metrics that have been developed in related literature. This literature spans almost 30 years of history, and implementations are not easy to find. Thus, we implemented these metrics and provide them as an open-source library, SOMperf⁶ [18]. First, SOM performance metrics can be categorized into two families:

• 1. Clustering metrics. Any clustering quality measure that relies solely on the prototype vectors and not on their topological organization. This encompasses all quality indices used in clustering literature (e.g., purity, normalized mutual information (NMI), Rand index, etc.).

• 2. Topographic metrics. Under this term, we coin quality measures that, on the contrary, assess the topological organization of the model. Some indices also evaluate the clustering quality, but we call a metrics topographic as soon as it incorporates the map topology.

On another level, we can also classify them into two well-known families, depending on the use or not of ground-truth label knowledge:

• 1. Internal indices, using only intrinsic properties of the model and the data.

• 2. External indices, relying on external ground-truth class labels to evaluate results, as in supervised classification.

For instance, quantization error falls into the clustering metric category (as it measures how the SOM cluster centers fit the data distribution, without using any topology information) and is an internal indices (not depending on external labels). On the other side, the Class Scatter Index (introduced in [10]) is a topographic metric and an external indices, as it measures how ground-truth class labels are organized into groups of neighboring map units.

1.1 Clustering metrics

1.1.1 Internal indices

Quantization error Quantization error is the average error made by projecting data on the SOM, as measured by Euclidean distance, i.e., the mean Euclidean distance between a data sample and its best-matching unit. It can be measured in the original space, using the prototypes reconstructed by the decoder ($\text {QE}$), or in latent space ($\hat{\text {QE}}$), introducing the notations ${\tilde{b}}_i$ for the best-matching unit of data point ${\mathbf {x}}_i$ in original space, and $b_i$ for the best-matching unit of ${\mathbf {z}}_i$ in latent space.

$$\begin{aligned}&\text {QE}(\{\tilde{{\mathbf {m}}}_k\}, {\mathbb {X}}) = \frac{1}{N} \sum _{i=1}^N ||{\mathbf {x}}_i - \tilde{{\mathbf {m}}}_{b_i}||_2\\&\hat{\text {QE}}(\{{\mathbf {m}}_k\}, {\mathbb {Z}}) = \frac{1}{N} \sum _{i=1}^N ||{\mathbf {z}}_i - {\mathbf {m}}_{b_i}||_2 \end{aligned}$$

Prototype sharpness ratio In the case of images, we want the prototype images to be realistic and as sharp as the original images. When visualizing a self-organized map of an image dataset, we usually simply visualize the image corresponding to each prototype vector. If they are blurry (due to the averaging induced by the SOM algorithm), the visualization will be of poor quality, even if the quantization error is low (because QE is only an average Euclidean distance). We have chosen a very simplistic way to compute the sharpness of an image, which is the average norm of pixel gradients, measured in two dimensions. The sharpness of a SOM is defined as the average sharpness of its prototypes. This sharpness measure can then be compared with the average sharpness of images in the original dataset. We introduce the prototype sharpness ratio (PSR), defined as follows:

$$\begin{aligned} \text {PSR}(\{\tilde{{\mathbf {m}}}_k\}, {\mathbb {X}}) = \frac{\text {average prototype sharpness}}{\text {average dataset sharpness}} = \frac{\frac{1}{K} \sum _{k=1}^K ||\nabla _{\text {2D}}\tilde{{\mathbf {m}}}_k||_2^2}{\frac{1}{N} \sum _{i=1}^N ||\nabla _{\text {2D}}{\mathbf {x}}_i||_2^2} \end{aligned}$$

A score lower than 1 means that the prototypes are in average blurrier than the original images; on the contrary, if it is larger than 1, they are less blurry (i.e., more crisp or noisy) than the originals. The closer the PSR is to 1, the better is the score.

1.1.2 External indices

A clustering with K clusters is described by the sets of data points belonging to each cluster, noted ${\mathbf {Q}} = \{Q_k\}, k = 1 \ldots K$. In order to define the external clustering criteria, we assume that labels are associated with each data point, corresponding to a set of C different classes. We note ${\mathbf {Y}} = \{Y_j\}, j = 1 \ldots C$ the sets of elements belonging to each class.

Purity Purity is one of the most commonly used external quality indices. It measures the purity of clusters with respect to ground-truth class labels. To compute the purity of a clustering ${\mathbf {Q}}$, each cluster is assigned to the class which is most frequent in the cluster, and then the accuracy of this assignment is measured by counting the number of correctly assigned points and dividing by the total number of points. Formally:

$$\begin{aligned} \text {Pur}({\mathbf {Q}}, {\mathbf {Y}}) = \frac{1}{N} \sum _{k=1}^K \underset{j = 1 \ldots C}{\max } |Q_k \cap Y_j| \end{aligned}$$

(9)

High purity is easy to achieve when the number of data points per cluster is small; in particular, purity is equal to 1 if all points get their own cluster. Thus, purity cannot be used to trade off the validity of the clustering against the number of clusters.

Normalized mutual information Normalized mutual information (NMI) is also one of the most widespread external clustering indices:

$$\begin{aligned} \text {NMI}({\mathbf {Q}}, {\mathbf {Y}}) = \frac{I({\mathbf {Q}}, {\mathbf {Y}})}{\frac{1}{2}\left( H({\mathbf {Q}}) + H({\mathbf {Y}})\right) } \end{aligned}$$

where I is mutual information:

$$\begin{aligned} I({\mathbf {Q}}, {\mathbf {Y}}) = \sum _k \sum _j P(Q_k \cap Y_j) \log \frac{P(Q_k \cap Y_j)}{P(Q_k)P(Y_j)} = \sum _k \sum _j \frac{|Q_k \cap Y_j|}{N} \log \frac{N|Q_k \cap Y_j|}{|Q_k||Y_j|} \end{aligned}$$

and H is entropy:

$$\begin{aligned} H({\mathbf {Q}}) = -\sum _k P(Q_k) \log P(Q_k) = -\sum _k \frac{|Q_k|}{N} \log \frac{|Q_k|}{N} \end{aligned}$$

Unsupervised clustering accuracy Accuracy is the most common metric used in supervised classification: it corresponds to the number of examples that are assigned to the correct class divided by the total number of examples in the dataset. It can also be used in clustering (i.e., unsupervised classification) as an external quality measure if labels are available and if the number of clusters is equal to the number of classes. It consists in the accuracy of the resulting classification using the best one-to-one mapping between clusters and class labels. Denoting this mapping by m, the expression of unsupervised clustering accuracy is:

$$\begin{aligned} \text {Acc}({\mathbf {Q}}, {\mathbf {Y}}) = \frac{1}{N} \underset{m}{\max }\sum _{k=1}^K |Q_k \cap Y_{m(k)}| \end{aligned}$$

(10)

The best mapping can be found using the Hungarian assignment algorithm, also known as the Kuhn–Munkres algorithm.

1.2 Topographic metrics

Distortion Distortion is the loss function minimized by the SOM learning algorithm. It is similar to the within-cluster sum of squared errors (WSSE) minimized by k-means, but with an additional topology constraint introduced by the neighborhood function. It is calculated by the sum of squared Euclidean distances between each map prototype and data sample, weighted by the neighborhood function according to the distance between the sample and the prototype. As quantization error, it can be measured in the original data space or in latent space:

$$\begin{aligned}&\text {D}(\{\tilde{{\mathbf {m}}}_k\}, {\mathbb {X}}, T) = \frac{1}{N} \sum _{i=1}^N \sum _{k=1}^K {\mathcal {K}}^T\left( \delta ({\tilde{b}}_i, k)\right) ||{\mathbf {x}}_i - \tilde{{\mathbf {m}}}_k||_2^2\\&\hat{\text {D}}(\{{\mathbf {m}}_k\}, {\mathbb {Z}}, T) = \frac{1}{N} \sum _{i=1}^N \sum _{k=1}^K {\mathcal {K}}^T\left( \delta (b_i, k)\right) ||{\mathbf {z}}_i - {\mathbf {m}}_k||_2^2 \end{aligned}$$

In DESOM, the SOM loss corresponds to latent distortion.

Topographic error Topographic error assesses the self-organization of a SOM model. It is calculated as the fraction of samples whose best and second-best matching units are not neighbors on the map. In other words, this error quantifies the smoothness of projections on the self-organized map. Using the notations ${\tilde{b}}^k_i$ and $b^k_i$ for the k-th best-matching units of ${\mathbf {x}}_i$ and ${\mathbf {z}}_i$ in original and latent space, respectively, we define topographic error $\text {TE}$ and latent topographic error $\hat{\text {TE}}$:

$$\begin{aligned}&\text {TE}(\{\tilde{{\mathbf {m}}}_k\}, {\mathbb {X}}) = \frac{1}{N} \sum _{i=1}^N \mathbb {1}_{\delta ({\tilde{b}}^1_i, {\tilde{b}}^2_i)> 1}\\&\hat{\text {TE}}(\{{\mathbf {m}}_k\}, {\mathbb {Z}}) = \frac{1}{N} \sum _{i=1}^N \mathbb {1}_{\delta (b^1_i, b^2_i) > 1} \end{aligned}$$

Combined error Combined error [25] is an error measure that combines and extends quantification and topographic errors. Its computation is more complex than the previous indices. For a given data sample ${\mathbf {x}}_i$, we first compute its two best matching units ${\tilde{b}}^1_i$ and ${\tilde{b}}^2_i$. Then, we compute a sum of Euclidean distances from ${\mathbf {x}}_i$ to the second BMU’s prototype vector ${\mathbf {m}}_{{\tilde{b}}^2_i}$, starting with the distance from ${\mathbf {x}}_i$ to ${\mathbf {m}}_{{\tilde{b}}^1_i}$, and thereafter following a shortest path until ${\mathbf {m}}_{{\tilde{b}}^2_i}$, going through only neighboring units on the map. Finally, the combined error (CE) is the average of this distance over the input samples.

Let p be a path on the map of length $P \ge 1$, from $p(0) = {\tilde{b}}^1_i$ to $p(L) = {\tilde{b}}^2_i$, such that p(k) and $p(k+1)$ must be neighbors for $k = 0 \ldots P-1$. The distance along the shortest path on the map is computed as:

$$\begin{aligned} \text {CE}_i = ||{\mathbf {x}}_i - \tilde{{\mathbf {m}}}_{{\tilde{b}}^1_i}||_2^2 + \underset{p}{\min } \sum _{k=0}^{P-1} ||\tilde{{\mathbf {m}}}_{p(k+1)} - \tilde{{\mathbf {m}}}_{p(k)}||_2^2 \end{aligned}$$

Finally, the combined error is:

$$\begin{aligned} \text {CE}(\{\tilde{{\mathbf {m}}}_k\}, {\mathbb {X}}) = \frac{1}{N} \sum _{i=1}^N \text {CE}_i \end{aligned}$$

As usual, we also define combined error in latent space:

$$\begin{aligned}&\hat{\text {CE}}_i = ||{\mathbf {z}}_i - {\mathbf {m}}_{b^1_i}||_2^2 + \underset{p}{\min } \sum _{k=0}^{L-1} ||{\mathbf {m}}_{p(k+1)} - {\mathbf {m}}_{p(k)}||_2^2\\&\hat{\text {CE}}(\{{\mathbf {m}}_k\}, {\mathbb {Z}}) = \frac{1}{N} \sum _{i=1}^N \hat{\text {CE}}_i \end{aligned}$$

Topographic product The topographic product (TP) [3] measures the preservation of neighborhood relations between vector space and the map. It depends only on the prototype vectors and map topology, and is able to indicate whether the dimension of the map is appropriate to fit the dataset, or if it introduced neighborhood violations.

We will note d the Euclidean distance in vector space, and $\delta$ the topographic distance on the map. The computation of TP starts by defining two ratios between the distance of a prototype j to its k-th nearest neighbor on the map $n_k^{\delta }(j)$, and to its k-th nearest neighbor in vector space $n_k^d(j)$:

$$\begin{aligned} Q_1(j, k) = \frac{d\left( \tilde{{\mathbf {m}}}_j, \tilde{{\mathbf {m}}}_{n_k^{\delta }(j)} \right) }{d\left( \tilde{{\mathbf {m}}}_j, \tilde{{\mathbf {m}}}_{n_k^d(j)} \right) }, \; Q_2(j, k) = \frac{\delta \left( j, n_k^{\delta }(j)\right) }{\delta \left( j, n_k^d(j)\right) } \end{aligned}$$

Naturally, we always have $Q_1 \ge 1$ and $Q_2 \le 1$. The ratios are combined into a product in order to obtain a symmetric measure and mitigate local magnification factors:

$$\begin{aligned} P_3(j, k) = \left[ \prod _{l=1}^k Q_1(j, l) Q_2(j, l) \right] ^{\frac{1}{2k}} \end{aligned}$$

Finally, the topographic product is obtained by taking the logarithm and averaging over all map units and neighborhood orders:

$$\begin{aligned} \text {TP} = \frac{1}{K(K-1)}\sum _{j=1}^K \sum _{k=1}^{K-1} \log P_3(j, k) \end{aligned}$$

$\text {TP} < 0$ indicates the map dimension is too low to correctly represent the dataset; $\text {TP} = 0$ means the dimension is adequate; and $\text {TP} > 0$ indicates a dimension too high and neighborhood violations. A latent topographic product $\hat{\text {TP}}$ can be defined in exactly the same manner, only replacing the prototypes $\tilde{{\mathbf {m}}}_j$ by their latent counterparts ${\mathbf {m}}_j$.

Appendix 2: map visualizations

See Fig. 17.

Fig. 17

20-by-20 DESOM maps of MNIST and Fashion-MNIST