Fast t-SNE algorithm with forest of balanced LSH trees and hybrid computation of repulsive forces

Abstract

An acceleration of the well-known t-Stochastic Neighbor Embedding (t-SNE) (Hinton and Roweis, 2003; Maaten and Hinton, 2008) algorithm, probably the best (nonlinear) dimensionality reduction and visualization method, is proposed in this article.

By using a specially-tuned forest of balanced trees constructed via locality sensitive hashing is improved significantly upon the results presented in Maaten (2014), achieving a complexity significantly closer to true $O(nlogn)$, and vastly improving behavior for huge numbers of instances and attributes. Such acceleration removes the necessity to use PCA to reduce dimensionality before the start of t-SNE.

Additionally, a fast hybrid method for repulsive forces computation (a part of the t-SNE algorithm), which is currently the fastest method known, is proposed.

A parallelized version of our algorithm, characterized by a very good speedup factor, is proposed.

Introduction

Data visualization is an important task in machine learning, as people prefer visual representations of data over numerical ones.

We can enumerate simple (in terms of the underlying mathematics) methods like scatter plots, parallel coordinates, histograms, heat maps, survey plots, dimensional stacking, rad viz, etc. [1], [2], but such methods can be useful for detecting only some (simple) regularities in data.

One of the most universally known methods of dimensionality reduction is Principal Component Analysis (PCA) [3]. PCA can be used to select directions with the highest variances in the given data. PCA results in a linear and orthogonal transformation. However, such a transformation from multidimensional space to 2D or 3D will rarely produce readable shapes, because a strong reduction cannot preserve all the initial distances from a high-dimensional space. An advantage of PCA is its $O\left(m{n}^2\right)$ complexity, where $m$ is the number of instances and the $n$ is the number of attributes. The well-known kernel trick [4] changes the linear PCA into kernel nonlinear PCA. Schölkopf [5] has proposed this idea.

Multidimensional scaling (MDS) was one of the first very practical nonlinear dimensionality reduction methods [6]. The goal of MDS is to preserve distances from a high-dimensional space in a low-dimensional space using the below definition of a cost function:

$$C_{MDS}=\frac{1}{\sum _{i<j}{\Vert {\mathbf{x}}_i-{\mathbf{x}}_j\Vert }^2}\sum _{i<j}^m{(\Vert {\mathbf{x}}_i-{\mathbf{x}}_j\Vert -\Vert {\mathbf{y}}_i-{\mathbf{y}}_j\Vert )}^2.$$

Let us assume we have dataset $\mathcal{D}$ which consists of vectors ${\mathbf{x}}_1,\dots ,{\mathbf{x}}_m$ in ${\mathcal{R}}^n$. ${\mathbf{y}}_i$ are the points corresponding, respectively, to ${\mathbf{x}}_i$ in a low-dimensional space. Typically, PCA is used to initialize the positions of the low-dimensional points ${\mathbf{y}}_i$. The complexity of MDS is $O(m{n}^2+m^{2}n+m^{2}l)$ and is composed of PCA, a single computation of all distances in high dimensional space and the main loop of MDS.¹

The Sammon mapping [7] can be seen as a variant of MDS. The goal of Sammon mapping is similar to the MDS.

A neural approach called Self-Organized Maps (SOM) was proposed by Kohonen [8]. SOM creates a map/grid from the original data. Another nonlinear and neural approach was the Neuronal Gas [9], [10].

The IsoMap [11] algorithm is based on the idea of the geodesic distance. A closely related graph-based approach was proposed in [12]. Both of the above methods are locally oriented, as is another, the Local Linear Embedding [13].

Stochastic Neighbor Embedding (SNE) was proposed by Hinton and Roweis [14]. SNE and its variants are the main subjects of this article. In [15], the authors described two important improvements. Another type of acceleration in the domain of document analysis was proposed in [16].

One of the newer approaches in dimensionality reduction is LargeVis proposed by Tang et al. [17]. This algorithm is based on the construction of an approximated K-nearest neighbor graph. However, while this algorithm is faster than tree-based t-SNE (around two times), it is significantly slower than our algorithm because LargerVis is also slower than UMAP.

Another new approach in dimensionality reduction is UMAP [18], which also uses cross-entropy as t-SNE. The speed of this algorithm is compared with our results in Section 4.

For a review of dimensionality reduction methods please see [19], where several algorithms are presented, their short descriptions and additional remarks and comparison. In the review, you can find that most of algorithms are strongly time-consuming.

The next section describes the Stochastic Neighbor Embedding and its improvements in more detail. The following section describes several accelerations of t-SNE which exhibit better complexity. Section 4 presents several testing procedures and an analysis of the obtained results, which show the superiority of the proposed methods.