Efficient regularized spectral data embedding

Abstract

Data embedding (DE) or dimensionality reduction techniques are particularly well suited to embedding high-dimensional data into a space that in most cases will have just two dimensions. Low-dimensional space, in which data samples (data points) can more easily be visualized, is also often used for learning methods such as clustering. Sometimes, however, DE will identify dimensions that contribute little in terms of the clustering structures that they reveal. In this paper we look at regularized data embedding by clustering, and we propose a simultaneous learning approach for DE and clustering that reinforces the relationships between these two tasks. Our approach is based on a matrix decomposition technique for learning a spectral DE, a cluster membership matrix, and a rotation matrix that closely maps out the continuous spectral embedding, in order to obtain a good clustering solution. We compare our approach with some traditional clustering methods and perform numerical experiments on a collection of benchmark datasets to demonstrate its potential.

Appendix A: Proof of Theorem 1

The solution of (5) comes from the singular value decomposition (SVD) of $\mathbf {X}^\top \mathbf {B}$. Let $\mathbf {U}\mathbf {D}\mathbf {V}^\top$ be the SVD for $\mathbf {X}^\top \mathbf {B}$, then $\mathbf {Q}_{*} = \mathbf {U}\mathbf {V}^\top$.

Proof

We expand the matrix norm

$$\begin{aligned} \left\| \mathbf {X}- \mathbf {B}\mathbf {Q}^\top \right\| ^{2} = Tr(\mathbf {X}^\top \mathbf {X}) - 2 Tr(\mathbf {X}^\top \mathbf {B}\mathbf {Q}^\top )+ Tr(\mathbf {Q}\mathbf {B}^\top \mathbf {B}\mathbf {Q}^\top ). \end{aligned}$$

(16)

Since $\mathbf {Q}^\top \mathbf {Q}= \mathbf {I}$, the last term is equal to $Tr(\mathbf {B}^\top \mathbf {B})$ and hence the original minimization problem (4) is equivalent to the maximization of the middle term, i.e (5).

With the SVD of $\mathbf {X}^\top \mathbf {B}$ = $\mathbf {U}\mathbf {D}\mathbf {V}^\top$, the middle term becomes

$$\begin{aligned} Tr(\mathbf {X}^\top \mathbf {B}\mathbf {Q}^\top )= & {} Tr(\mathbf {U}\mathbf {D}\mathbf {V}^\top \mathbf {Q}^\top ) \nonumber \\= & {} Tr(\mathbf {U}\mathbf {D}\hat{\mathbf {Q}}^\top ) \quad \text{ where } \quad \hat{\mathbf {Q}}=\mathbf {Q}\mathbf {V}\nonumber \\= & {} Tr(\hat{\mathbf {Q}}^\top \mathbf {U}\mathbf {D}). \end{aligned}$$

(17)

where $\mathbf {D}=\mathbf {Diag}(d_1,\ldots ,d_k) \in {\mathbb {R}}_{+}^{k \times k}$. In vector form we have $\mathbf {U}=[\mathbf {u}_1|\ldots |\mathbf {u}_k]\in {\mathbb {R}}^{d \times k}$ and $\hat{\mathbf {Q}}=[\varvec{\hat{q}}_1|\ldots |\varvec{\hat{q}}_k] \in {\mathbb {R}}^{d \times k}$. Aplying the Cauchy-Shwartz inequality and since $\mathbf {U}^\top \mathbf {U}=\mathbf {I}$, $\hat{\mathbf {Q}}^\top \hat{\mathbf {Q}} = \mathbf {I}$ due to $\mathbf {V}\mathbf {V}^\top =\mathbf {I}$, we get

$$\begin{aligned} Tr(\hat{\mathbf {Q}}^\top \mathbf {U}\mathbf {D}) \le \sum _{i}d_i ||\mathbf {u}_i||\times ||\varvec{\hat{q}}_i||=\sum _i d_i= Tr(\mathbf {D}). \end{aligned}$$

Then the upper bound is clearly attained by setting $\hat{\mathbf {Q}}= \mathbf {U}$. This leads to $\hat{\mathbf {Q}}=\mathbf {Q}\mathbf {V}= \mathbf {U}$ and $\mathbf {Q}\mathbf {V}\mathbf {V}^\top = \mathbf {U}\mathbf {V}^\top$. Hence we get $\mathbf {Q}_* = \mathbf {U}\mathbf {V}^\top .$

$\square$

Remark 1

Note that the problem in (5) can be considered as a special case of the Orthogonal Procrustes Problem (OPP) (Schonemann, 1966) in which $\mathbf {Q}$ is a square orthogonal rotation matrix (i.e $\mathbf {Q}^\top \mathbf {Q}=\mathbf {Q}\mathbf {Q}^\top = \mathbf {I}$). Hence, the difference between the problem in Theorem 1 and OPP is the constraint.