Supervised Distance Preserving Projections

Abstract

In this work, we consider dimensionality reduction in supervised settings and, specifically, we focus on regression problems. A novel algorithm, the supervised distance preserving projection (SDPP), is proposed. The SDPP minimizes the difference between pairwise distances among projected input covariates and distances among responses locally. This minimization of distance differences leads to the effect that the local geometrical structure of the low-dimensional subspace retrieved by the SDPP mimics that of the response space. This, not only facilitates an efficient regressor design but it also uncovers useful information for visualization. The SDPP achieves this goal by learning a linear parametric mapping and, thus, it can easily handle out-of-sample data points. For nonlinear data, a kernelized version of the SDPP is also derived. In addition, an intuitive extension of the SDPP is proposed to deal with classification problems. The experimental evaluation on both synthetic and real-world data sets demonstrates the effectiveness of the SDPP, showing that it performs comparably or superiorly to state-of-the-art approaches.

Appendix A: Derivation of a Compact form for \({\nabla _\mathbf{W }J}\)

We want to rewrite the gradient ${\nabla _\mathbf{W }J}= \frac{4}{n} \sum _{ij} \mathbf G _{ij}(\mathbf D _{ij}-{\varvec{\varDelta }}_{ij}) {\varvec{\tau }}_{ij} {\varvec{\tau }}_{ij}^T \mathbf W$ into a more compact form. Firstly, we denote $\mathbf Q =\mathbf G \odot (\mathbf D -{\varvec{\varDelta }})$, where $\odot$ represents the element-wise product of two matrices, symmetric matrix $\mathbf R = \mathbf Q + \mathbf Q ^T$, and $\mathbf S$ is a diagonal matrix with $\mathbf S _{ii} = \sum _{j}\mathbf R _{ij}$. Then, we manipulate ${\nabla _\mathbf{W }J}$ as follows:

$$\begin{aligned} {\nabla _\mathbf{W }J}&= \frac{4}{n} \sum _{ij} \mathbf Q _{ij} (\mathbf{x }_i -\mathbf{x }_j)(\mathbf{x }_i-\mathbf{x }_j)^T \mathbf W \nonumber \\&= \frac{4}{n} \sum _{ij} (\mathbf{x }_i \mathbf Q _{ij}\mathbf{x }_i^T + \mathbf{x }_j \mathbf Q _{ij}\mathbf{x }_j^T - \mathbf{x }_i \mathbf Q _{ij}\mathbf{x }_j^T -\mathbf{x }_j \mathbf Q _{ij}\mathbf{x }_i^T) \mathbf W \nonumber \\&= \frac{4}{n} \left[ \sum _{ij}\mathbf{x }_i(\mathbf Q _{ij} + \mathbf Q _{ji})\mathbf{x }_i^T - \sum _{ij}\mathbf{x }_i(\mathbf Q _{ij} + \mathbf Q _{ji})\mathbf{x }_j^T \right] \mathbf W \nonumber \\&= \frac{4}{n} \left( \sum _{ij}\mathbf{x }_i\mathbf R _{ij}\mathbf{x }_i^T - \sum _{ij}\mathbf{x }_i\mathbf R _{ij}\mathbf{x }_j^T \right) \mathbf W \nonumber \\&= \frac{4}{n} \left( \sum _{i}\mathbf{x }_i \sum _{j}\mathbf R _{ij}\mathbf{x }_j^T - \sum _{ij}\mathbf{x }_i\mathbf R _{ij}\mathbf{x }_j^T \right) \mathbf W \nonumber \\&= \frac{4}{n} \left( \sum _{i}\mathbf{x }_i \mathbf S _{ii} \mathbf{x }_j^T - \sum _{ij}\mathbf{x }_i\mathbf R _{ij}\mathbf{x }_j^T \right) \mathbf W \nonumber \\&= \frac{4}{n} \left( \mathbf X ^T\mathbf S \mathbf X - \mathbf X ^T \mathbf R \mathbf X \right) \mathbf W \nonumber \\&= \frac{4}{n} \mathbf X ^T(\mathbf S - \mathbf R ) \mathbf X \mathbf W , \nonumber \end{aligned}$$

where each row of data matrix $\mathbf X$ is a data point $\mathbf{x }_i$ and $\mathbf L = \mathbf S - \mathbf R$ is the Laplacian matrix.