Alternating Optimization of Unsupervised Regression with Evolutionary Embeddings

Abstract

Unsupervised regression is a dimensionality reduction method that allows embedding high-dimensional patterns in low-dimensional latent spaces. In the line of research on iterative unsupervised regression, numerous methodological variants have been proposed in the recent past. This works extends the set of methods by evolutionary embeddings. We propose to use a $(1+\lambda )$-ES with Rechenberg mutation strength control to iteratively embed patterns and show that the learned manifolds are better with regard to the data space reconstruction error than the embeddings generated with naive Gaussian sampling. Further, we introduce a hybrid optimization approach of alternating gradient descent and the iterative evolutionary embeddings. Experimental comparisons on artificial test data sets confirm the expectation that a hybrid approach is superior or at least competitive to known methods like principal component analysis or Hessian local linear embedding.

A Gradient Descent

HybUKR requires the gradient of the Nadaraya-Watson estimator, which is defined as:

$$\begin{aligned} \frac{\partial f(\mathbf {X})}{\partial x_{mn}}= & {} \sum _{i=1}^N \mathbf {y}_i \cdot \left( \frac{ 1 }{ \sum _{j=1}^N \mathbf {K}_h(\mathbf {x} - \mathbf {x}_j) } \cdot \frac{\partial \mathbf {K}_h(\mathbf {x} - \mathbf {x}_i)}{\partial x_{mn}} \right. \\&\left. - \frac{\mathbf {K}(\mathbf {x} - \mathbf {x}_i)}{\left( \sum _{j=1}^N \mathbf {K}_h(\mathbf {x} - \mathbf {x}_j) \right) ^2 } \cdot \sum _{j=1}^N \frac{\partial \mathbf {K}_h(\mathbf {x} - \mathbf {x}_j)}{\partial x_{mn}} \right) \end{aligned}$$

The derivative of the Gaussian kernel function is:

$$\begin{aligned} \mathbf {K}_h( \mathbf {x}_i - \mathbf {x}_j ) = K_h( \left\| \mathbf {x}_i - \mathbf {x}_j \right\| _2^2 ) = \frac{1}{h} \cdot K \left( \frac{\left\| \mathbf {x}_i - \mathbf {x}_j \right\| _2^2}{h} \right) \end{aligned}$$