Local tangent space alignment based on Hilbert–Schmidt independence criterion regularization

Abstract

Local tangent space alignment (LTSA) is a famous manifold learning algorithm, and many other manifold learning algorithms are developed based on LTSA. However, from the viewpoint of dimensionality reduction, LTSA is only a local feature preserving algorithm. What the community of dimensionality reduction is now pursuing are those algorithms capable of preserving both local and global features at the same time. In this paper, a new algorithm for dimensionality reduction, called HSIC-regularized LTSA (HSIC–LTSA), is proposed, in which a HSIC regularization term is added to the objective function of LTSA. HSIC is an acronym for Hilbert–Schmidt independence criterion and has been used in many applications of machine learning. However, HSIC has not been directly applied to dimensionality reduction so far, neither used as a regularization term to combine with other machine learning algorithms. Therefore, the proposed HSIC–LTSA is a new try for both HSIC and LTSA. In HSIC–LTSA, HSIC makes the high- and low-dimensional data statistically correlative as much as possible, while LTSA reduces the data dimension under the local homeomorphism-preserving criterion. The experimental results presented in this paper show that, on several commonly used datasets, HSIC–LTSA performs better than LTSA as well as some state-of-the-art local and global preserving algorithms.

Appendix: Reproducing kernel Hilbert Spaces (RKHS)

HSIC is based on RKHS. Let ${{{ L}}^{{ 2}}}\left( \varOmega \right) =\left\{ f\left| f:\varOmega \rightarrow R,\int \limits _{\varOmega }{{{\left| f\left( x \right) \right| }^{2}}}<+\infty \right. \right\}$ be the space of square integrable functions. An inner product $\left\langle \bullet ,\bullet \right\rangle$ can be defined over ${{{ L}}^{{ 2}}}\left( \varOmega \right)$ [30]:

$$\begin{aligned} \left\langle f,g \right\rangle =\int \limits _{\varOmega }{f\left( x \right) g\left( x \right) {\mathrm{d}}x} \end{aligned}$$

(52)

It can be proven that $H=\left( {{{ L}}^{{ 2}}}\left( \varOmega \right) ,\left\langle \bullet ,\bullet \right\rangle \right)$ is a complete inner product space, i.e., Hilbert space.

Definition

[30] Let $H=\left( {{{ L}}^{{ 2}}}\left( \varOmega \right) ,\left\langle \bullet ,\bullet \right\rangle \right)$; if there is a function $k:\varOmega \times \varOmega \rightarrow R$ such that

• For all $x\in \varOmega$,${{k}_{x}}=k\left( \bullet ,x \right) \in H$;

• For all $f\in H$, $f\left( x \right) =\left\langle f,k\left( \bullet ,x \right) \right\rangle$

then H is called a reproducing kernel Hilbert space (RKHS) and k called the reproducing kernel of H.

The reproducing kernel k can used to define a map: $\varphi :\varOmega \rightarrow H$ such that for all $x\in \varOmega$,

$$\begin{aligned} \varphi \left( x \right) =k\left( \bullet ,x \right) \in H \end{aligned}$$

(53)

It can be easily proven that

$$\begin{aligned} \left\langle \varphi \left( x \right) ,\varphi \left( y \right) \right\rangle =\left\langle {{k}_{x}},k\left( \bullet ,y \right) \right\rangle ={{k}_{x}}\left( y \right) =k\left( y,x \right) =k\left( x,y \right) \end{aligned}$$

(54)

The above equation is often used in many kernel methods of machine learning such as kPCA [3], kLDA [31], kSVM [32], etc.

Furthermore, if X is a random variable on $\varOmega$, then $\varphi \left( X \right)$ is a random process and its mean function is defined

$$\begin{aligned} {{\mu }_{{ X}}}\left( { u} \right)& = {{{ E}}_{{ X}}}\left[ \varphi \left( { X} \right) \left( { u} \right) \right] = {{{ E}}_{{ X}}}\left[ { k}\left( { u,X} \right) \right] \\& = \int \limits _{\varOmega }{{ k}\left( { u,x} \right) {{p}_{X}}\left( x \right) {\mathrm{d}}x} \end{aligned}$$

(55)

Then, for all $f\in H$,

$$\begin{aligned} \left\langle {{\mu }_{{ X}}},f \right\rangle&= \int \limits _{\varOmega }{{{\mu }_{X}}\left( u \right) { f}\left( { u} \right) {\mathrm{d}}u} \\&=\int \limits _{\varOmega }{\left( \int \limits _{\varOmega }{k\left( u,x \right) {{p}_{X}}\left( x \right) {\mathrm{d}}x} \right) { f}\left( { u} \right) {\mathrm{d}}u} \\&=\int \limits _{\varOmega }{\left( \int \limits _{\varOmega }{k\left( u,x \right) { f}\left( { u} \right) {\mathrm{d}}u} \right) {{p}_{X}}\left( x \right) {\mathrm{d}}x} \\&=\int \limits _{\varOmega }{\left\langle f,k\left( \bullet ,x \right) \right\rangle {{p}_{X}}\left( x \right) {\mathrm{d}}x} \\&=\int \limits _{\varOmega }{f\left( { x} \right) {{p}_{x}}\left( x \right) {\mathrm{d}}u}={{E}_{x}} \left[ { f}\left( X \right) \right] \end{aligned}$$

(56)

In mathematics, it can be proven that RKHS can be generated from kernel functions. The definition of kernel functions is as follows:

Definition

[33] Let $k:\varOmega \times \varOmega \rightarrow R$, if k satisfies the following conditions:

• Symmetric: for all $x,y\in \varOmega$, $k\left( x,y \right) =k\left( y,x \right)$

• Square integrable: for all $x\in \varOmega$, ${{k}_{x}}=k\left( \bullet ,x \right)$ is square integrable

• Positive definite: for all ${{x}_{1}},\ldots ,{{x}_{N}}\in \varOmega$, the matrix $\left[ \begin{array}{lll} k\left( {{x}_{1}},{{x}_{1}} \right) &{} \ldots &{} k\left( {{x}_{1}},{{x}_{N}} \right) \\ \vdots &{} \ddots &{} \vdots \\ k\left( {{x}_{N}},{{x}_{1}} \right) &{} \ldots &{} k\left( {{x}_{N}},{{x}_{N}} \right) \\ \end{array} \right]$ is positive definite then k is called a kernel function.

Remark

Kernel functions and reproducing kernels are not the same concept. Kernel functions are defined on their own, while reproducing kernels are defined based on RKHS.

Theorem

[30] A kernel function k can be used to generate a unique RHHS${{H}_{k}}$such that k becomes the reproducing kernel of${{H}_{k}}.$

Based on this theorem, as long as a kernel function is determined, a RKHS is also determined too.