Robust semi-supervised discriminant embedding method with soft label in kernel space

Abstract

Considering some problems of local linear embedding methods in semi-supervised scenarios, a robust scheme for generating soft labels is designed and a semi-supervised discrimination embedding method combined with soft labels in the kernel space is proposed in this paper. The method uses the kernel trick to perform the following related operations in the regenerative kernel Hilbert space. Firstly, in order to deal with the different distribution of labeled data, the confidence of generating soft labels is introduced in the label propagation process, and then the label density of data within the hypersphere whose unlabeled data is the center of the sphere is used to generate final soft labels. The scheme is experimentally demonstrated to generate more reliable soft labels even with lower labeling ratios. In order to make full use of the prior information provided by soft labels and the confidence of soft labels, a distance distortion function is used in feature learning to introduce the soft label prior information, and a penalty term about the global information is added to the optimization objective function, so that a more suitable low-dimensional representation for data classification can be obtained. Finally, the method is experimentally compared with various feature learning methods and visualized on handwritten digit dataset. The experiments show that this method performs well on various datasets of the UCI database and is successfully applied in image recognition, especially when the ratio of labeled data is in a low range.

Appendix

1.1 Appendix 1: Proof of convergence of Eq. (12)

By mathematical induction, Eq. (12) can be reduced to the following generalized form with initial values.

$$\begin{array}{*{20}c} {\overline{\user2{Y}}\left( t \right) = \overline{\user2{Y}}\left( 0 \right)\left( {{\varvec{FD}}_{\alpha } } \right)^{t} + \overline{\user2{Y}}\left( {{\varvec{I}}_{n} - {\varvec{D}}_{\alpha } } \right)\mathop \sum \limits_{\tau = 0}^{t - 1} \left( {{\varvec{FD}}_{\alpha } } \right)^{\tau } } \\ \end{array}$$

(24)

When $t$ tends to positive infinity, we can get $\underset{t\to \infty }{\mathrm{lim}}{\left({\varvec{F}}{{\varvec{D}}}_{\alpha }\right)}^{t}=0$ because the elements in ${\varvec{F}}$ and ${{\varvec{D}}}_{\alpha }$ are ranging from 0 to 1. Using the summation formula of the infinite series, we can get $\sum_{\tau =0}^{\infty }{\left({\varvec{F}}{{\varvec{D}}}_{\alpha }\right)}^{\tau }={\left({{\varvec{I}}}_{n}-{\varvec{F}}{{\varvec{D}}}_{\alpha }\right)}^{-1}$. Since each term in $\underset{t\to \infty }{\mathrm{lim}}\overline{{\varvec{Y}} }\left(t\right)$ exists a definite real number corresponding to it, the iteration of Eq. (12) is eventually convergent and converges to the definite matrix $\overline{{\varvec{Y}} }\left({{\varvec{I}}}_{n}-{{\varvec{D}}}_{\alpha }\right){\left({{\varvec{I}}}_{n}-{\varvec{F}}{{\varvec{D}}}_{\alpha }\right)}^{-1}$.

1.2 Appendix 2: Proof that the summation of each column of $\overline{{\varvec{Y}} }\left({\varvec{t}}\right)$ in Eq. (13) is constant 1

Assuming that ${{\varvec{e}}}_{n}={\left[\mathrm{1,1},\dots ,1\right]}_{1\times n}$ is an all-ones vector, the problem can be transformed into proving that ${{\varvec{e}}}_{c+1}\overline{{\varvec{Y}} }\left(t\right)={{\varvec{e}}}_{n}$ holds for any $t$. In fact, the previous work has obtained that the following constraints hold, i.e., ${{\varvec{e}}}_{n}{\varvec{F}}={{\varvec{e}}}_{n}, {{\varvec{e}}}_{c+1}\overline{{\varvec{Y}} }={{\varvec{e}}}_{n}$. Using some tricks, the two equations are, respectively, right multiplied by the matrices ${{\varvec{D}}}_{\alpha },\left({{\varvec{I}}}_{n}-{{\varvec{D}}}_{\alpha }\right)$. Subsequently, by adding the two equations, the following form can be obtained.

$$\begin{array}{*{20}c} {\begin{array}{*{20}l} {} \hfill & {{\varvec{e}}_{c + 1} \overline{\user2{Y}}\left( {{\varvec{I}}_{n} - {\varvec{D}}_{\alpha } } \right) + {\varvec{e}}_{n} {\varvec{FD}}_{\alpha } = {\varvec{e}}_{n} } \hfill \\ \Rightarrow \hfill & {{\varvec{e}}_{c + 1} \overline{\user2{Y}}\left( {{\varvec{I}}_{n} - {\varvec{D}}_{\alpha } } \right) = {\varvec{e}}_{n} \left( {{\varvec{I}}_{n} - {\varvec{FD}}_{\alpha } } \right)} \hfill \\ \end{array} } \\ \end{array}$$

(25)

Thus, the problem can be proved by mathematical induction. In this paper, we choose $\overline{{\varvec{Y}} }\left(0\right)=\overline{{\varvec{Y}} }$. When $t=0$, according to Eq. (12) and Eq. (25), we get

$$\begin{array}{*{20}c} {{\varvec{e}}_{c + 1} \overline{\user2{Y}}\left( 1 \right) = {\varvec{e}}_{c + 1} \overline{\user2{Y}}F{\varvec{D}}_{\alpha } + {\varvec{e}}_{c + 1} \overline{\user2{Y}}\left( {{\varvec{I}}_{n} - {\varvec{D}}_{\alpha } } \right) = {\varvec{e}}_{n} } \\ \end{array}$$

(26)

When $t=k>1$, assuming that there is ${{\varvec{e}}}_{c+1}\overline{{\varvec{Y}} }\left(k\right)={{\varvec{e}}}_{n}$ holds, according to Eq. (12) and the result of Eq. (25), it is obtained that

$$\begin{aligned} {\varvec{e}}_{c + 1} \overline{\user2{Y}}\left( {k + 1} \right) & = {\varvec{e}}_{c + 1} \overline{\user2{Y}}\left( k \right){\varvec{FD}}_{\alpha } + {\varvec{e}}_{c + 1} \overline{\user2{Y}}\left( {{\varvec{I}}_{n} - {\varvec{D}}_{\alpha } } \right) \\ & = {\varvec{e}}_{n} {\varvec{FD}}_{\alpha } + {\varvec{e}}_{n} \left( {{\varvec{I}}_{n} - {\varvec{FD}}_{\alpha } } \right) = {\varvec{e}}_{n} \\ \end{aligned}$$

(27)

Thus, the conclusion is proved to be valid.

1.3 Appendix 3: Proof of the range of values of the Euclidean distance in the kernel space in this paper

Centering the kernel matrix is equivalent to centering the data in the kernel space, which is a translation operation and does not change the distance between data points. Thus, the study of the maximum distance between data points in the kernel space is equivalent to the study of the data before kernel matrix centrality. For any ${{\varvec{x}}}_{i},{{\varvec{x}}}_{j}\in {\varvec{X}}$, there are two points $\phi \left({{\varvec{x}}}_{i}\right),\phi \left({{\varvec{x}}}_{j}\right)\in {\mathfrak{R}}^{\nu }$ on the kernel space. It is not difficult to know from Eq. (7) that each element ${\varvec{K}}\left({{\varvec{x}}}_{i},{{\varvec{x}}}_{j}\right)$ of the kernel matrix takes values in the range $(\mathrm{0,1}]$. The original problem can be understood as follows: solve for the maximum distance between two points in the region of the positive semi-axis of a hypersphere with a radius of 1, in a high-dimensional kernel space. Thus, Eq. (8) can be simplified as follows.

$$\begin{aligned} d_{ij} & = \left\| {\phi \left( {{\varvec{x}}_{i} } \right) - \phi \left( {{\varvec{x}}_{j} } \right)} \right\| = \sqrt {{\varvec{K}}\left( {{\varvec{x}}_{i} ,{\varvec{x}}_{i} } \right) + {\varvec{K}}\left( {{\varvec{x}}_{j} ,{\varvec{x}}_{j} } \right) - 2{\varvec{K}}\left( {{\varvec{x}}_{i} ,{\varvec{x}}_{j} } \right)} \\ & = \sqrt {\left\| {\phi \left( {{\varvec{x}}_{i} } \right)} \right\|^{2} + \left\| {\phi \left( {{\varvec{x}}_{j} } \right)} \right\|^{2} - 2\phi \left( {{\varvec{x}}_{i} } \right)^{T} \phi \left( {{\varvec{x}}_{j} } \right)} \\ & \le \sqrt {1 + 1 - 0} = \sqrt 2 \\ \end{aligned}$$

(28)

where $\Vert \cdot \Vert$ denotes the modulus of the vector formed between $\phi \left({{\varvec{x}}}_{i}\right)$ and the origin, whose value is less than or equal to 1. On the other hand, for $\nu \ge 2$, the points can be found that satisfies the given condition and make the distance of Eq. (28) obtain its maximum value, e.g., the point $\phi \left( {{\varvec{x}}_{i} } \right) = \underbrace {{\left[ {1,0, \ldots ,0} \right]}}_{\nu }$ and $\phi \left( {{\varvec{x}}_{i} } \right) = \underbrace {0,1, \ldots ,0}_{\nu }$. More generally, the equality sign holds when and only when the vectors formed by each of these two points and the origin satisfy the modulus of 1 and the inner product of 0. Moreover, since the distance is non-negative, the Euclidean distance between two points on the kernel space of this paper takes values between 0 and $\sqrt{2}$.

1.4 Appendix 4: Proof of a simplified form

It is assumed that there are two data sets ${\varvec{X}}={[{{\varvec{x}}}_{1},{{\varvec{x}}}_{2},\dots ,{{\varvec{x}}}_{n}]}^{T}\in {\mathfrak{R}}^{n\times d}$ and ${\varvec{Y}}={[{{\varvec{y}}}_{1},{{\varvec{y}}}_{2},\dots ,{{\varvec{y}}}_{n}]}^{T}\in {\mathfrak{R}}^{n\times d}$. For any ${{\varvec{x}}}_{i},{{\varvec{y}}}_{i}$, the following simplification can be performed.

$$\begin{array}{*{20}c} {\left( {{\varvec{x}}_{i} - {\varvec{x}}_{j} } \right)^{T} \left( {{\varvec{y}}_{i} - {\varvec{y}}_{j} } \right) = \mathop \sum \limits_{\ell = 1}^{d} \left( {x_{i\ell } - x_{j\ell } } \right)\left( {y_{i\ell } - y_{j\ell } } \right) = tr\left( {{\varvec{X}}^{T} {\varvec{A}}^{ij} {\varvec{Y}}} \right)} \\ \end{array}$$

(29)

where ${\varvec{A}}^{ij} = \left[ {A^{ij}_{pq} } \right]_{n \times n} ,A^{ij}_{pq} = \left\{ {\begin{array}{*{20}l} 1 \hfill & {p = q\,{\text{ and}}\,{ }p,q \in \left\{ {i,j} \right\}} \hfill \\ { - 1} \hfill & {p \ne q\,{\text{ and}}\,{ }p,q \in \left\{ {i,j} \right\}} \hfill \\ 0 \hfill & {{\text{otherwise}}} \hfill \\ \end{array} } \right.$.