Adaptive semi-supervised learning on labeled and unlabeled data with different distributions

Abstract

Developing methods for designing good classifiers from labeled samples whose distribution is different from that of test samples is an important and challenging research issue in the fields of machine learning and its application. This paper focuses on designing semi-supervised classifiers with a high generalization ability by using unlabeled samples drawn by the same distribution as the test samples and presents a semi-supervised learning method based on a hybrid discriminative and generative model. Although JESS-CM is one of the most successful semi-supervised classifier design frameworks based on a hybrid approach, it has an overfitting problem in the task setting that we consider in this paper. We propose an objective function that utilizes both labeled and unlabeled samples for the discriminative training of hybrid classifiers and then expect the objective function to mitigate the overfitting problem. We show the effect of the objective function by theoretical analysis and empirical evaluation. Our experimental results for text classification using four typical benchmark test collections confirmed that with our task setting in most cases, the proposed method outperformed the JESS-CM framework. We also confirmed experimentally that the proposed method was useful for obtaining better performance when classifying data samples into either known or unknown classes, which were included in given labeled samples or not, respectively.

Appendix

1.1 Derivation of objective function for parameter estimation

We derive Eq. (14) from Eq. (12). By substituting Eq. (7) for $P(k|{\varvec{{x}}})$ in Eq. (9) for labeled samples, $D_l = \{({\varvec{{x}}}_n,y_n)\}_{n=1}^N$, and substituting Eq. (13) for $P(k|{\varvec{{x}}})$ in Eq. (9) for unlabeled samples, $D_u =\{{\varvec{{x}}}_m\}_{m=1}^M$, we can transform Eq. (8) to

$$\begin{aligned} J_d(W)&= \sum _{n=1}^N \log P_d(y_n|{\varvec{{x}}}_n;W)\nonumber \\&+ \sum _{m=1}^M \sum _{k=1}^K P(k|{\varvec{{x}}}_m;W,\Theta ,\beta ) \log \frac{P_d(k|{\varvec{{x}}}_m;W)}{P(k|{\varvec{{x}}}_m;W,\Theta ,\beta )} + \log p(W) \nonumber \\&= \sum _{n=1}^N \log P_d(y_n|{\varvec{{x}}}_n;W) + \sum _{m=1}^M \log \sum _{k=1}^K P_d(k|{\varvec{{x}}}_m;W)p_g({\varvec{{x}}}_m,k;\varvec{\theta }_k)^{\beta } \nonumber \\&- \beta \sum _{m=1}^M \sum _{k=1}^K P(k|{\varvec{{x}}}_m;W,\Theta ,\beta ) \log p_g({\varvec{{x}}}_m,k;\varvec{\theta }_k) + \log p(W). \end{aligned}$$

(24)

By substituting Eqs. (7) and (13) for $P(k|{\varvec{{x}}})$ in Eq. (11) for $D_l = \{({\varvec{{x}}}_n,y_n)\}_{n=1}^N$ and $D_u =\{{\varvec{{x}}}_m\}_{m=1}^M$, respectively, we can transform Eq. (10) to

$$\begin{aligned} J_g(\Theta )&= \sum _{n=1}^N p_g({\varvec{{x}}}_n,y_n;|\varvec{\theta }_{y_n}) + \sum _{m=1}^M P(k|{\varvec{{x}}}_m;W,\Theta ,\beta ) \log p_g({\varvec{{x}}}_m,k;\varvec{\theta }_k) + \log p(\Theta ).\nonumber \\ \end{aligned}$$

(25)

By substituting these equations for $J_d(W)$ and $J_g(\Theta )$ in Eq. (12), we can obtain Eq. (14).

1.2 Proof of inequality about $Q$-function

We prove the inequality, $J(\Psi ) - J (\Psi ^{(t)}) \ge Q(\Psi , \Psi ^{(t)}) - Q(\Psi ^{(t)}, \Psi ^{(t)})$, described in Sect. 4.2. From Eq. (14), we can obtain the equation,

$$\begin{aligned}&J(\Psi )- J (\Psi ^{(t)}) \nonumber \\&\quad = \log \frac{p(W)}{p(W^{(t)})} + \beta \log \frac{p(\Theta )}{p(\Theta ^{(t)})} + \sum _{n=1}^N \log \frac{P_d(y_n|{\varvec{{x}}}_n;W)p_g({\varvec{{x}}}_n,y_n;\varvec{\theta }_{y_n})^{\beta }}{P_d(y_n|{\varvec{{x}}}_n;W^{(t)})p_g({\varvec{{x}}}_n,y_n;\varvec{\theta }_{y_n}^{(t)})^{\beta }} \nonumber \\&\quad + \sum _{m=1}^M \log \frac{\sum _{k=1}^K P_d(k|{\varvec{{x}}}_m;W)p_g({\varvec{{x}}}_m,k;\varvec{\theta }_k)^{\beta }}{\sum _{k=1}^K P_d(k|{\varvec{{x}}}_m;W^{(t)})p_g({\varvec{{x}}}_m,k;\varvec{\theta }_k^{(t)})^{\beta }} \nonumber \\&\quad = \log \frac{p(W)}{p(W^{(t)})} + \beta \log \frac{p(\Theta )}{p(\Theta ^{(t)})} + \sum _{n=1}^N \log \frac{P_d(y_n|{\varvec{{x}}}_n;W)}{P_d(y_n|{\varvec{{x}}}_n;W^{(t)})} \frac{p_g({\varvec{{x}}}_n,y_n;\varvec{\theta }_{y_n})^{\beta }}{p_g({\varvec{{x}}}_n,y_n;\varvec{\theta }_{y_n}^{(t)})^{\beta }} \nonumber \\&\quad + \sum _{m=1}^M \sum _{k=1}^K P(k|{\varvec{{x}}}_m;\Psi ^{(t)},\beta ) \log \frac{P_d(k|{\varvec{{x}}}_m;W)}{P_d(k|{\varvec{{x}}}_m;W^{(t)})} \frac{p_g({\varvec{{x}}}_m,k;\varvec{\theta }_k)^{\beta }}{p_g({\varvec{{x}}}_m,k;\varvec{\theta }_k^{(t)})^{\beta }} \frac{P(k|{\varvec{{x}}}_m;\Psi ^{(t)},\beta )}{P(k|{\varvec{{x}}}_m;\Psi ,\beta )}.\nonumber \\ \end{aligned}$$

(26)

According to Eqs. (15)–(17), we can transform the above equation to

$$\begin{aligned}&J(\Psi ) - J (\Psi ^{(t)}) \nonumber \\&\quad = Q(\Psi , \Psi ^{(t)}) - Q(\Psi ^{(t)}, \Psi ^{(t)}) + \sum _{m=1}^M \sum _{k=1}^K P(k|{\varvec{{x}}}_m;\Psi ^{(t)},\beta ) \log \frac{P(k|{\varvec{{x}}}_m;\Psi ^{(t)},\beta )}{P(k|{\varvec{{x}}}_m;\Psi ,\beta )}. \end{aligned}$$

(27)

Since $\log b \le b - 1$, $\sum _{k=1}^K P(k|{\varvec{{x}}}_m;\Psi ,\beta ) = 1$, and $\sum _{k=1}^K P(k|{\varvec{{x}}}_m;\Psi ^{(t)},\beta ) = 1$,

$$\begin{aligned}&J(\Psi ) - J (\Psi ^{(t)}) \nonumber \\&\quad \ge Q(\Psi , \Psi ^{(t)}) - Q(\Psi ^{(t)}, \Psi ^{(t)}) + \sum _{m=1}^M \sum _{k=1}^K P(k|{\varvec{{x}}}_m;\Psi ^{(t)},\beta ) \left\{ 1- \frac{P(k|{\varvec{{x}}}_m;\Psi ,\beta )}{P(k|{\varvec{{x}}}_m;\Psi ^{(t)},\beta )} \right\} \nonumber \\&\quad = Q(\Psi , \Psi ^{(t)}) - Q(\Psi ^{(t)}, \Psi ^{(t)}). \end{aligned}$$

(28)

1.3 Proof that $g_d$ is Concave

If the Hessian matrix of $g_d (W,\Psi ^{(t)})$ shown in Eq. (16) is negative semidefinite, $g_d (W,\Psi ^{(t)})$ is a concave function with respect to $W$. We prove that the Hessian matrix of $g_d (W,\Psi ^{(t)})$ is negative semidefinite when applying the MLR model and Gaussian prior described in Sect. 4.4.

Using the MLR model and Gaussian prior, $P_d(y|{\varvec{{x}}};W) = \exp \left( {\varvec{{w}}}_y^T {\varvec{{x}}}\right)/\sum _{k=1}^K \exp \left( {\varvec{{w}}}_k^T {\varvec{{x}}}\right)$ and $p(W) = \prod _{k=1}^K \exp \left(-{\varvec{{w}}}_k^T {\varvec{{w}}}_k / 2 \sigma ^2\right)$, the objective function, $g_d (W,\Psi ^{(t)})$, shown in Eq. (16) is rewritten as

$$\begin{aligned} g_d (W;\Psi ^{(t)})&= - \sum _{k=1}^K \frac{{\varvec{{w}}}_k^T {\varvec{{w}}}_k}{2 \sigma ^2} + \sum _{n=1}^N \left\{ {\varvec{{w}}}_{y_n}^T {\varvec{{x}}}_n - \log \sum _{k=1}^K \exp ({\varvec{{w}}}_k^T {\varvec{{x}}}_n) \right\} \nonumber \\&+ \sum _{m=1}^M \left\{ \sum _{k=1}^K P(k|{\varvec{{x}}}_m;\Psi ^{(t)},\beta ) {\varvec{{w}}}_k^T {\varvec{{x}}}_m - \log \sum _{k=1}^K \exp ({\varvec{{w}}}_k^T {\varvec{{x}}}_m) \right\} . \end{aligned}$$

(29)

To obtain the Hessian matrix $\left[\partial ^2 g_d/\partial {\varvec{{w}}}_k \partial {\varvec{{w}}}_{k^{\prime }}^T\right]_{k,k^{\prime }}$ of $g_d$, we partially differentiate $g_d$ with respect to ${\varvec{{w}}}_k$ such that

$$\begin{aligned} \frac{\partial g_d}{\partial {\varvec{{w}}}_k}&= - \frac{{\varvec{{w}}}_k}{\sigma ^2} + \sum _{n=1}^N \left\{ I_{y_n} (k) - P_d(k|{\varvec{{x}}}_n;W)\right\} {\varvec{{x}}}_n \nonumber \\&+ \sum _{m=1}^M \left\{ P(k|{\varvec{{x}}}_m;\Psi ^{(t)},\beta ) - P_d(k|{\varvec{{x}}}_m;W)\right\} {\varvec{{x}}}_m, \end{aligned}$$

(30)

where $I_{y_n} (k)$ is an indicator function that satisfies $I_{y_n} (k) = 1~(I_{y_n} (k) = 0)$ when $k = y_n (k \ne y_n)$. Then, we partially differentiate $\partial g_d/\partial {\varvec{{w}}}_k$ with respect to ${\varvec{{w}}}_{k^{\prime }}$ such that

$$\begin{aligned} \frac{\partial ^{2} g_{d}}{\partial {\varvec{{w}}}_k \partial {\varvec{{w}}}_{k^\prime }^{T}}&= - \frac{1}{\sigma ^2} I_k (k^{\prime }) {\varvec{{I}}}_V - \sum _{n=1}^N P_d(k|{\varvec{{x}}}_n;W) \left\{ I_{k} (k^\prime ) - P_d(k^\prime |{\varvec{{x}}}_n;W)\right\} {\varvec{{x}}}_n {\varvec{{x}}}_n^T \nonumber \\&- \sum _{m=1}^M P_d(k|{\varvec{{x}}}_m;W) \left\{ I_{k} (k^\prime ) - P_d(k^\prime |{\varvec{{x}}}_m;W)\right\} {\varvec{{x}}}_m {\varvec{{x}}}_{m}^{T}, \end{aligned}$$

(31)

where ${\varvec{{I}}}_V$ is the $(V \times V)$-dimensional identity matrix, and $V$ is consistent with the dimension of ${\varvec{{w}}}_k$. Then, for arbitrary $VK$-dimensional vector ${\varvec{{u}}}=({\varvec{{u}}}_{1}^{T},\ldots ,{\varvec{{u}}}_{k}^{T},\ldots ,{\varvec{{u}}}_{K}^{T})^T$, where ${\varvec{{u}}}_k = (u_{k1},\ldots ,u_{ki},\ldots ,u_{kV})^{T}$,

$$\begin{aligned}&{\varvec{{u}}}^T \left[ \frac{\partial ^{2} g_d}{\partial {\varvec{{w}}}_{k} \partial {\varvec{{w}}}_{k^\prime }^{T}} \right]_{k,k^\prime }{\varvec{{u}}}\nonumber \\&\quad = - \sum _{k=1}^K \frac{{\varvec{{u}}}_{k}^{T} {\varvec{{u}}}_{k}}{\sigma ^{2}} - \sum _{n=1}^N \sum _{k=1}^K P_d(k|{\varvec{{x}}}_n;W) {\varvec{{u}}}_{k}^{T} {\varvec{{x}}}_n \left\{ {\varvec{{x}}}_{n}^{T} {\varvec{{u}}}_k - \sum _{k^\prime =1}^{K} P_d (k^\prime |{\varvec{{x}}}_{n};W) {\varvec{{x}}}_{n}^{T} {\varvec{{u}}}_{k^\prime }\right\} \nonumber \\&\quad - \sum _{m=1}^M \sum _{k=1}^K P_d(k|{\varvec{{x}}}_m;W) {\varvec{{u}}}_k^T {\varvec{{x}}}_m \left\{ {\varvec{{x}}}_m^T {\varvec{{u}}}_k - \sum _{k^\prime =1}^K P_d (k^\prime |{\varvec{{x}}}_m;W) {\varvec{{x}}}_{m}^{T}{\varvec{{u}}}_{k^\prime }\right\} \nonumber \\&\quad = - \sum _{k=1}^K \frac{{\varvec{{u}}}_{k}^{T} {\varvec{{u}}}_k}{\sigma ^{2}} - \sum _{n=1}^N \sum _{k=1}^K P_d(k|{\varvec{{x}}}_n;W) \left\{ {\varvec{{x}}}_{n}^{T}{\varvec{{u}}}_k - \sum _{k^\prime =1}^K P_d (k^\prime |{\varvec{{x}}}_n;W) {\varvec{{x}}}_{n}^{T} {\varvec{{u}}}_{k^\prime }\right\} ^2 \nonumber \\&\quad - \sum _{m=1}^M \sum _{k=1}^K P_d(k|{\varvec{{x}}}_m;W) \left\{ {\varvec{{x}}}_{m}^{T} {\varvec{{u}}}_k - \sum _{k^\prime =1}^K P_d (k^\prime |{\varvec{{x}}}_m;W) {\varvec{{x}}}_{m}^{T} {\varvec{{u}}}_{k^\prime } \right\} ^2, \end{aligned}$$

(32)

because $\sum _{k=1}^K \!P_d (k|{\varvec{{x}}};\!W) \!=\! 1$ and $P_d (k|{\varvec{{x}}};\!W) \!\ge \! 0$. When ${\varvec{{u}}}\!\ne \! \mathbf{0}, {\varvec{{u}}}^{T} \!\left[\partial ^2 g_d /\partial {\varvec{{w}}}_k \partial {\varvec{{w}}}_{k^\prime }^T \right]_{k,k^\prime }$ ${\varvec{{u}}}\!<\! 0$ for arbitrary $W$. This shows that the Hessian matrix of $g_d(W,\Psi ^{(t)})$ with respect to $W$ is negative semidefinite.