Effective Multiclass Transfer for Hypothesis Transfer Learning

Abstract

In this paper, we investigate the visual domain adaptation problem under the setting of Hypothesis Transfer Learning (HTL) where we can only access the source model instead of the data. However, previous studies of HTL are limited to either leveraging the knowledge from certain type of source classifier or low transfer efficiency on a small training set. In this paper, we aim at two important issues: effectiveness of the transfer on small target training set and compatibility of the transfer model for real-world HTL problems. To solve these two issues, we proposed our method, Effective Multiclass Transfer Learning (EMTLe). We demonstrate that EMTLe, which uses the prediction of the source models as the transferable knowledge can exploit the knowledge of different types of source classifiers. We use the transfer parameter to weigh the importance the prediction of each source model as the auxiliary bias. Then we use the bi-level optimization to estimate the transfer parameter and demonstrate that we can effectively obtain the optimal transfer parameter with our novel objective function. Empirical results show that EMTLe can effectively exploit the knowledge and outperform other HTL baselines when the size of the target training set is small.

Appendix

Theorem 1

Let $L(\beta )$ be a $\lambda$-strongly convex function and $\beta ^*$ be its optimal solution. Let $\beta _1,...,\beta _{T+1}$ be a sequence such that $\beta _1 \in B$ and for $t>1$, we have $\beta _{t+1} = \beta _t - \eta _t \varDelta _t$, where $\varDelta _t$ is the sub-gradient of $L(\beta _t)$ and $\eta _t = 1/(\lambda t)$. Assume we have $||\varDelta _t|| \le G$ for all t. Then we have:

$$\begin{aligned} L(\beta _{T+1}) \le L(\beta ^*)+\frac{G^2(1+\ln (T))}{2\lambda T} \end{aligned}$$

(9)

Proof

As $L(\beta )$ is strongly convex and $\varDelta _t$ is in its sub-gradient set at $\beta _t$, according to the definition of $\lambda$-strong convexity [15], the following inequality holds:

$$\begin{aligned} \left\langle {\beta _t - \beta ^*,\varDelta _t} \right\rangle \ge L(\beta _t)-L(\beta ^*)+\frac{\lambda }{2}||\beta _t - \beta ^*||^2 \end{aligned}$$

(10)

For the term $\left\langle {\beta _t - \beta ^*,\varDelta _y} \right\rangle$, it can be written as:

$$\begin{aligned} \begin{aligned} \left\langle {\beta _t - \beta ^*,\varDelta _t} \right\rangle&= \left\langle {\beta _t - \frac{1}{2}\eta _t\varDelta _t + \frac{1}{2}\eta _t\varDelta _t- \beta ^*,\varDelta _t} \right\rangle =\frac{1}{2}\left\langle {{\beta _{t + 1}} + {\beta _t} - 2{\beta ^*},{\varDelta _t}} \right\rangle + \frac{1}{2}{\eta _t}\varDelta _t^2 \end{aligned} \end{aligned}$$

(11)

Then we have:

$$\begin{aligned} \begin{aligned} ||\beta _t-\beta ^*||^2-||\beta _{t+1}-\beta ^*||^2 =\left\langle {{\beta _{t + 1}} + {\beta _t} - 2{\beta ^*},{\eta _t\varDelta _t}} \right\rangle \end{aligned} \end{aligned}$$

(12)

Using the assumption $||\varDelta _t|| \le G$, we can rearrange (10) and plug (11) and (12) into it, we have:

$$\begin{aligned} \begin{aligned}&{Diff}_t = L(\beta _t)-L(\beta ^*) \le \frac{\lambda (t-1)}{2}{||{\beta _t} - {\beta ^*}||^2}- \frac{\lambda t}{2}{||{\beta _{t+1}} - {\beta ^*}||^2}+\frac{1}{2}{\eta _t} G^2 \end{aligned} \end{aligned}$$

(13)

Due to the convexity, for each pair of $L(\beta _t)$ and $L(\beta _{t+1})$ for $t=1,...,T$, we have the following sequence $L(\beta ^*) \le L(\beta _T) \le L(\beta _{T-1}) \le ...\le L(\beta _1)$. For the sequence $Diff_t$ for $t=1,...,T$, we have:

$$\begin{aligned} \sum _{t=1}^{T} Diff_t = \sum _{t=1}^{T}L(\beta _t)-TL(\beta ^*) \ge T\left[ L(\beta _T)-L(\beta ^*)\right] \end{aligned}$$

(14)

Next, we show that

$$\begin{aligned} \begin{aligned}&\sum _{t=1}^{T} Diff_t = \sum _{t=1}^{T}\left\{ \frac{\lambda (t-1)}{2}{||{\beta _t} - {\beta ^*}||^2}- \frac{\lambda t}{2}{||{\beta _{t+1}} - {\beta ^*}||^2}+\frac{1}{2}{\eta _t} G^2\right\} \\&=-\frac{\lambda T}{2}{||{\beta _{T+1}-\beta ^*}||^2} + \frac{G^2}{2 \lambda }\sum _{t=1}^{T} \frac{1}{t}\le \frac{G^2}{2 \lambda }\sum _{t=1}^{T} \frac{1}{t} \le \frac{G^2}{2 \lambda }(1+\ln (T)) \end{aligned} \end{aligned}$$

(15)

Combining (14) and rearranging the result, we have:

$$\begin{aligned} L(\beta _{T+1}) \le L(\beta ^*)+\frac{G^2(1+\ln (T))}{2\lambda T} \end{aligned}$$