Learning Kernels for Unsupervised Domain Adaptation with Applications to Visual Object Recognition

Abstract

Domain adaptation aims to correct the mismatch in statistical properties between the source domain on which a classifier is trained and the target domain to which the classifier is to be applied. In this paper, we address the challenging scenario of unsupervised domain adaptation, where the target domain does not provide any annotated data to assist in adapting the classifier. Our strategy is to learn robust features which are resilient to the mismatch across domains and then use them to construct classifiers that will perform well on the target domain. To this end, we propose novel kernel learning approaches to infer such features for adaptation. Concretely, we explore two closely related directions. In the first direction, we propose unsupervised learning of a geodesic flow kernel (GFK). The GFK summarizes the inner products in an infinite sequence of feature subspaces that smoothly interpolates between the source and target domains. In the second direction, we propose supervised learning of a kernel that discriminatively combines multiple base GFKs. Those base kernels model the source and the target domains at fine-grained granularities. In particular, each base kernel pivots on a different set of landmarks—the most useful data instances that reveal the similarity between the source and the target domains, thus bridging them to achieve adaptation. Our approaches are computationally convenient, automatically infer important hyper-parameters, and are capable of learning features and classifiers discriminatively without demanding labeled data from the target domain. In extensive empirical studies on standard benchmark recognition datasets, our appraches yield state-of-the-art results compared to a variety of competing methods.

Appendices

Appendices

1.1 Appendix 1: Derivation of Geodesic Flow Kernel (GFK)

Let $\varvec{\varOmega }^{\mathrm{T}}$ denote the following matrix

$$\begin{aligned} \varvec{\varOmega }^{\mathrm{T}}= [\varvec{P}_\mathcal {S}\quad \varvec{R}_\mathcal {S}]\left[ \begin{aligned} \varvec{U}_1&\varvec{0}\\ \varvec{0}&\varvec{U}_2\end{aligned} \right] . \end{aligned}$$

(21)

The geodesic flow $\varvec{\varPhi }(t),\; t\in (0,1)$, between $\varvec{P}_\mathcal {S}$ and $\varvec{P}_\mathcal {T}$ can be written as

$$\begin{aligned} \varvec{\varPhi }(t) = \varvec{P}_\mathcal {S}\varvec{U}_1 \varvec{\varGamma }(t) - \varvec{R}_\mathcal {S}\varvec{U}_2 \varvec{\varSigma }(t) = \varvec{\varOmega }^{\mathrm{T}}\left[ \begin{aligned} \varvec{\varGamma }(t)\\ -\varvec{\varSigma }(t)\end{aligned} \right] . \end{aligned}$$

(22)

Recall that the geodesic flow kernel (gfk) is defined as,

$$\begin{aligned} \langle \varvec{z}_i^\infty , \varvec{z}_j^\infty \rangle =\int \limits _0^1 (\varvec{\varPhi }(t)^{\mathrm{T}}\varvec{x}_i)^{\mathrm{T}}(\varvec{\varPhi }(t)^{\mathrm{T}}\varvec{x}_j)\ dt = \varvec{x}_i^{\mathrm{T}}\varvec{G}\varvec{x}_j, \end{aligned}$$

(23)

where

$$\begin{aligned} \varvec{G} = \int \limits _{0}^1{\varvec{\varPhi }(t)\varvec{\varPhi }(t)^{\mathrm{T}}}dt. \end{aligned}$$

(24)

Substituting the expression of $\varvec{\varPhi }(t)$ of Eq. (22) into above, we have (ignoring $\varvec{\varOmega }$ for the moment),

$$\begin{aligned} \varvec{G} \propto \int \limits _{0}^1 \left[ \begin{aligned} \varvec{\varGamma }(t)\varvec{\varGamma }(t)&-\varvec{\varGamma }(t)\varvec{\varSigma }(t)\\ -\varvec{\varSigma }(t)\varvec{\varGamma }(t)&\varvec{\varSigma }(t)\varvec{\varSigma }(t)\end{aligned} \right] dt \end{aligned}$$

(25)

Both $\varvec{\varGamma }(t)$ and $\varvec{\varSigma }(t)$ are diagonal matrices with elements being $\cos (t\theta _i)$ and $\sin (t\theta _i)$. Thus, we can integrate in close-form,

$$\begin{aligned} \lambda _{1i}&= \int \limits _0^1 \cos ^2(t\theta _i) dt =1+\frac{\sin (2\theta _i)}{2\theta _i}, \end{aligned}$$

(26)

$$\begin{aligned} \lambda _{2i}&= - \int \limits _0^1 \cos (t\theta _i) \sin (t\theta _i)dt = \frac{\cos (2\theta _i)-1}{2\theta _i}\end{aligned}$$

(27)

$$\begin{aligned} \lambda _{3i}&= \int \limits _0^1 \sin ^2(t\theta _i) dt = 1-\frac{\sin (2\theta _i)}{2\theta _i}, \end{aligned}$$

(28)

which become the $i$th diagonal elements of diagonal matrices $\varvec{\varLambda }_1,\; \varvec{\varLambda }_2$, and $\varvec{\varLambda }_3$, respectively. In terms of these matrices, the inner product Eq. (23) is a linear kernel $\varvec{x}_i^{\mathrm{T}}\varvec{G}\varvec{x}_j$ with the matrix $\varvec{G}$ given by

$$\begin{aligned} \varvec{G}= \varvec{\varOmega }^{\mathrm{T}}\left[ \begin{aligned} \varvec{\varLambda }_1&\varvec{\varLambda }_2\\ \varvec{\varLambda }_2&\varvec{\varLambda }_3\end{aligned} \right] \varvec{\varOmega }. \end{aligned}$$

(29)

1.2 Appendix 2: Derivation of the Rank of Domain (ROD) Metric

1.2.1 Principal Angles and Vectors

Let $\varvec{P}_\mathcal {S}$ and $\varvec{P}_\mathcal {T}$ be the basis of two subspaces. The principal angles $\theta _i$ between the two subspaces are recursively defined as,

$$\begin{aligned} \cos (\theta _i) = \max _{\varvec{s}_i \in \mathrm {span}(\varvec{P}_\mathcal {S})}\max _{\varvec{t}_i \in \mathrm {span}(\varvec{P}_\mathcal {T})} \frac{\langle \varvec{s}_i, \varvec{t}_i\rangle }{\Vert \varvec{s}_i\Vert \Vert \varvec{t}_i\Vert }, \end{aligned}$$

(30)

such that

$$\begin{aligned} \begin{array}{cc} \varvec{s}_k \in \mathrm {span}(\varvec{P}_\mathcal {S}), &{} \varvec{s}_i \bot \varvec{s}_k, \\ \varvec{t}_k \in \mathrm {span}(\varvec{P}_\mathcal {T}), &{} \varvec{t}_i \bot \varvec{t}_k, \\ \end{array} k = 1,2,\ldots i-1. \end{aligned}$$

In the above, $\varvec{s}_i$ and $\varvec{t}_i$ are called the principal vectors associated with $\theta _i$. Essentially, principal vectors are new basis for the two subspaces such that after the change of the basis, the two subspaces maximally overlap. The degrees of overlapping are measured in the principal angles—the smallest angles between basis.

Given the singular value decomposition,

$$\begin{aligned} \varvec{P}_\mathcal {S}^{\mathrm{T}}\varvec{P}_\mathcal {T}= \varvec{U}_1\varvec{\varGamma }\varvec{V}^{\mathrm{T}}\end{aligned}$$

(31)

both the principal angles and vectors can be computed efficiently

$$\begin{aligned} \theta _i = \arccos \gamma _{i}, \varvec{s}_i = (\varvec{P}_\mathcal {S}\varvec{U}_1)_{\cdot ,i}, \varvec{t}_i = (\varvec{P}_\mathcal {T}\varvec{V})_{\cdot ,i}, \end{aligned}$$

(32)

where $\gamma _{i}$ is the $i$th diagonal element of the diagonal matrix $\varvec{\varGamma }$. $(\varvec{M})_{\cdot ,i}$ returns the $i$th column of the matrix $\varvec{M}$.

1.2.2 Computing ROD

Let $\varvec{X}_\mathcal {S}\in \mathbb {R}^{\mathsf {N}_S \times \mathsf {D}}$ and $\varvec{X}_\mathcal {T}\in \mathbb {R}^{\mathsf {N}_T \times \mathsf {D}}$ denote the data from the source and the target domains. We use their PCA subspaces to compute the ROD metric. The optimal dimensionality $\mathsf {d}^*$ of the subspaces is selected with our subspace disagreement measure, described in Sect. 2.4 in the main text.

The ROD metric integrates both geometrical and statistical information between two domains by

$$\begin{aligned} \mathcal {R}(\mathcal {S},\mathcal {T}) = \frac{1}{\mathsf {d}^*}\sum _{i}^{{\mathsf {d}^*}}\theta _i \left[ KL(\mathcal {S}_i \Vert \mathcal {T}_i) + KL(\mathcal {T}_i\Vert \mathcal {S}_i)\right] , \end{aligned}$$

(33)

where $\mathcal {S}_i$ and $\mathcal {T}_i$ are two one-dimensional distributions of $\varvec{X}_\mathcal {S}^{\mathrm{T}}\varvec{s}_i$ and $\varvec{X}_\mathcal {T}^{\mathrm{T}}\varvec{t}_i$, respectively. In other words, we project data onto the principal vectors and compare how (dis)similar the data are distributed across domains.

We approximate these two distributions with one- dimensional Gaussians. Note that $\varvec{X}_\mathcal {S}$ and $\varvec{X}_\mathcal {T}$ have zero-means. We thus need only to compute the variances in order to specify the Gaussians. These variances can be readily computed from the projections and the covariance matrices of the original data:

$$\begin{aligned} \sigma _{i\mathcal {S}}^2 = \frac{1}{\mathsf {N}_S} \varvec{s}_i^{\mathrm{T}}\varvec{X}_\mathcal {S}^{\mathrm{T}}\varvec{X}_\mathcal {S}\varvec{s}_i,\quad \sigma _{i\mathcal {T}}^2 = \frac{1}{\mathsf {N}_T} \varvec{t}_i^{\mathrm{T}}\varvec{X}_\mathcal {T}^{\mathrm{T}}\varvec{X}_\mathcal {T}\varvec{t}_i, \end{aligned}$$

(34)

In terms of the approximating Gaussians, the ROD metric is computed in close-form

$$\begin{aligned} \mathcal {R}(\mathcal {S},\mathcal {T}) = \frac{1}{\mathsf {d}^*}\sum _{i}^{{\mathsf {d}^*}}\theta _i \left[ \frac{1}{2}\frac{\sigma _{i\mathcal {S}}^2}{\sigma _{i\mathcal {T}}^2} + \frac{1}{2}\frac{\sigma _{i\mathcal {T}}^2}{\sigma _{i\mathcal {S}}^2} -1\right] . \end{aligned}$$

(35)

1.3 Appendix 3: Proof of Theorem 1

We first prove the following lemma.

Lemma 1

Under the condition of the Theorem 1, the following inequality holds,

$$\begin{aligned} KL(P_{S}\Vert P_L)\le KL(P_{S}\Vert P_T) \end{aligned}$$

(36)

Proof

We start with

$$\begin{aligned} KL(P_{S}\Vert P_T)&= KL(\alpha P_N+(1-\alpha )P_L\Vert P_T) \\&=\int \left[ \alpha P_N+(1-\alpha )P_L\right] \\&\qquad \log \frac{\alpha P_N+(1-\alpha )P_L}{P_T}\, dX \end{aligned}$$

We now use the property that $\log$ function is concave to arrive at

$$\begin{aligned} KL(P_{S}\Vert P_T)&\ge \int \left[ \alpha P_N+(1-\alpha )P_L\right] \left[ \alpha \log \frac{P_N}{P_T}\right. \nonumber \\&\quad \left. +(1-\alpha )\log \frac{P_L}{P_T}\right] \, dX \nonumber \\&=\alpha ^{2}KL(P_N\Vert P_T)+(1-\alpha )^{2}KL(P_L\Vert P_T) \nonumber \\&\quad + \alpha (1-\alpha )C(P_L,P_N,P_T), \end{aligned}$$

(37)

where

$$\begin{aligned}&C(P_L,P_N,P_T)\nonumber \\&\quad =\int \left( P_N\log \frac{P_L}{P_T}+P_L\log \frac{P_N}{P_T}\right) \, dX \nonumber \\&\quad =\int \left( P_N\log \frac{P_N}{P_T}-P_N\log \frac{P_N}{P_L}+P_L\log \frac{P_L}{P_T}-P_L\log \frac{P_L}{P_N}\right) \, dX \nonumber \\&\quad =KL(P_N\Vert P_T)-KL(P_N\Vert P_L)+KL(P_L\Vert P_T)-KL(P_L\Vert P_N) \end{aligned}$$

(38)

Substituting Eq. (38) into Eq. (37), we have

$$\begin{aligned} KL(P_{S}\Vert P_T)&\ge \alpha KL(P_N\Vert P_T)+(1-\alpha )KL(P_L\Vert P_T) \nonumber \\&\quad -\alpha (1-\alpha )\left[ KL(P_N\Vert P_L)+KL(P_L\Vert P_N)\right] \end{aligned}$$

(39)

Applying to the right hand side of the inequality the condition of the Theorem 1, we have

$$\begin{aligned} KL(P_{S}\Vert P_T) \ge \left[ \frac{9}{8} - 2\alpha (1-\alpha )\right] A \end{aligned}$$

(40)

where $A = \max \left\{ KL(P_N\Vert P_L),\, KL(P_L\Vert P_N)\right\}$.

Note that

$$\begin{aligned} \frac{9}{8} - 2\alpha (1-\alpha ) \ge \alpha \end{aligned}$$

as the maximum of $2\alpha (1-\alpha )+\alpha$ is $9/8$, attained at $\alpha = 3/4$. This leads to

$$\begin{aligned} KL(P_{S}\Vert P_T) \ge \alpha A \ge \alpha KL(P_N\Vert P_L) \end{aligned}$$

(41)

To complete the proof the lemma, note that due to the convexity of KL-divergence, we have

$$\begin{aligned} KL(P_{S}\Vert P_L)&= KL(\alpha P_N+(1-\alpha )P_L\Vert P_L)\\&\le \alpha KL(P_N\Vert P_L) \end{aligned}$$

Combining the last two inequalities together, we complete the proof of the lemma.$\square$

Proof of the Theorem We start by applying the convex property of the KL-divergence again,

$$\begin{aligned} KL(P_{S}\Vert Q_{T})&= KL(P_{S}\Vert \beta P_T+(1-\beta )P_L) \nonumber \\&\le \beta KL(P_{S}\Vert P_T)+(1-\beta )KL(P_{S}\Vert P_L) \nonumber \\&\le \beta KL(P_{S}\Vert P_T)+(1-\beta )KL(P_{S}\Vert P_T) \nonumber \\&\le KL (P_S \Vert P_T) \end{aligned}$$

(42)

where we have applied the Lemma 1 in the penultimate inequality. The last inequality states the desired result of the theorem.