Transferable Adversarial Cycle Alignment for Domain Adaption

Abstract

Domain adaption is definitely critical for success in bridging source and target domains that data distribution shifts exist in domain or task. The state-of-the-art of the adversarial feature learning model named Bidirectional Generative Adversarial Networks (BiGAN), forces generative models to align with an arbitrarily complex distribution in a latent space. However, BiGAN only matches single data distribution without exploiting multi-domain structure, which means the learned latent representation could not transfer to related target domains. Recent research has proved that GANs combined with Cycle Consistent Constraints are effective at image translation. Therefore, we propose a novel framework named Transferable Bidirectional Generative Adversarial Networks combining with Cycle-Consistent Constraints (Cycle-TBiGAN) be applied in cross-domain translation, which aims at learning an alignment latent feature representation and achieving a mapping function between domains. Our framework is suitable for a wide variety of domain adaption scenarios. We show the surprising results in the task of image translation without prior ground-truth knowledge. Extensive experiments are presented on several public datasets. Quantitative comparisons demonstrate the superiority of our approach against previous methods.

A Supplement for Experiment Implementation

As we have described in Sects. 4.1 and 4.2, all the network architectures are repetition by two blocks, full connected block and convolutional or deconvolutional block, each defined by a fully connected layer at the last top layer, a Batch Normalization layer (BN), and a Dropout layer(P), followed by a fully connected (FC) layer or Convolutional layer (CON) with RELU or Leak_RELU activation functions. The Generator consists of Deconvolution layers (DCON) and the full connected output layers with sigmoid hidden units. Image preprocessing includes linear scaling all image sizes to 28 $\times$ 28, each image is represented by a 256-dimensional feature vector in feature representation space, which encodes the pixel information of the image.

In this section, we will give a detailed introduction about the specific design used to generate the result presented for Transferable Bidirectional Generative Adversarial Networks (TBiGAN) and Cycle-Consistent TBiGAN. A detailed description of architectures and hyperparameters (learning rate, batch sizes, etc.) is displayed in the following sections. We provide a basic necessary understanding of our experiments.

1.1 A.1 Transferable Bidirectional Generative Adversarial Networks (TBiGAN)

We apply TBiGAN to a task that aims at learning an invariant feature representation from the different domain distributions. We attempt to verify whether TBiGAN can learn a latent code space between domains by the objective we define in (3), (6) and (7).

For MNIST$\rightarrow$USPS, MNIST$\rightarrow$MNIST_m in Table 3, the generative model networks only contain several fully connected layers, the discriminator and Encoder both have the same structure with the generator. Since MNIST and USPS have similar domain distributions, a relatively simple network structure is proposed.

Table 3. Network architectures of TBiGAN for MNIST$\rightarrow$USPS, MNIST$\rightarrow$MNIST_m experiments

For MNIST$\rightarrow$MNIST_m in Table 4, we define a different network like conv-pool-conv-pool-fc-softmax. The Discriminator contains three conv-pool layers followed by two fully connected layers (depends on the different image preprocessing methods) activated by sigmoid units. In particular the Encoder for MNIST_m domain only has two hidden layers activated by ReLU units. A fully connected layer still be used as the last output layer.

Since SVHN has its own domain-specific properties, a single image contains several adjacent digits. The architectures of network need more convolutional layers to capture the domain information. Therefore, the discriminator has five conv-pool layers followed by last two full-connected layers activated with a sigmoid unit. The specific details of the Generator and Encoder are shown in Table 5.

1.2 A.2 Cycle-Consistent Crossing Domain Translation

The fundamental network architectures of Cycle-TBiGAN are similar to TBiGAN. We assume that necessary components such as generators ($G_S,\ G_T$) and encoder (E) corresponding to specific domain have been obtained from TBiGAN. The Generator $G_c(\cdot )$ is actually a translator that maps the latent code space of target domain $Z_T$ and source domain $Z_S$ to a synthesized code space $Z_{syn}$, which means the invariant feature representation space is regarded as input for $G_c(\cdot )$. A specific network description of mapping function $G_c(\cdot )$ is showed in Table 6.

Table 4. Network architectures of TBiGAN for MNIST$\rightarrow$MNIST_m

As defined in (16), the $L_m (\cdot )$ is a “similarity measurement”, which is used to find a subset of $z_{s_j}$ that similar to target image latent code $z_{t_i}$. Since the training of Cycle-TBiGAN is high computational cost and it should be relaxed, we use the K-Nearest-Neighbor (KNN) algorithm to find a subset of $z_{s_{1\cdot \cdot \cdot k}}$ with size k from source latent code space. In other words, the latent subset should be similar to $z_{t_i}$. Therefore, the relaxed (16) could be presented as:

$$\begin{aligned} Z_{syn}=G_c(Z_S,Z_T ) =\sum _{z_{t_i}} \omega _{i,j} \cdot KNN_k(z_{t_i},z_{s_j}) \end{aligned}$$

(17)

The relaxed objective could be optimized using SGD.

Table 5. Network architectures of TBiGAN for SVHN$\rightarrow$ MNIST_m

Table 6. Network architectures of Cycle-TBiGAN for MNIST$\rightarrow$USPS, SVHN$\rightarrow$MNIST_m image translation