A one-to-many conditional generative adversarial network framework for multiple image-to-image translations

Abstract

Image-to-Image translation was proposed as a general form of many image learning problems. While generative adversarial networks were successfully applied on many image-to-image translations, many models were limited to specific translation tasks and were difficult to satisfy practical needs. In this work, we introduce a One-to-Many conditional generative adversarial network, which could learn from heterogeneous sources of images. This is achieved by training multiple generators against a discriminator in synthesized learning way. This framework supports generative models to generate images in each source, so output images follow corresponding target patterns. Two implementations, hybrid fake and cascading learning, of the synthesized adversarial training scheme are also proposed, and experimented on two benchmark datasets, UTZap50K and MVOD5K, as well as a new high-quality dataset BehTex7K. We consider five challenging image-to-image translation tasks: edges-to-photo, edges-to-similar-photo translation on UTZap50K, cross-view translation on MVOD5K, and grey-to-color, grey-to-Oil-Paint on BehTex7K. We show that both implementations are able to faithfully translate from an image to another image in edges-to-photo, edges-to-similar-photo, grey-to-color, and grey-to-Oil-Paint translation tasks. The quality of output images in cross-view translation need to be further boosted.

Appendix

1.1 Derivations and Proofs

1.1.1 Derivation of the objective of a set of generators

As for optimizing over multiple generators, Ghosh et al. [7] modified the objective function of the discriminator where along with finding the fakes, the discriminator has to find the generator that produced the given fake.

$$ \underset{D}{\max }{\mathbb{E}}_{x\sim {p}_{data}(x)}\left[\log {D}_{k+1}(x)\right]+\sum \limits_{i=1}^k{\mathbb{E}}_{x_i\sim {p}_{g_i}(x)}\left[\log {D}_i\left({x}_i\right)\right] $$

(15)

For a fixed generator, the objective is to minimize:

$$ {\mathbb{E}}_{x\sim {p}_d}\log {D}_{k+1}(x)+\sum \limits_{i=1}^k{\mathbb{E}}_{x\sim {p}_{g_i}}\log \left(1-{D}_{k+1}(x)\right) $$

(16)

For a set of generators, the objective is:

$$ \underset{G_1,{G}_2,\dots, {G}_k}{\min }{\mathbb{E}}_{x\sim {p}_{data}(x)}\left[\log {D}_{k+1}(x)\right]+\sum \limits_{i=1}^k{\mathbb{E}}_{x_i\sim {p}_{g_i}(x)}\left[\log \left(1-{D}_{k+1}(x)\right)\right] $$

(17)

where k is the number of generators, denoted as M, p_d the true data distribution, denoted as p_data, ${p}_{g_i}$ the distribution learned by the i-th generator, denoted as ${p}_{G_m}$ in this context.

Introducing the conditioned variable into (17), and replacing notations by those used in this paper, the objective of a set of conditional generators is:

$$ \underset{G_1,{G}_2,\dots, {G}_M}{\min }{\mathbb{E}}_{x,y\sim {p}_{data}\left(x,y\right)}\log {D}^{M+1}\left(x,y\right)+\sum \limits_{m=1}^M{\mathbb{E}}_{x,y\sim {p}_{G_m}\left(x,y\right)}\log \left(1-{D}^{M+1}\left(x,y\right)\right) $$

(18)

Likely, the objective of conditional discriminator is:

$$ \underset{D}{\max }{\mathbb{E}}_{x,y\sim {p}_{data}\left(x,y\right)}\log {D}^{M+1}\left(x,y\right)+\sum \limits_{m=1}^M{\mathbb{E}}_{x,y\sim {p}_{G_m}\left(x,y\right)}\log {D}^m\left(x,y\right) $$

(19)

We add L₁ regularization term to reduce blurs in image [9], yield the final objective (12), (13) and (14). Note that as for hybrid fake implementation, a hybrid instance $\mathbb{I}$ is used by discriminator, rather than a individual instance x.

1.1.2 Proofs

[7] has provided detailed propositions and theorems about the objective of training a set of generators and a discriminator for an unconditional GAN. The proofs of the One-to-Many cGAN are inspired by these propositions and theorems. We introduce conditioned variable into the optimal distribution learned by the unconditional discriminator [7], and proposed a general format of the optimal distribution learned by a conditional discriminator:

$$ {D}^m\left(x,y\right)=\frac{p_{G_m}\left(x,y\right)}{p_{data}\left(x,y\right)+\sum \limits_{m=1}^M{p}_{G_m}\left(x,y\right)},\forall m\in \left\{1,2,\dots, M+1\right\} $$

(20)

Note that the unknown ${p}_{G_{M+1}}:= {p}_{data}$ to avoid clutter [7].

Then, replacing D^m and D^M + 1 in (18) using (20), yields

$$ {\mathbb{E}}_{x,y\sim {p}_{data}\left(x,y\right)}\log \left[\frac{p_{data}\left(x,y\right)}{p_{data}\left(x,y\right)+\sum \limits_{m=1}^M{p}_{G_m}\left(x,y\right)}\right]+\sum \limits_{m=1}^M{\mathbb{E}}_{x,y\sim {p}_{G_m}\left(x,y\right)}\left[\log \left(1-\frac{p_{G_{M+1}}\left(x,y\right)}{p_{data}\left(x,y\right)+\sum \limits_{m=1}^M{p}_{G_m}\left(x,y\right)}\right)\right] $$

(21)

Using ${\sum}_{m=1}^{M+1}{D}^m=1$,

$$ {\displaystyle \begin{array}{l}{\mathbb{E}}_{x,y\sim {p}_{data}\left(x,y\right)}\log \left[\frac{p_{data}\left(x,y\right)}{p_{data}\left(x,y\right)+\sum \limits_{m=1}^M{p}_{G_m}\left(x,y\right)}\right]+\sum \limits_{m=1}^M{\mathbb{E}}_{x,y\sim {p}_{G_m}\left(x,y\right)}\left[\log \left(\frac{\sum \limits_{m=1}^M{p}_{G_m}\left(x,y\right)}{p_{data}\left(x,y\right)+\sum \limits_{m=1}^M{p}_{G_m}\left(x,y\right)}\right)\right]\\ {}:= {\mathbb{E}}_{x,y\sim {p}_{data}\left(x,y\right)}\log \left[\frac{p_{data}\left(x,y\right)}{p_{avg}\left(x,y\right)}\right]+M{\mathbb{E}}_{x,y\sim {p}_G\left(x,y\right)}\log \left[\frac{p_G\left(x,y\right)}{p_{avg}\left(x,y\right)}\right]-\left(M+1\right)\log \left(M+1\right)+M\log M\end{array}} $$

(22)

where ${p}_G=\frac{\sum_{m=1}^M{p}_{G_m}\left(x,y\right)}{M}$, ${p}_{avg}\left(x,y\right)=\frac{p_{data}(x)+{\sum}_{m=1}^M{p}_{G_m}\left(x,y\right)}{M+1}$, and ${\mathit{\sup}}_D\left({p}_G\right)={\bigcup}_{m=1}^M{\mathit{\sup}}_D\left({p}_{G_m}\right)$. The final term (22) obtains its minimum –(M + 1) log(M + 1) + M log M, when ${p}_{data}=\frac{\sum_{m=1}^M{p}_{G_m}\left(x,y\right)}{M}$ [7]. When the number of generator M is equal to 1, the One-to-Many cGAN obtains the minimum value of log 4 of the Jensen-Shannon divergence based objective function in the original GAN [8].

The convergence of ${p}_{G_m}$ can be shown by computing gradient descent update at the optimal D giving the corresponding G_m. Each ${\mathit{\sup}}_D\left({p}_{G_m},D\right)$ forms convex in ${p}_{G_m}$ with a unique global optimal value as proven in [7]. Therefore, with sufficiently small updates of ${p}_{G_m}$, ${p}_{G_m}$ converges to the corresponding p_data(x_m).

1.2 Architecture of generator and discriminator

We denote C(k) a Convolution-BatchNorm-ReLU layer with k filters, CD(k) a Convolution-BatchNorm-Dropout-ReLU layer with a dropout rate of 50%. All ReLUs in discriminator and the encoder of generator are leaky, with slop 0.2. All ReLUs in the decoder are not leaky. The generator is a modified encoder-decoder architecture called U-Net [9]:

• Encoder: C(64)-C(128)-C(256)-C(512)-C(512)-C(512)-C(512)-C(512)

• Decoder: CD(512)-CD(1024)-CD(1024)-CD(1024)-CD(1024)-C(512)-C(256)-C(128)

The discriminator is a 70 × 70 Markovian discriminator (PatchGAN) [9]: C(64)-C(128)-C(256)-C(512).

BatchNorm is not applied to the first layer C(64).