A novel multi-discriminator deep network for image segmentation

Abstract

Several studies have shown the excellent performance of deep learning in image segmentation. Usually, this benefits from a large amount of annotated data. Medical image segmentation is challenging, however, since there is always a scarcity of annotated data. This study constructs a novel deep network for medical image segmentation, referred to as asymmetric U-Net generative adversarial networks with multi-discriminators (AU-MultiGAN). Specifically, the asymmetric U-Net is designed to produce multiple segmentation maps simultaneously and use the dual-dilated blocks in the feature extraction stage only. Further, the multi-discriminator module is embedded into the asymmetric U-Net structure, which can capture the available information of samples sufficiently and thereby promote the information transmission of features. A hybrid loss by the combination of segmentation and discriminator losses is developed, and an adaptive method of selecting the scale factors is devised for this new loss. More importantly, the convergence of the proposed model is proved mathematically. The proposed AU-MultiGAN approach is implemented on some standard medical image benchmarks. Experimental results show that the proposed architecture can be successfully applied to medical image segmentation, and obtain superior performance in comparison with the state-of-the-art baselines.

Appendix A

1.1 A.1 Proof of Lemma 1

For a given generator G_i, the training criterion for the sub-discriminator D_i(i = 1, 2,…,k) is to minimize the discriminator loss, L_GAN(G_i,D_i). Let

$$ \begin{array}{@{}rcl@{}} && L_{\text{GAN}}(G_{i},D_{i})\\ &&= \mathbb{E}_{\boldsymbol{Y}_{i}\sim p_{_{\boldsymbol{Y}_{i}}}}\left[\frac{1}{2}(D_{i}(\boldsymbol{Y}_{i})-1)^{2}\right]+\mathbb{E}_{\boldsymbol{X}\sim p_{_{\boldsymbol{X}}}}\left[\frac{1}{2}{D_{i}^{2}}(G_{i}(\boldsymbol{X}))\right]\\ &&= \mathbb{E}_{\boldsymbol{Y}_{i}\sim p_{_{\boldsymbol{Y}_{i}}}}[\frac{1}{2}(D_{i}(\boldsymbol{Y}_{i})-1)^{2}]+\mathbb{E}_{\boldsymbol{Y}^{\prime}_{i}\sim p_{g_{i}}}\left[\frac{1}{2}{D_{i}^{2}}(\boldsymbol{Y}^{\prime}_{i})\right]\\ &&= \frac{1}{2}{\int}_{\boldsymbol{Y}_{i}}p_{_{\boldsymbol{Y}_{i}}}(\boldsymbol{Y}_{i})(D_{i}(\boldsymbol{Y}_{i})-1)^{2}+ p_{g_{i}}(\boldsymbol{Y}_{i}){D_{i}^{2}}(\boldsymbol{Y}_{i})d\boldsymbol{Y}_{i}. \end{array} $$

(22)

By the formula in (22), the optimisation problem of the sub-discriminator can be transformed into a least squares problem:

$$ \underset{D_{i}}{\min} (D_{i}(\boldsymbol{Y}_{i})-1)^{2}p_{_{\boldsymbol{Y}_{i}}}(\boldsymbol{Y}_{i})+{D_{i}^{2}}(\boldsymbol{Y}_{i})p_{g_{i}}(\boldsymbol{Y}_{i}). $$

(23)

It achieves the minimum at $\frac {p_{_{\boldsymbol {Y}_{i}}}(\boldsymbol {Y}_{i})}{p_{_{\boldsymbol {Y}_{i}}}(\boldsymbol {Y}_{i})+p_{g_{i}}(\boldsymbol {Y}_{i})} (i=1,2,\ldots ,k)$ in [0, 1]. This completes the proof.

1.2 A.2 Proof of Lemma 2

For all i(i = 1, 2,…,k), if the relation $p_{g_{i}}=p_{_{\boldsymbol {Y}_{i}}}$ is satisfied, then $D_{i}^{*}=\frac {1}{2}$ is calculated by (16). Hence,

$$ \begin{array}{@{}rcl@{}} && \underset{D_{i}}{\min} L_{\text{GAN}}(G_{i},D_{i})\\ && = \mathbb{E}_{\boldsymbol{Y}_{i}\sim p_{_{\boldsymbol{Y}_{i}}}}[\frac{1}{2}(D_{i}^{*}(\boldsymbol{Y}_{i})-1)^{2}]+\mathbb{E}_{\boldsymbol{X}\sim p_{_{\boldsymbol{X}}}}[\frac{1}{2}D_{i}^{*2}(G_{i}(\boldsymbol{X}))]\\ && = \mathbb{E}_{\boldsymbol{Y}_{i}\sim p_{_{\boldsymbol{Y}_{i}}}}[\frac{1}{2}(D_{i}^{*}(\boldsymbol{Y}_{i})-1)^{2}]+\mathbb{E}_{\boldsymbol{Y}^{\prime}_{i}\sim p_{g_{i}}}[\frac{1}{2}D_{i}^{*2}(\boldsymbol{Y}^{\prime}_{i})]\\ && = \frac{1}{4}. \end{array} $$

(24)

To see that $\frac {1}{4}$ is the best possible value of $\underset {D_{i}}{\min \limits } L_{\text {GAN}}(G_{i},D_{i})$, reached only for $p_{g_{i}}=p_{_{\boldsymbol {Y}_{i}}}$, we observe

$$ \begin{array}{@{}rcl@{}} && \underset{D_{i}}{\min} L_{\text{GAN}}(G_{i},D_{i})\\ &&= \mathbb{E}_{\boldsymbol{Y}_{i}\sim p_{_{\boldsymbol{Y}_{i}}}}[\frac{1}{2}(D_{i}^{*}(\boldsymbol{Y}_{i})-1)^{2}]+\mathbb{E}_{\boldsymbol{X}\sim p_{_{\boldsymbol{X}}}}[\frac{1}{2}D_{i}^{*2}(G_{i}(\boldsymbol{X}))]\\ && = \mathbb{E}_{\boldsymbol{Y}_{i}\sim p_{_{\boldsymbol{Y}_{i}}}}[\frac{1}{2}(D_{i}^{*}(\boldsymbol{Y}_{i})-1)^{2}]+\mathbb{E}_{\boldsymbol{Y}^{\prime}_{i}\sim p_{g_{i}}}[\frac{1}{2}D_{i}^{*2}(\boldsymbol{Y}^{\prime}_{i})]\\ && = \frac{1}{2}{\int}_{\boldsymbol{Y}_{i}}[p_{_{\boldsymbol{Y}_{i}}}(\boldsymbol{Y}_{i})(D_{i}^{*}(\boldsymbol{Y}_{i})-1)^{2}+ p_{g_{i}}(\boldsymbol{Y}_{i})D_{i}^{*2}(\boldsymbol{Y}_{i})]d\boldsymbol{Y}_{i},\\ \end{array} $$

(25)

where the relationship between $D_{i}^{*2}(\boldsymbol {Y}_{i})$, the label distribution $p_{_{\boldsymbol {Y}_{i}}}$ and the generated segmentation distribution $p_{g_{i}}$ are obtained in Lemma 1. Here, we introduce this relationship into the above formula.

$$ \begin{array}{@{}rcl@{}} \!\!\!\!\!&& \frac{1}{2}{\int}_{\boldsymbol{Y}_{i}}\!\left[\!\frac{p_{_{\boldsymbol{Y}_{i}}}(\boldsymbol{Y}_{i})p_{g_{i}}^{2}(\boldsymbol{Y}_{i})}{(p_{_{\boldsymbol{Y}_{i}}}(\boldsymbol{Y}_{i})+p_{g_{i}}(\boldsymbol{Y}_{i}))^{2}}+\frac{p_{g_{i}}(\boldsymbol{Y}_{i})p_{_{\boldsymbol{Y}_{i}}}^{2}(\boldsymbol{Y}_{i})}{(p_{_{\boldsymbol{Y}_{i}}}(\boldsymbol{Y}_{i})+p_{g_{i}}(\boldsymbol{Y}_{i}))^{2}}\!\right]d\boldsymbol{Y}_{i}\\ \!\!\!\!\!&& = \frac{1}{4} + \frac{1}{8}{\int}_{\boldsymbol{Y}_{i}} \!\!\left[\!\frac{p_{_{\boldsymbol{Y}_{i}}}\!(\boldsymbol{Y}_{i})p_{g_{i}}\!(\boldsymbol{Y}_{i}\!) - p_{_{\boldsymbol{Y}_{i}}}^{2}\!(\boldsymbol{Y}_{i}\!)}{p_{_{\boldsymbol{Y}_{i}}}\!(\boldsymbol{Y}_{i})+p_{g_{i}}\!(\boldsymbol{Y}_{i})} + \frac{p_{_{\boldsymbol{Y}_{i}}}\!(\boldsymbol{Y}_{i}\!)p_{g_{i}}(\boldsymbol{Y}_{i}\!) - p_{g_{i}}^{2}\!(\boldsymbol{Y}_{i}\!)}{p_{_{\boldsymbol{Y}_{i}}}\!(\boldsymbol{Y}_{i}) + p_{g_{i}}\!(\boldsymbol{Y}_{i})}\!\right]\!d\boldsymbol{Y}_{i}\\ \!\!\!\!\!&& = \frac{1}{4}-\frac{1}{8}{\int}_{\boldsymbol{Y}_{i}} \!\left[\!\frac{(2p_{_{\boldsymbol{Y}_{i}}}(\boldsymbol{Y}_{i})-(p_{_{\boldsymbol{Y}_{i}}}(\boldsymbol{Y}_{i})+p_{g_{i}}(\boldsymbol{Y}_{i})))^{2\!}}{p_{_{\boldsymbol{Y}_{i}}}(\boldsymbol{Y}_{i})+p_{g_{i}}(\boldsymbol{Y}_{i})}\right]d\boldsymbol{Y}_{i}\\ \!\!\!\!\!&& = \frac{1}{4}-\frac{1}{8}\chi^{2}(p_{_{\boldsymbol{Y}_{i}}}+p_{g_{i}}\|2p_{_{\boldsymbol{Y}_{i}}}), \end{array} $$

(26)

where χ² is the Pearson χ² divergence and $\chi ^{2}(p_{_{\boldsymbol {Y}_{i}}}$ $+p_{g_{i}}\|2p_{_{\boldsymbol {Y}_{i}}})$ denotes the simplified representation of ${\int \limits }_{\boldsymbol {Y}_{i}} \left [\frac {(2p_{_{\boldsymbol {Y}_{i}}}(\boldsymbol {Y}_{i}) - (p_{_{\boldsymbol {Y}_{i}}}(\boldsymbol {Y}_{i}) + p_{g_{i}}(\boldsymbol {Y}_{i})))^{2}}{p_{_{\boldsymbol {Y}_{i}}} (\boldsymbol {Y}_{i}) + p_{g_{i}}(\boldsymbol {Y}_{i})}\right ]d\boldsymbol {Y}_{i}$. Thus, the results of (26) achieve the value $\frac {1}{4}$ when $p_{_{\boldsymbol {Y}_{i}}}$ and $p_{g_{i}}$ are equal. We have shown that $L_{\text {GAN}}=\frac {1}{4}$ and that the only solution is $p_{g_{i}}=p_{_{\boldsymbol {Y}_{i}}}$. Thus, the asymmetric U-Net can perfectly replicate the distribution of the real segmented image. This completes the proof.

1.3 A.3 Proof of Theorem 1

Consider $L_{\text {GAN}}(G_{i},D_{i})=\mathrm {V}(p_{g_{i}},D_{i})$ as a function of $p_{g_{i}}$, as done in the above criterion (17), in which $\mathrm {V}(p_{g_{i}},D_{i})$ is the criterion. Note that $\mathrm {V}(p_{g_{i}},D_{i})$ is convex on $p_{g_{i}}$. The inf-derivatives of an infimum of convex functions are the derivative of the function at the point where the minimum is attained. This is equivalent to computing a gradient descent update for $p_{g_{i}}$ at the optimal D_i, given the corresponding G_i. From [45], $\inf _{D_{i}}L_{\text {GAN}}(G_{i},D_{i})$ is convex on $p_{g_{i}}$. Moreover, $\inf _{D_{i}}L_{\text {GAN}}(G_{i},D_{i})$ takes the value $\frac {1}{4}$ as proven in Lemma 2; therefore, with sufficiently small updates of $p_{g_{i}}$, $p_{g_{i}}$ converges to $p_{_{\boldsymbol {Y}_{i}}}(i=1,2,\ldots ,k)$. This completes the proof.