Nonblind Image Deconvolution via Leveraging Model Uncertainty in An Untrained Deep Neural Network

Abstract

Nonblind image deconvolution (NID) is about restoring the latent image with sharp details from a noisy blurred one using a known blur kernel. This paper presents a dataset-free deep learning approach for NID using untrained deep neural networks (DNNs), which does not require any external training data with ground-truth images. Based on a spatially-adaptive dropout scheme, the proposed approach learns a DNN with model uncertainty from the input blurred image, and the deconvolution result is obtained by aggregating the multiple predictions from the learned dropout DNN. It is shown that the solution approximates a minimum-mean-squared-error estimator in Bayesian inference. In addition, a self-supervised loss function for training is presented to efficiently handle the noise in blurred images. Extensive experiments show that the proposed approach not only performs noticeably better than existing non-learning-based methods and unsupervised learning-based methods, but also performs competitively against recent supervised learning-based methods.

Appendices

Proof of Proposition 1

First, we rewrite the loss function as follows.

$$\begin{aligned} \begin{aligned}&\sum _{\ell } \left\| {{\varvec{k}}}* f_{{{\varvec{\beta }}}}({\widehat{{{\varvec{y}}}}}_\ell ) -{{\varvec{y}}}\right\| _{{{\varvec{m}}}_\ell }^2 \\&\quad =\sum _{\ell }\left\| {{\varvec{k}}}* f_{{{\varvec{\beta }}}}({\widehat{{{\varvec{y}}}}}_\ell )-{{\varvec{k}}}*{{\varvec{x}}}\right\| _{{{\varvec{m}}}_\ell }^2 + \sum _{\ell }\left\| {{\varvec{n}}}\right\| _{{{\varvec{m}}}_\ell }^2 \\&\qquad - 2{{\varvec{n}}}^{\top }\big (\sum _{\ell } ({{\varvec{1}}}-{{\varvec{m}}}_\ell )\odot ({{\varvec{k}}}* f_{{{\varvec{\beta }}}}({\widehat{{{\varvec{y}}}}}_\ell )-{{\varvec{k}}}*{{\varvec{x}}})\big ). \end{aligned} \end{aligned}$$

(16)

The expectation of the second term is given by

$$\begin{aligned} \begin{aligned}&{\mathbb {E}}_{{{\varvec{n}}}}\big [\sum _{\ell }\left\| {{\varvec{n}}}\right\| _{{{\varvec{m}}}_\ell }^2\big ]= {\mathbb {E}}_{{{\varvec{n}}}}\big [\sum _{\ell }\left\| ({{\varvec{1}}}-{{\varvec{m}}}_\ell )\odot {{\varvec{n}}}\right\| _2^2\big ] \\&\quad =\sum _{\ell } \left\| ({{\varvec{1}}}-{{\varvec{m}}}_\ell ) \odot {{\varvec{\sigma }}}\right\| _2^2= \sum _{\ell } \left\| {{\varvec{\sigma }}}\right\| _{{{\varvec{m}}}_\ell }^2. \end{aligned} \end{aligned}$$

(17)

Regarding the last term, for simplicity we define

$$\begin{aligned} \begin{aligned} {{\varvec{r}}}&=\sum _{\ell }({{\varvec{1}}}-{{\varvec{m}}}_\ell )\odot ({{\varvec{k}}}* f_{{{\varvec{\beta }}}}({\widehat{{{\varvec{y}}}}}_\ell )-{{\varvec{k}}}*{{\varvec{x}}})\\&=\sum _{\ell }({{\varvec{1}}}-{{\varvec{m}}}_\ell )\odot ({{\varvec{k}}}* f_{{{\varvec{\beta }}}}({{\varvec{m}}}_\ell \odot ({{\varvec{k}}}*{{\varvec{x}}})+{{\varvec{m}}}_\ell \odot {{\varvec{n}}}\\&\qquad +({{\varvec{1}}}-{{\varvec{m}}}_\ell )\odot (\mathcal {A}\circ ({{\varvec{m}}}_\ell \odot {{\varvec{y}}})))-{{\varvec{k}}}*{{\varvec{x}}}). \end{aligned} \end{aligned}$$

(18)

It can be seen that ${{\varvec{k}}}* f_{{{\varvec{\beta }}}}({{\varvec{m}}}_\ell \odot ({{\varvec{k}}}*{{\varvec{x}}})+{{\varvec{m}}}_\ell \odot {{\varvec{n}}}+({{\varvec{1}}}-{{\varvec{m}}}_\ell )\odot (\mathcal {A}\circ ({{\varvec{m}}}_\ell \odot {{\varvec{y}}})))$ contributes to ${{\varvec{r}}}(i)$ only if ${{\varvec{m}}}_\ell (i)=0$. But in this case, ${{\varvec{n}}}(i)$ is erased by ${{\varvec{m}}}_\ell (i)$. This means that ${{\varvec{n}}}(i)$ has no contribution to ${{\varvec{r}}}(i)$. Together with that ${{\varvec{n}}}(i)$ is independent of ${{\varvec{n}}}(j)$ for any $i\ne j$, It is concluded that ${{\varvec{r}}}(i)$ is independent to ${{\varvec{n}}}(i)$ for all i. Therefore, we have

$$\begin{aligned} {\mathbb {E}}_{{{\varvec{n}}}} \big [{{\varvec{n}}}^{\top }{{\varvec{r}}}\big ] = ({\mathbb {E}}_{{{\varvec{n}}}} \big [{{\varvec{n}}}\big ])^\top ({\mathbb {E}}_{{{\varvec{n}}}} \big [{{\varvec{r}}}\big ]) = 0. \end{aligned}$$

(19)

Combining (16), (17) and (19) gives that

$$\begin{aligned} \begin{aligned}&{\mathbb {E}}_{{{\varvec{n}}}}[\sum _{\ell } \Vert (1-{{\varvec{m}}}_\ell )\odot ({{\varvec{k}}}* f_{{{\varvec{\beta }}}}({\widehat{{{\varvec{y}}}}}_\ell )-{{\varvec{y}}})\Vert _2^2]\\&\quad =\sum _{\ell }\left\| {{\varvec{k}}}* f_{{{\varvec{\beta }}}}({\widehat{{{\varvec{y}}}}})-{{\varvec{k}}}*{{\varvec{x}}}\right\| _{{{\varvec{m}}}_\ell }^2 + \sum _{\ell } \left\| {{\varvec{\sigma }}}\right\| _{{{\varvec{m}}}_\ell }^2. \end{aligned} \end{aligned}$$

(20)

The proof is done.

MMSE Approximation

The derivation is based on the theory of variational inference; please see (Blei et al. 2017) for a comprehensive introduction. First, we take $p({{\varvec{\beta }}})$ as a uniform distribution on a sufficiently large bounded set $\mathbb {S}=[-C/2,C/2]^d$, where C is a sufficient large positive number and d is the dimensionality of ${{\varvec{\beta }}}$. Additionally, the entries of the noise ${{\varvec{n}}}$ are assumed to be i.i.d. variables following the Gaussian distribution of mean zero and variance $\sigma ^2$. The KL divergence between $p({{\varvec{\beta }}}\vert \mathcal {D})$ and $q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})$ is given by

$$\begin{aligned} \begin{aligned}&\mathrm{KL}(q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})\Vert p({{\varvec{\beta }}}\vert \mathcal {D})) \\&\quad =\mathbb {E}_{ q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})} \log q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})-\mathbb {E}_{q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})} \log p({{\varvec{\beta }}}\vert \mathcal {D})\\&\quad =\mathbb {E}_{ q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})} \log q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})-\mathbb {E}_{q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})} \big ( \log p({{\varvec{\beta }}})\\&\qquad +\log p(\mathcal {D}\vert {{\varvec{\beta }}})-\log p(\mathcal {D}) \big )\\&\quad = \mathrm{KL}(q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})\Vert p({{\varvec{\beta }}})) - {\mathbb {E}}_{{{\varvec{\beta }}}\sim q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})}\log p(\mathcal {D}\vert {{\varvec{\beta }}})\\&\qquad + {\mathbb {E}}_{{{\varvec{\beta }}}\sim q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})}\log p(\mathcal {D}). \end{aligned} \end{aligned}$$

(21)

Since $\log p(\mathcal {D})$ is irrelevant to the model parameter ${{\varvec{\beta }}}$, we have ${\mathbb {E}}_{{{\varvec{\beta }}}\sim q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})}\log p(\mathcal {D}) = \log p(\mathcal {D})$ which is a constant. Therefore, we have

$$\begin{aligned} \begin{aligned}&\mathrm{KL}(q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})\Vert p({{\varvec{\beta }}}\vert \mathcal {D})) \\&\quad = \mathrm{KL}(q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})\Vert p({{\varvec{\beta }}})) \\&\qquad - {\mathbb {E}}_{{{\varvec{\beta }}}\sim q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})}\log p(\mathcal {D}\vert {{\varvec{\beta }}}) + \text{ const }., \end{aligned} \end{aligned}$$

(22)

where the KL divergence between $q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})$ and $p({{\varvec{\beta }}})$ can be written as

$$\begin{aligned} \begin{aligned}&\mathrm{KL}(q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})\Vert p({{\varvec{\beta }}})) \\&\quad = \mathbb {E}_{q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})} \log q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})-\mathbb {E}_{q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})} \log p({{\varvec{\beta }}}). \end{aligned} \end{aligned}$$

(23)

The first term on the right hand of the above equation is the negative entropy of $q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})$. Recall that $q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})$ is the probability of Bernoulli:

$$\begin{aligned} {{\varvec{\beta }}}= {{\varvec{\theta }}}\odot {{\varvec{b}}}, \text{ where } \ {{\varvec{b}}}(i) \sim \mathrm{Bernoulli}(p_i). \end{aligned}$$

(24)

Its entropy is given by

$$\begin{aligned} \begin{aligned}&- \mathbb {E}_{q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})} \log q({{\varvec{\beta }}}\vert {{\varvec{\theta }}}) \\&\quad =-\prod _i \big ( p_i\log p_i+ (1-p_i)\log (1-p_i) \big ), \end{aligned} \end{aligned}$$

(25)

which is a constant irrelevant to the parameter ${{\varvec{\theta }}}$ and can be ignored during training. Since $p({{\varvec{\beta }}})=0$ outside $\mathbb {S}$, $\log p({{\varvec{\beta }}}) = -\infty$ outside $\mathbb {S}$. Then $\int q({{\varvec{\beta }}}\vert {{\varvec{\theta }}}) \log p({{\varvec{\beta }}})=-\infty$ if $\int _{\mathbb {R}^d\setminus \mathbb {S}} q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})\ne 0$ (which means that ${{\varvec{\theta }}}\notin \mathbb {S}$). When ${{\varvec{\theta }}}\in \mathbb {S}$, we have $\int _{\mathbb {S}}q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})=1$ and thus

$$\begin{aligned} \begin{aligned}&\mathbb {E}_{q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})} \log p({{\varvec{\beta }}}) \\&\quad = \int q({{\varvec{\beta }}}\vert {{\varvec{\theta }}}) \log p({{\varvec{\beta }}}) = \int _{\mathbb {S}} q({{\varvec{\beta }}}\vert {{\varvec{\theta }}}) \log \frac{1}{C^d} \\&\quad = \log \frac{1}{C^d}. \end{aligned} \end{aligned}$$

(26)

It yields that

$$\begin{aligned} \int q({{\varvec{\beta }}}\vert {{\varvec{\theta }}}) \log p({{\varvec{\beta }}}) =\left\{ \begin{array}{ll} \log \frac{1}{C^d}, &{}{{\varvec{\theta }}}\in \mathbb {S}\\ -\infty , &{}{{\varvec{\theta }}}\notin \mathbb {S}. \end{array}\right. \end{aligned}$$

(27)

Finally, we obtained that

$$\begin{aligned} \mathrm{KL}(q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})\Vert p({{\varvec{\beta }}})) = \delta _{\mathbb {S}}({{\varvec{\theta }}}) +\text{ const }, \end{aligned}$$

(28)

where

$$\begin{aligned} \delta _{\mathbb {S}}({{\varvec{\theta }}}) = \left\{ \begin{array}{ll} 0, &{}\text{ if } {{\varvec{\theta }}}\in \mathbb {S},\\ +\infty , &{}\text{ otherwise }. \end{array}\right. \end{aligned}$$

(29)

Recall that the samples in $\mathcal {D}=\{{\widehat{{{\varvec{y}}}}}_\ell ,({{\varvec{1}}}-{{\varvec{m}}}_\ell )\odot {{\varvec{y}}}\}_\ell$ are related by

$$\begin{aligned} ({\varvec{1}}-m_{\ell }) \odot {{\varvec{y}}}= ({\varvec{1}}-m_{\ell }) \odot ({{\varvec{k}}}* f_{{{\varvec{\beta }}}}(\widehat{{{\varvec{y}}}_{\ell }}) + {{\varvec{n}}}). \end{aligned}$$

(30)

Roughly, we assume these samples are independent from each other. Then we can obtain

$$\begin{aligned} \begin{aligned}&{\mathbb {E}}_{{{\varvec{\beta }}}\sim q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})}\log p(\mathcal {D}\vert {{\varvec{\beta }}}) \\&\quad = {\mathbb {E}}_{{{\varvec{\beta }}}\sim q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})} \sum _{{\widehat{{{\varvec{y}}}}}_\ell \sim \varvec{\Omega }} \log p(({{\varvec{1}}}-{{\varvec{m}}}_\ell )\odot {{\varvec{y}}}\vert {{\varvec{\beta }}},{\widehat{{{\varvec{y}}}}}_\ell ) \\&\qquad +{\mathbb {E}}_{{{\varvec{\beta }}}\sim q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})} \sum _{{\widehat{{{\varvec{y}}}}}_\ell \sim \varvec{\Omega }} \log p({\widehat{{{\varvec{y}}}}}_\ell \vert {{\varvec{\beta }}}) \\&\quad = -\frac{1}{2\sigma ^2}\mathbb {E}_{{{\varvec{b}}}}\sum _{{\widehat{{{\varvec{y}}}}}\sim \varvec{\Omega }}\ {{\mathcal {C}}}({{\varvec{y}}},{{\varvec{k}}}* f_{{{\varvec{\theta }}}\odot {{\varvec{b}}}}({{\widehat{{{\varvec{y}}}}}})) + \text{ const. } \end{aligned} \end{aligned}$$

(31)

Finally, we have

$$\begin{aligned} \begin{aligned}&\mathrm{KL}(q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})\Vert p({{\varvec{\beta }}}\vert \mathcal {D})) \\&\quad = \frac{1}{2\sigma ^2}\mathbb {E}_{{{\varvec{b}}}}\sum _{{\widehat{{{\varvec{y}}}}}\sim \varvec{\Omega }}\ {{\mathcal {C}}}({{\varvec{y}}},{{\varvec{k}}}* f_{{{\varvec{\theta }}}\odot {{\varvec{b}}}}({{\widehat{{{\varvec{y}}}}}})) + \delta _{\mathbb {S}}({{\varvec{\theta }}}) + \text{ const. } \end{aligned} \end{aligned}$$

(32)

Thus, minimizing the KL divergence between $p({{\varvec{\beta }}}\vert \mathcal {D})$ and $q({{\varvec{\beta }}}\vert {{\varvec{\theta }}})$ is equivalent to

$$\begin{aligned} \min _{{{\varvec{\theta }}}\in \mathbb {S}} \mathbb {E}_{{\widehat{{{\varvec{y}}}}}\sim \varvec{\Omega }}\mathbb {E}_{{{\varvec{b}}}}\ {{\mathcal {C}}}({{\varvec{y}}},{{\varvec{k}}}* f_{{{\varvec{\theta }}}\odot {{\varvec{b}}}}({{\widehat{{{\varvec{y}}}}}})). \end{aligned}$$

(33)

Since the feasible set $\mathbb {S}$ is sufficiently large, the constraint ${{\varvec{\theta }}}\in \mathbb {S}$ can be omitted in practice, which results in the our training loss in (5).