Iterative scheme-inspired network for impulse noise removal

Abstract

This paper presents a supervised data-driven algorithm for impulse noise removal via iterative scheme-inspired network (IIN). IIN is defined over a data flow graph, which is derived from the iterative procedures in Alternating Direction Method of Multipliers (ADMM) algorithm for optimizing the L1-guided variational model. In the training phase, the L1-minimization is reformulated into an augmented Lagrangian scheme through adding a new auxiliary variable. In the testing phase, it has computational overhead similar to ADMM but uses optimized parameters learned from the training data for restoration task. Experimental results demonstrate that the newly proposed method can obtain very significantly superior performance than current state-of-the-art variational and dictionary learning-based approaches for salt-and-pepper noise removal.

Appendix

The detailed training procedures of IIN are as follows:

• 1. Multiplier update layer (${\mathbf{M}}^{{\left( {\mathbf{n}} \right)}}$):

This layer has three sets of inputs: $\left\{ {\beta_{l}^{(n - 1)} } \right\},\left\{ {c_{l}^{(n)} } \right\}$ and {z⁽ⁿ⁾_l}. Its output $\left\{ {\beta_{l}^{(n)} } \right\}$ is the input to compute $\left\{ {\beta_{l}^{(n + 1)} } \right\},\left\{ {z_{l}^{(n + 1)} } \right\}$ and $\left\{ {x_{l}^{(n + 1)} } \right\}$. The parameters in this layer are $\left\{ {\eta_{l}^{(n)} } \right\},\;l = \left[ {1,2, \ldots ,L} \right]$. The gradients of loss w.r.t. the parameters can be computed as:

$$\frac{\partial E}{{\partial \eta_{l}^{\left( n \right)} }} = \frac{\partial E}{{\partial \beta_{l}^{\left( n \right)} }}\frac{{\partial \beta_{l}^{\left( n \right)} }}{{\partial \eta_{l}^{\left( n \right)} }},\quad {\text{where}}\;\frac{\partial E}{{\partial \beta_{l}^{\left( n \right)} }} = \frac{\partial E}{{\partial \beta_{l}^{{\left( {n + 1} \right)}} }}\frac{{\partial \beta_{l}^{{\left( {n + 1} \right)}} }}{{\partial \beta_{l}^{\left( n \right)} }} + \frac{\partial E}{{\partial z_{l}^{{\left( {n + 1} \right)}} }}\frac{{\partial z_{l}^{{\left( {n + 1} \right)}} }}{{\partial \beta_{l}^{\left( n \right)} }} + \frac{\partial E}{{\partial x^{{\left( {n + 1} \right)}} }}\frac{{\partial x^{{\left( {n + 1} \right)}} }}{{\partial \beta_{l}^{\left( n \right)} }}.$$

We also compute gradients of the output in this layer w.r.t. its inputs: $\frac{\partial E}{{\partial \beta_{l}^{\left( n \right)} }}\frac{{\partial \beta_{l}^{\left( n \right)} }}{{\partial \beta_{l}^{{\left( {n - 1} \right)}} }},\frac{\partial E}{{\partial \beta_{l}^{\left( n \right)} }}\frac{{\partial \beta_{l}^{\left( n \right)} }}{{\partial C_{l}^{\left( n \right)} }},\frac{\partial E}{{\partial \beta_{l}^{\left( n \right)} }}\frac{{\partial \beta_{l}^{\left( n \right)} }}{{\partial z_{l}^{\left( n \right)} }}$.

• 2. Multiplier update layer (${\mathbf{P}}^{{\left( {\mathbf{n}} \right)}}$):

This layer has three sets of inputs: {p⁽ⁿ⁻¹⁾}, {x⁽ⁿ⁾} and {t⁽ⁿ⁾}. Its output {p⁽ⁿ⁾} is the input to compute {t⁽ⁿ⁺¹⁾}, {p⁽ⁿ⁺¹⁾} and {x⁽ⁿ⁺¹⁾}. The parameter in this layer is {τ⁽ⁿ⁾}. The gradients of loss w.r.t. the parameters can be computed as:

$$\frac{\partial E}{{\partial \tau^{\left( n \right)} }} = \frac{\partial E}{{\partial p^{\left( n \right)} }}\frac{{\partial p^{\left( n \right)} }}{{\partial \tau^{\left( n \right)} }},\quad {\text{where}}\;\frac{\partial E}{{\partial p^{\left( n \right)} }} = \frac{\partial E}{{\partial t^{{\left( {n + 1} \right)}} }}\frac{{\partial t^{{\left( {n + 1} \right)}} }}{{\partial p^{\left( n \right)} }} + \frac{\partial E}{{\partial p^{{\left( {n + 1} \right)}} }}\frac{{\partial p^{{\left( {n + 1} \right)}} }}{{\partial p^{\left( n \right)} }} + \frac{\partial E}{{\partial x^{{\left( {n + 1} \right)}} }}\frac{{\partial x^{{\left( {n + 1} \right)}} }}{{\partial p^{\left( n \right)} }}.$$

The gradient of layer output w.r.t. input is computed as $\frac{\partial E}{{\partial p^{\left( n \right)} }}\frac{{\partial p^{\left( n \right)} }}{{\partial p^{{\left( {n - 1} \right)}} }},\frac{\partial E}{{\partial p^{\left( n \right)} }}\frac{{\partial p^{\left( n \right)} }}{{\partial t^{\left( n \right)} }},\frac{\partial E}{{\partial p^{\left( n \right)} }}\frac{{\partial p^{\left( n \right)} }}{{\partial x^{\left( n \right)} }}$ .

• 3. Nonlinear transform layer (${\mathbf{T}}^{{\left( {\mathbf{n}} \right)}}$):

This layer has two sets of inputs: {p⁽ⁿ⁻¹⁾}, {x⁽ⁿ⁾}, and its output {t⁽ⁿ⁾} is the input to compute {p⁽ⁿ⁾}, {x⁽ⁿ⁺¹⁾}. The parameter of this layer is $\left\{ {V_{i}^{(n)} } \right\}_{i = 1}^{{N_{c} }}$. The gradients of loss w.r.t. the parameters can be computed as:

$$\frac{\partial E}{{\partial V_{i}^{\left( n \right)} }} = \frac{\partial E}{{\partial t^{\left( n \right)} }}\frac{{\partial t^{\left( n \right)} }}{{\partial V_{i}^{\left( n \right)} }},\quad {\text{where}}\;\frac{\partial E}{{\partial t^{\left( n \right)} }} = \frac{\partial E}{{\partial p^{\left( n \right)} }}\frac{{\partial p^{\left( n \right)} }}{{\partial t^{\left( n \right)} }} + \frac{\partial E}{{\partial x^{{\left( {n + 1} \right)}} }}\frac{{\partial x^{{\left( {n + 1} \right)}} }}{{\partial t^{\left( n \right)} }}.$$

The gradient of layer output w.r.t. input is computed as $\frac{\partial E}{{\partial t^{\left( n \right)} }}\frac{{\partial t^{\left( n \right)} }}{{\partial x^{\left( n \right)} }},\frac{\partial E}{{\partial t^{\left( n \right)} }}\frac{{\partial t^{\left( n \right)} }}{{\partial p^{{\left( {n - 1} \right)}} }}$.

• 4. Nonlinear transform layer (${\mathbf{Z}}^{{\left( {\mathbf{n}} \right)}}$):

This layer has two sets of inputs: {β⁽ⁿ⁻¹⁾_l} and {c⁽ⁿ⁾_l}, and its output {z⁽ⁿ⁾_l} is the input to compute {β⁽ⁿ⁾_l} and {x⁽ⁿ⁺¹⁾}. The parameters of this layer are $\left\{ {h_{l,i}^{(n)} } \right\}_{i = 1}^{{N_{c} }} ,\;l = \left[ {1,2, \ldots ,L} \right]$. The gradients of loss w.r.t. the parameters can be computed as:

$$\frac{\partial E}{{\partial h_{l,i}^{\left( n \right)} }} = \frac{\partial E}{{\partial z_{l}^{\left( n \right)} }}\frac{{\partial z_{l}^{\left( n \right)} }}{{\partial h_{l,i}^{\left( n \right)} }},\quad {\text{where}}\;\frac{\partial E}{{\partial z_{l}^{\left( n \right)} }} = \frac{\partial E}{{\partial \beta_{l}^{\left( n \right)} }}\frac{{\partial \beta_{l}^{\left( n \right)} }}{{\partial z_{l}^{\left( n \right)} }} + \frac{\partial E}{{\partial x^{{\left( {n + 1} \right)}} }}\frac{{\partial x^{{\left( {n + 1} \right)}} }}{{\partial z_{l}^{\left( n \right)} }}.$$

The gradient of layer output w.r.t. input is computed as $\frac{\partial E}{{\partial z_{l}^{\left( n \right)} }}\frac{{\partial z_{l}^{\left( n \right)} }}{{\partial \beta_{l}^{{\left( {n - 1} \right)}} }},\frac{\partial E}{{\partial z_{l}^{\left( n \right)} }}\frac{{\partial z_{l}^{\left( n \right)} }}{{\partial C_{l}^{\left( n \right)} }}$.

• 5. Convolution layer (${\mathbf{C}}^{{\left( {\mathbf{n}} \right)}}$):

The parameters of this layer are {D⁽ⁿ⁾_l}. We represent the filter by $D_{l}^{\left( n \right)} = \sum\nolimits_{m = 1}^{t} {\omega_{l,m}^{(n)} B_{m} }$, where {B_m} is a basis element and {ω⁽ⁿ⁾_l,m} is the set of filter coefficients to be learned. The gradients of loss w.r.t. filter coefficients are computed as:

$$\frac{\partial E}{{\partial \omega_{l,m}^{\left( n \right)} }} = \frac{\partial E}{{\partial c_{l}^{\left( n \right)} }}\frac{{\partial c_{l}^{\left( n \right)} }}{{\partial \omega_{l,m}^{\left( n \right)} }},\quad {\text{where}}\;\frac{\partial E}{{\partial c_{l}^{\left( n \right)} }} = \frac{\partial E}{{\partial \beta_{l}^{\left( n \right)} }}\frac{{\partial \beta_{l}^{\left( n \right)} }}{{\partial c_{l}^{\left( n \right)} }} + \frac{\partial E}{{\partial z_{l}^{\left( n \right)} }}\frac{{\partial z_{l}^{\left( n \right)} }}{{\partial c_{l}^{\left( n \right)} }}.$$

The gradient of layer output w.r.t. input is computed as $\frac{\partial E}{{\partial c_{l}^{\left( n \right)} }}\frac{{\partial c_{l}^{\left( n \right)} }}{{\partial x^{\left( n \right)} }}$.

• 6. Reconstruction layer (${\mathbf{X}}^{{\left( {\mathbf{n}} \right)}}$):

The parameters of this layer are {H⁽ⁿ⁾_l}, {ρ⁽ⁿ⁾_l} and {ɛ⁽ⁿ⁾}. Similar to convolution layer, we represent the filter by $H_{l}^{\left( n \right)} = \sum\nolimits_{m = 1}^{r} {\gamma_{l,m}^{(n)} B_{m} }$. The gradient of layer output w.r.t. input is computed as:

$$\frac{\partial E}{{\partial \gamma_{l,m}^{\left( n \right)} }} = \frac{\partial E}{{\partial x^{\left( n \right)} }}\frac{{\partial x^{\left( n \right)} }}{{\partial \gamma_{l,m}^{\left( n \right)} }},\frac{\partial E}{{\partial \rho_{l}^{\left( n \right)} }} = \frac{\partial E}{{\partial x^{\left( n \right)} }}\frac{{\partial x^{\left( n \right)} }}{{\partial \rho_{l}^{\left( n \right)} }},\frac{\partial E}{{\partial \varepsilon^{\left( n \right)} }} = \frac{\partial E}{{\partial x^{\left( n \right)} }}\frac{{\partial x^{\left( n \right)} }}{{\partial \varepsilon^{\left( n \right)} }},$$

where $\frac{\partial E}{{\partial x^{\left( n \right)} }} = \frac{\partial E}{{\partial c_{l}^{\left( n \right)} }}\frac{{\partial c_{l}^{\left( n \right)} }}{{\partial x^{\left( n \right)} }} + \frac{\partial E}{{\partial t^{\left( n \right)} }}\frac{{\partial t^{\left( n \right)} }}{{\partial x^{\left( n \right)} }} + \frac{\partial E}{{\partial p^{\left( n \right)} }}\frac{{\partial p^{\left( n \right)} }}{{\partial x^{\left( n \right)} }},$ if n ≤ N_s; $\frac{\partial E}{{\partial x^{\left( n \right)} }} = \frac{1}{\left| \psi \right|}\frac{{\left( {x^{\left( n \right)} - x^{gt} } \right)}}{{\sqrt {\left\| {x^{\left( n \right)} - x^{\text{gt}} } \right\|_{2}^{2} } \sqrt {\left\| {x^{\text{gt}} } \right\|_{2}^{2} } }}$, if n = N_s + 1.

The gradient of layer output w.r.t. input is computed as $\frac{\partial E}{{\partial x^{\left( n \right)} }}\frac{{\partial x^{\left( n \right)} }}{{\partial z_{l}^{{\left( {n - 1} \right)}} }},\frac{\partial E}{{\partial x^{\left( n \right)} }}\frac{{\partial x^{\left( n \right)} }}{{\partial \beta_{l}^{{\left( {n - 1} \right)}} }},\frac{\partial E}{{\partial x^{\left( n \right)} }}\frac{{\partial x^{\left(n \right)} }}{{\partial t^{n - 1} }},\frac{\partial E}{{\partial x^{\left( n \right)} }}\frac{{\partial x^{\left( n \right)} }}{{\partial p^{n - 1} }}$.