DDNSR: a dual-input degradation network for real-world super-resolution

Abstract

Recently, Real-World Super-Resolution has become one of the most popular research fields in the scope of Single Image Super-Resolution, as it focuses on real-world applications. Due to the lack of paired training data, developing real-world super-resolution is considered a more challenging problem. Previous works intended to model the real image degradation process so that paired training images could be obtained. Specifically, some methods attempt to explicitly estimate degradation kernels and noise patterns, while others introduce degradation networks to learn maps from high-resolutions (HRs) to low-resolutions (LRs), which is a more direct and practical way. However, previous degradation networks take only one HR image as an input and therefore can hardly learn the real sensor noise contained in LR samples. In this paper, we propose a novel dual-input degradation network that takes a real LR image as an additional input to better learn the real sensor noise. Furthermore, we propose an effective self-supervised learning method to synchronously train the degradation network along with the reconstruction network. Extensive experiments showed that our dual-input degradation network can better simulate the real degradation process, thereby indicating that the reconstruction network outperforms state-of-the-art methods. Original codes and most of the testing data can be found on our website.

Appendices

Appendix A: Convolution with Zero-padding

In the scope of SISR, deep-learning-based models always use zero padding to ensure that the convolution outputs have the same size as inputs. We consider a single-input CNN-based network G (as used in most existing unsupervised/self-supervised methods) containing n convolution layers, and an input image x of size $h \times w$, with all pixel values set to be the same (e.g., full-white color). We set G with random weights and random bias values; then, we calculate G(x), we will obtain an output with a patch size $(h-n) \times (w-n)$ having the same pixel value (Fig. 9).

Fig. 9

Results of a single-input CNN-based network G with zero-padding. If G is a shallow network, most of the pixels in the output will have the same value. The input image has a typical size of $32 \times 32$, and ‘n’ is the number of convolution layers

This could be a fatal weakness. First, all the hidden outputs of which patch size $(h-n) \times (w-n)$ is positive will contain no noise at all. In addition, zero padding is completely artificial, which is a very low-efficiency way for training. Based on our observation, those single-input CNN-based networks can hardly learn the real sensor noise. We have shown this phenomenon in our ablation study (Sect. 4.5).

Appendix B: Cleaned-up result

As mentioned in Sect. 4.2, the clean-up method was helpful. We adopted this method in Track2 and evaluated it above. On the one hand, this method could help our model reconstruct more details (Fig. 8). On the other hand, it will introduce unnecessary noise if we look at the reconstruction results carefully.

Fig. 10

The perceptual quality of the cleaned-up results are still blurry and noisy. It is notable that images in the same row are cropped from the same scene but are not fully matched

In Fig. 10, we show some of the cleaned-up results compared with the original noisy LR images and the target images. As we can see, the cleaned-up image is not completely clean since the bicubic algorithm is not an ideal clean-up algorithm.

Appendix C: Full-reference metrics of Track1

We used only the $L_2$ loss as the loss function instead of Eq. (2) for the reconstruction network, trained two independent models, and calculated the PSNR, SSIM, and LPIPS [19] values on the 100 validation image pairs.

As shown in Table 3, $L_2$ loss could lead to a higher PSNR and SSIM value, which is a common phenomenon that appeared in many other works. In the context of Image Super-Resolution, there is no persuasive measurement of the image quality yet. Thus, we provided these results for reference only.

Table 3 PSNR and SSIM Values Calculated on Track1

Appendix D: Full results on both tracks

See Figs. 11, 12.

Fig. 11

Full result on Track1. The left part is the LR image upscaled by a 4 × 4 factor, the right part is the SR result generated by our DDNSR model

Fig. 12

Full results on Track2. The left part is the LR image upscaled by a 4 × 4 factor, the right part is the SR result generated by our DDNSR model