Residual scale attention network for arbitrary scale image super-resolution

Abstract

Research on super-resolution has achieved great success on synthetic data with deep convolutional neural networks. Some recent works tend to apply super-resolution to practical scenarios. Learning an accurate and flexible model for super-resolution of arbitrary scale factor is important for realistic applications, while most existing works only focus on integer scale factor. In this work, we present a residual scale attention network for super-resolution of arbitrary scale factor. Specifically, we design a scale attention module to learn discriminative features of low-resolution images by introducing the scale factor as prior knowledge. Then, we utilize quadratic polynomial of the coordinate information and scale factor to predict pixel-wise reconstruction kernels and achieve super-resolution of arbitrary scale factor. Besides, we use the predicted reconstruction kernels in image domain to interpolate low-resolution image and obtain coarse high-resolution image first, then make our main network learn high-frequency residual image from feature domain. Extensive experiments on both synthetic and real data show that the proposed method outperforms state-of-the-art super-resolution methods of arbitrary scale factor in terms of both objective metrics and subjective visual quality.

Introduction

Single image super-resolution (SISR) is a fundamental low-level vision task, and aims to restore a high-resolution (HR) image ${\mathit{I}}^{\mathit{HR}}$ from a single low-resolution (LR) image ${\mathit{I}}^{\mathit{LR}}$. It is a severely ill-posed inverse problem and has been studied for decades. In the past few years, convolutional neural network (CNN) based methods have made a remarkable improvement on SISR since Dong et al. proposed a three-layers convolutional neural network called SRCNN [1].

Recently, research tends to seek for realistic applications, which needs an accurate and flexible model. To obtain the accurate model, some CNN-based super-resolution works [2], [3], [4] make efforts to capture real data for training the network model. For flexibility, some work [5] focuses on developing upsampling method.

To upsample an image, some CNN-based super-resolution methods [1], [6], [7] interpolate the LR image to the target resolution before feed it into the network. This kind of method can achieve super-resolution of arbitrary scale factor using interpolation algorithm such as bicubic [8], but it may introduce some side effects, e.g. noise amplification and blurring. More importantly, these methods increase the network’s calculation and make it hard to run in real-time for realistic applications.

To avoid these problems, some CNN-based methods [9], [10], [11] feed the LR image into a network and upsample it within the network, where deconvolution module [12] or sub-pixel convolution module [13] is usually utilized. These methods mostly upscale images with integer scale factor, which limits the flexibility of models for zooming in arbitrary scale factor in realistic applications. To tackle this limitation, Hu et al. [5] introduced the meta-upscale module to achieve super-resolution of arbitrary scale factor, but did not explicitly consider the effect of different scale factors during the feature learning process.

In this paper, we present a residual scale attention network for image super-resolution of arbitrary scale factor, where we introduce scale factor as prior knowledge for the network to distinguish different data distributions and generate appropriate parameters automatically to assist feature learning. In our scale attention module, the scale factor is learned with statistic encoding of feature maps, to rescale the convolution filters with attention coefficients adaptively. This mechanism can dynamically generate attention coefficients to guide the convolution filters to extract more informative features according to current LR data distribution. Then, we utilize the quadratic polynomial of coordinate offset and scale factor as encoding vectors dynamically to predict pixel-wise reconstruction kernels, which benefits the non-linearity learning of the mapping between the reconstruction kernels and coordinate information. Besides, residual learning strategy is utilized in our reconstruction process, where the predicted kernels in image domain directly interpolate the LR image to coarse super-resolution image, and the predicted kernels in feature domain transform the learned feature maps into residual image. This makes the main network focus on the recovery of high-frequency information. Experimental results of our network on both synthetic and real data outperform state-of-the-art super-resolution methods of arbitrary scale factor under both comprehensive quantitative metrics and perceptive quality.

In summary, our main contributions are that

• We present a scale attention module to assist the network to distinguish LR images of different downsampling scale factors, and adaptively rescale the convolution filters in terms of different scale factors and convolve it with feature maps. It improves the ability to learn discriminative features for arbitrary scale factor.

• We utilize the quadratic polynomial of coordinate information and scale factor to dynamically predict pixel-wise reconstruction kernels. The quadratic polynomial contributes to learning the non-linearity relationship between reconstruction kernels and coordinate information, which helps more accurate reconstruction.

• We combine the predicted pixel-wise reconstruction kernels with residual learning strategy to boost the reconstruction accuracy. Experimental results demonstrate the effectiveness of the proposed residual scale attention network on both synthetic and real datasets for image super-resolution of arbitrary scale factor.