Single image super-resolution using regularization of non-local steering kernel regression☆
Introduction
The objective of single image super-resolution (SISR) is to produce a high-resolution (HR) image with more details using only one low-resolution (LR) observation [1]. The SISR technique has attracted much attention in recent two decades due to its wide range of practical applications such as computer vision, medical and remote sensing imaging, video surveillance, and mobile multimedia devices. Up to now, a large number of methods have been proposed to yield a higher quality image which cannot be directly captured from imaging sensors. Generally speaking, existing SISR methods can be divided into three categories: interpolation-based methods, reconstruction-based methods, and example learning-based methods.
Interpolation-based methods [2], [3] usually apply a base function or an interpolation kernel to predict the unknown pixels in the HR grids, so it is probably an efficient way to yield an HR image for real time applications. However, because of the discontinuities in natural images, these methods tend to generate blurring details and ringing artifacts in the result, so the SR capability of this kind cannot meet the application requirement in practice.
Reconstruction-based methods achieve an SR estimate by imposing a certain prior knowledge on the reconstructed image. These approaches also require that the reconstructed HR image should be consistent with the LR input image via back-projection. A variety of edge-directed priors (e.g., edge prior [4], [5], gradient profile prior [6], and total variation prior [7]) have been proposed to preserve edges. Another popular prior is self-similarity redundancies within the input image that can be found in [8]. In addition, many Bayesian prior models, including Markov random fields [9] and the Gibbs distribution with a Huber potential function [10], have also been shown to be particularly effective for the reconstruction-based SR approaches. Nevertheless, these previous priors are beneficial only to suppress artifacts and preserve edges, but are clumsy at adding novel details lost in the LR image.
The third is example-based SR methods [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], which presume that the high frequency details missing in an LR image can be predicted from a database containing a large number of LR and HR image pairs. According to how the input and the output are formulated and how the mapping relationship is established, the representative learning techniques include k-nearest neighbor (k-NN) learning [11], [12], manifold learning [13], [14], [15], [16], sparse coding [17], [18], [19], and regression methods [20]. The k-NN learning methods are simple and intuitional and perform detail synthesis by similarity search among an enormous database, so it is time-consuming to perform SR reconstruction. Manifold learning-based methods are good at generalization in the case of smaller training database. Nonetheless, with a fixed number of nearest neighbors, they are prone to produce blurring results due to over- or under-fitting. Sparse representation methods use sparse prior to adaptively choose the most related neighbors to synthesize the details, so they enable to achieve better results than the previous example-based methods. However, one of the most important issues of them is the difficulty to determine an optimal dictionary for a generic natural image. Kim et al. [20] utilized kernel ridge regression to learn the correspondence between the LR and HR images for SR. However, this method cannot effectively control blurring effects along dominant edges. The example-based SR approaches exceed the reconstruction-based methods in that they are capable of generating new details which cannot be found in the LR input. However, the SR quality remains largely affected by the quality of training examples or applied dictionary.
In this paper, we propose to develop a novel SISR algorithm based on the reconstruction-based model. We utilize an effective regression model in [21], i.e., the non-local steering kernel regression (NLSKR), to design a unified regularization term for solving reconstruction-based SISR problem. The major characteristics of newly proposed method are three-fold.
- 1.
We jointly use two complementary characteristics of the local structural regularity and non-local self-similarity in natural images to develop a unified regularization term for ill-posed SR problem. The local structural regularity is helpful to preserve sharp edges. While non-local self-similarity is beneficial to yield a reliable result.
- 2.
We frame the proposed regularization term under the standard reconstruction-based SR framework and solve a local optimal solution to the desired HR image using gradient descent rule.
- 3.
Our approach that employs the regularization term constructed from the input LR image itself can yield state-of-the-art SR performance on a large public database.
The rest of this paper is organized as follows. Section 2 briefly reviews related works for SISR in the literature. Section 3 describes our NLSKR regularization based SISR method. Experimental results are presented in Section 4. Finally, Section 5 concludes the paper.
Section snippets
Related work
It is widely acknowledged that the performance of reconstruction-based methods largely depends upon the imposed prior knowledge. To produce an edge sharpened result, edge-directed prior is a very popular method applied in the reconstruction-based methods. For example, Fattal proposed to use edge-statistics [5] to conduct image upsampling. Sun et al. [6] utilized a gradient profile prior for SR reconstruction. A soft edge smoothness prior applied in the alpha-channel of salient edges was found
The proposed method
Recently, Zhang et al. [21] presented a non-local steering kernel regression (NLSKR) model for image and video SR restoration, where both non-local self-similarity and local structural regularity are integrated to perform the SR estimate in a local pixelwise fashion. There are two merits in the NLSKR framework. The first one is that the weights or kernels can locally adapt to the underlying structure of images, so the locally edge-directed filtering is capable of following dominant edge
Test datasets
To validate the effectiveness of the proposed SR method, we use five test databases as a benchmark to evaluate the SR performance of our method, including four standard databases Set5, Set14, D100, and Tex136 used in [29], and another newly created one (represented as Set8 in the paper) as shown in Fig. 2. The test datasets Set5, Set8, Set14, B100, and Tex136 comprise 5, 8, 14, 100, and 136 images, respectively. These images cover a wide range of contents including humans, animals, plants, and
Conclusion
In this paper we have presented a novel reconstruction-based method for single image SR reconstruction. The proposed method employs the NLSKR model to construct an effective regularization term and solve an MAP estimate of a least squares minimization problem for HR estimate. The experimental results on extensive test images indicate that our method shows potential SR capability in preserving sharp edges and reproducing visual details with fewer artifacts. Under the proposed SR framework, we
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grants (61471161, 61432014, 61572388, and 61501349), in part by the Fundamental Research Funds for the Central Universities under Grants (BDZ021403 and JB149901), in part by Microsoft Research Asia Project based Funding under Grant FY13-RES-OPP-034, in part by the Program for Changjiang Scholars and Innovative Research Team in University of China under Grant IRT13088, in part by the Shaanxi Innovative
References (34)
- et al.
Multi-frame image restoration and registration
Adv. Comput. Vis. Image Process.
(1984) - et al.
New edge-directed interpolation
IEEE Trans. Image Process.
(2001) - et al.
An edge-guided image interpolation algorithm via directional filtering and data fusion
IEEE Trans. Image Process.
(2006) - Y.W. Tai, S. Liu, M.S. Brown, S. Lin, Super resolution using edge prior and single image detail synthesis, in:...
Image upsampling via imposed edge statistics
ACM Trans. Graphics
(2007)- J. Sun, J. Zhu, M.F. Tappen, Context-constrained hallucination for image super-resolution, in: Proceedings of the IEEE...
- B. Morse, D. Schwartzwald, Image magnification using level-set reconstruction, in: Proceedings of the IEEE Conference...
- et al.
Single image super-resolution with non-local means and steering kernel regression
IEEE Trans. Image Process.
(2012) - et al.
Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images
IEEE Trans. Pattern Anal. Mach. Intell.
(1984) - L.C. Pickup, D.P. Capel, S.J. Roberts, A. Zisserman, Bayesian image super-resolution, continued, in: Proceedings of the...
Example-based super-resolution
IEEE Comput. Graph. Appl.
Learning low-level vision
Int. J. Comput. Vis.
Joint learning for single image super-resolution via a coupled constraint
IEEE Trans. Image Process.
Partially supervised neighbor embedding for example-based image super-resolution
IEEE J. Sel. Topic Signal Process.
Image super-resolution via sparse representation
IEEE Trans. Image Process.
Cited by (0)
- ☆
Fully documented templates are available in the elsarticle package on CTAN.