Elsevier

Signal Processing

Volume 123, June 2016, Pages 53-63
Signal Processing

Single image super-resolution using regularization of non-local steering kernel regression

https://doi.org/10.1016/j.sigpro.2015.11.025Get rights and content

Highlights

  • The non-local steering kernel regression (NLSKR) model is used to devise an effective regularization term for solving the single image SR problem.

  • A new regularization term is designed for preserving sharp edges and producing fine details in the resultant image.

  • By integrating the regularization term into the standard back-projection framework, we solve a least squares minimization problem to seek the desired high-resolution image.

Abstract

One promising technique for single image super-resolution (SR) is reconstruction-based framework, where the key issue is to apply reasonable prior knowledge to well pose the solution to upsampled images. In this paper, we employ the non-local steering kernel regression (NLSKR) model to devise an effective regularization term for solving single image SR problem. The proposed regularization term is based on the complementary properties of local structural regularity and non-local self-similarity existing in natural images, aiming at preserving sharp edges and producing fine details in the resultant image. By integrating the regularization term into the standard back-projection framework, we solve a least squares minimization problem to seek the desired high-resolution (HR) image. Extensive experimental results on several public databases indicate that the proposed method produces promising results in terms of both objective and subjective quality assessments.

Introduction

The objective of single image super-resolution (SISR) is to produce a high-resolution (HR) image with more details using only one low-resolution (LR) observation [1]. The SISR technique has attracted much attention in recent two decades due to its wide range of practical applications such as computer vision, medical and remote sensing imaging, video surveillance, and mobile multimedia devices. Up to now, a large number of methods have been proposed to yield a higher quality image which cannot be directly captured from imaging sensors. Generally speaking, existing SISR methods can be divided into three categories: interpolation-based methods, reconstruction-based methods, and example learning-based methods.

Interpolation-based methods [2], [3] usually apply a base function or an interpolation kernel to predict the unknown pixels in the HR grids, so it is probably an efficient way to yield an HR image for real time applications. However, because of the discontinuities in natural images, these methods tend to generate blurring details and ringing artifacts in the result, so the SR capability of this kind cannot meet the application requirement in practice.

Reconstruction-based methods achieve an SR estimate by imposing a certain prior knowledge on the reconstructed image. These approaches also require that the reconstructed HR image should be consistent with the LR input image via back-projection. A variety of edge-directed priors (e.g., edge prior [4], [5], gradient profile prior [6], and total variation prior [7]) have been proposed to preserve edges. Another popular prior is self-similarity redundancies within the input image that can be found in [8]. In addition, many Bayesian prior models, including Markov random fields [9] and the Gibbs distribution with a Huber potential function [10], have also been shown to be particularly effective for the reconstruction-based SR approaches. Nevertheless, these previous priors are beneficial only to suppress artifacts and preserve edges, but are clumsy at adding novel details lost in the LR image.

The third is example-based SR methods [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], which presume that the high frequency details missing in an LR image can be predicted from a database containing a large number of LR and HR image pairs. According to how the input and the output are formulated and how the mapping relationship is established, the representative learning techniques include k-nearest neighbor (k-NN) learning [11], [12], manifold learning [13], [14], [15], [16], sparse coding [17], [18], [19], and regression methods [20]. The k-NN learning methods are simple and intuitional and perform detail synthesis by similarity search among an enormous database, so it is time-consuming to perform SR reconstruction. Manifold learning-based methods are good at generalization in the case of smaller training database. Nonetheless, with a fixed number of nearest neighbors, they are prone to produce blurring results due to over- or under-fitting. Sparse representation methods use sparse prior to adaptively choose the most related neighbors to synthesize the details, so they enable to achieve better results than the previous example-based methods. However, one of the most important issues of them is the difficulty to determine an optimal dictionary for a generic natural image. Kim et al. [20] utilized kernel ridge regression to learn the correspondence between the LR and HR images for SR. However, this method cannot effectively control blurring effects along dominant edges. The example-based SR approaches exceed the reconstruction-based methods in that they are capable of generating new details which cannot be found in the LR input. However, the SR quality remains largely affected by the quality of training examples or applied dictionary.

In this paper, we propose to develop a novel SISR algorithm based on the reconstruction-based model. We utilize an effective regression model in [21], i.e., the non-local steering kernel regression (NLSKR), to design a unified regularization term for solving reconstruction-based SISR problem. The major characteristics of newly proposed method are three-fold.

  • 1.

    We jointly use two complementary characteristics of the local structural regularity and non-local self-similarity in natural images to develop a unified regularization term for ill-posed SR problem. The local structural regularity is helpful to preserve sharp edges. While non-local self-similarity is beneficial to yield a reliable result.

  • 2.

    We frame the proposed regularization term under the standard reconstruction-based SR framework and solve a local optimal solution to the desired HR image using gradient descent rule.

  • 3.

    Our approach that employs the regularization term constructed from the input LR image itself can yield state-of-the-art SR performance on a large public database.

The rest of this paper is organized as follows. Section 2 briefly reviews related works for SISR in the literature. Section 3 describes our NLSKR regularization based SISR method. Experimental results are presented in Section 4. Finally, Section 5 concludes the paper.

Section snippets

Related work

It is widely acknowledged that the performance of reconstruction-based methods largely depends upon the imposed prior knowledge. To produce an edge sharpened result, edge-directed prior is a very popular method applied in the reconstruction-based methods. For example, Fattal proposed to use edge-statistics [5] to conduct image upsampling. Sun et al. [6] utilized a gradient profile prior for SR reconstruction. A soft edge smoothness prior applied in the alpha-channel of salient edges was found

The proposed method

Recently, Zhang et al. [21] presented a non-local steering kernel regression (NLSKR) model for image and video SR restoration, where both non-local self-similarity and local structural regularity are integrated to perform the SR estimate in a local pixelwise fashion. There are two merits in the NLSKR framework. The first one is that the weights or kernels can locally adapt to the underlying structure of images, so the locally edge-directed filtering is capable of following dominant edge

Test datasets

To validate the effectiveness of the proposed SR method, we use five test databases as a benchmark to evaluate the SR performance of our method, including four standard databases Set5, Set14, D100, and Tex136 used in [29], and another newly created one (represented as Set8 in the paper) as shown in Fig. 2. The test datasets Set5, Set8, Set14, B100, and Tex136 comprise 5, 8, 14, 100, and 136 images, respectively. These images cover a wide range of contents including humans, animals, plants, and

Conclusion

In this paper we have presented a novel reconstruction-based method for single image SR reconstruction. The proposed method employs the NLSKR model to construct an effective regularization term and solve an MAP estimate of a least squares minimization problem for HR estimate. The experimental results on extensive test images indicate that our method shows potential SR capability in preserving sharp edges and reproducing visual details with fewer artifacts. Under the proposed SR framework, we

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants (61471161, 61432014, 61572388, and 61501349), in part by the Fundamental Research Funds for the Central Universities under Grants (BDZ021403 and JB149901), in part by Microsoft Research Asia Project based Funding under Grant FY13-RES-OPP-034, in part by the Program for Changjiang Scholars and Innovative Research Team in University of China under Grant IRT13088, in part by the Shaanxi Innovative

References (34)

  • T.S. Huang et al.

    Multi-frame image restoration and registration

    Adv. Comput. Vis. Image Process.

    (1984)
  • X. Li et al.

    New edge-directed interpolation

    IEEE Trans. Image Process.

    (2001)
  • L. Zhang et al.

    An edge-guided image interpolation algorithm via directional filtering and data fusion

    IEEE Trans. Image Process.

    (2006)
  • Y.W. Tai, S. Liu, M.S. Brown, S. Lin, Super resolution using edge prior and single image detail synthesis, in:...
  • R. Fattal

    Image upsampling via imposed edge statistics

    ACM Trans. Graphics

    (2007)
  • J. Sun, J. Zhu, M.F. Tappen, Context-constrained hallucination for image super-resolution, in: Proceedings of the IEEE...
  • B. Morse, D. Schwartzwald, Image magnification using level-set reconstruction, in: Proceedings of the IEEE Conference...
  • K. Zhang et al.

    Single image super-resolution with non-local means and steering kernel regression

    IEEE Trans. Image Process.

    (2012)
  • S. Geman et al.

    Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1984)
  • L.C. Pickup, D.P. Capel, S.J. Roberts, A. Zisserman, Bayesian image super-resolution, continued, in: Proceedings of the...
  • W.T. Freeman et al.

    Example-based super-resolution

    IEEE Comput. Graph. Appl.

    (2002)
  • W.T. Freeman et al.

    Learning low-level vision

    Int. J. Comput. Vis.

    (2000)
  • H. Chang, D.Y. Yeung, Y. Xiong, Super-resolution through neigbor embedding., in: Proceedings of the IEEE Conference on...
  • W. Fan, D.Y. Yeung, Image hallucination using neighbor embedding over visual primitive manifolds, in: Proceedings of...
  • X. Gao et al.

    Joint learning for single image super-resolution via a coupled constraint

    IEEE Trans. Image Process.

    (2012)
  • K. Zhang et al.

    Partially supervised neighbor embedding for example-based image super-resolution

    IEEE J. Sel. Topic Signal Process.

    (2011)
  • J. Yang et al.

    Image super-resolution via sparse representation

    IEEE Trans. Image Process.

    (2010)
  • Cited by (0)

    Fully documented templates are available in the elsarticle package on CTAN.

    View full text