Elsevier

Signal Processing

Volume 183, June 2021, 108033
Signal Processing

Two-direction self-learning super-resolution propagation based on neighbor embedding

https://doi.org/10.1016/j.sigpro.2021.108033Get rights and content

Highlights

  • We found the vertical similarity prior in the image pyramid.

  • We proposed strategy with random oscillation, horizontal propagation, and vertical propagation to update the matching patches.

  • We proposed a coarse-to-fine feature selection method for patch matching.

Abstract

Neighbor embedding (NE) is a widely used super-resolution (SR) algorithm, but the one-to-many problem always degrades the performance of NE. The simplest way to avoid this performance degradation is to extract image features from low-resolution (LR) patches, that correctly reflect the features in the corresponding high-resolution (HR) patches. In this paper, we propose several feature extraction methods to extract patch features in LR space, and use coarse-to-fine patch matching methods to select matching patches for each test patch and find the best matching patch candidate to update the matching patch. Since finding matching patches using a training set is exhaustive work in NE, we explore the traditional local/non-local similarity prior and propose vertical similarity in the image pyramid to accelerate the matching patch search process. To accomplish this, we propose a random oscillation+horizontal propagation+vertical propagation strategy to update the matching patches. The experimental results show that the proposed method is superior to many existing self-learning-based methods, but it is inferior to many external-learning-based methods. These results show that the proposed feature extraction method and oscillation propagation method are useful for finding proper matching patches in NE.

Introduction

Single image super-resolution (SISR) generates a high-resolution (HR) image according to one or a group of low-resolution (LR) images [1], [2]. The most difficult task of SISR is to generate HR details according to the information that is provided by the LR images, because it is not easy to estimate the degradation parameters [3]. If we want to recover precise details, machine learning technologies provide many tools to learn the relationship between LR and HR images [4]. These tools include neural networks (such as convolutional neural networks (CNN) [5], [6], [7], [8], [9], [10], recurrent neural networks (RNN) [11], deep residual networks (ResNet) [12], [13], [14], [15], [16], and generative adversarial networks (GAN) [17], [18], [19], [20]), and sparse representation dictionaries [21], [22], [23], [24], [25], [26], [27]. Thus, a group of training images should be collected, and a training set that contains corresponding LR and HR images should be produced.

One machine learning algorithm is called ’self-learning’ [28], [29], [30]. This algorithm generates corresponding LR and HR training sets by using only the input LR test image itself. Compared with ’external-learning’ algorithms that collect training sets using other images [8], [31], [32], [33], [34], [35], there are several advantages to self-learning.

(1) Self-learning combines the training and testing processes and can adapt to different magnification factors, because the corresponding LR and HR training data sets can be generated according to the magnification factors [36]. However, an external learning algorithm separates the training and testing processes. When the parameters of machine learning tools are determined, the magnification factor can not change.

(2) Self-learning can generate high quality training images through a pyramid structure [28], [37], [38], [39]. Several related works [40], [41], [42] have shown that external learning recovers more diverse details with sparser distributions than internal learning, but internal learning concentrates on recovering specific details with a denser distribution. Therefore, self-learning can obtain good results on images with many repeated structures.

(3) Self-learning can be used in some applications without enough training images [43]. For example, some doctors can only obtain a very small number of images for lesion locations.

(4) Since self-learning only generates a small number of training samples from the LR test image itself, it does not require high-end computational and storage equipment in most cases.

However, self-learning still has many problems.

(1) Since only a small number of training samples can be generated using the LR test image itself, some machine learning tools such as the CNN [44] and sparse representation dictionary [24] do not work well. Affine transformations [28], [45](such as scaling, flipping, rotating and cropping) are always applied to the input LR test images to generate more training samples [46], but the number of training samples is still not enough for many machine learning tools.

(2) For each test image, the self-learning process contains training and testing processes. When seeking to accelerate the SR process, some high computational complexity training algorithms can not be used for self-learning.

To solve the above problems and make good use of their advantages, we propose a self-learning algorithm. The proposed algorithm utilizes a random oscillation to generate matching patch candidates, and then selects the proper features for vertical and horizontal propagations. This algorithm is based on neighbor embedding (NE) [47], [48]. The NE finds similar LR training patches for each test patch and then uses the corresponding HR training patches to reconstruct the HR test image. The reasons for using NE are as follows. First, the performance of NE is highly related to the training set. If the training set is similar to the testing set, NE achieves good performance. Because of the self-similarity prior [21], [49], the training set that is generated by the pyramid structure [50] is similar to the testing set, especially when the LR testing image has repetitive structures. Second, several strategies can be used with the NE algorithm to accelerate the patch matching process.

However, there are two problems that should be solved in the NE algorithm. (1) Since NE works on image patches, proper patch features should be extracted to find proper matching patch. (2) It has high time costs when finding the matching patches for each test patch. To solve the problems above, the contributions of the proposed algorithm are as follows. (1) We use the local binary pattern (LBP) feature to perform coarse selection for patch matching and select the best patch candidate to update the matching patch. The dimension of the LBP feature is the same as the variance but it can better reflect the textures of image patches. (2) We propose a random oscillation+horizontal propagation+vertical propagation algorithm to find the candidate matching patches. By using both the local/non-local similarity prior and the vertical similarity prior, the proposed algorithm can improve the quality of the reconstruction result.

The remainder of this paper is organized as follows: Section 2 reviews the related work. Section 3 describes the proposed algorithm in detail. Section 4 presents the experimental results. Section 5 concludes this paper.

Section snippets

Related work

The main purpose of self-learning-based SR is to use a machine learning method to magnify an LR image without relying on external data set. There are three key problems that should be solved in self-learning SR based on NE. The first is how to build the training data set. The second is how to extract the features of patches. The third is how to find the best matching patch. We will illustrate the methods for dealing with these problems in the following. We define the variables in Table 1.

Proposed method

The existing feature extraction methods and patch matching methods have two shortcomings. (1) The variance feature can only distinguish smooth and textured patches but cannot distinguish more textured features. When finding matching patches using NE, a short feature that can distinguish the texture of patches, which can decrease the computational complexity and increase the matching reliability,should be explored. (2) Using the traditional local /non-local similarity in the image pyramid to

Experimental setting

To compare the proposed algorithm with the state-of-the-art SR methods, we use the Set5, Set14 and Urban100 data sets, which are obtained from the code package [28]. Set5 and Set14 contain animals, humans, and scenery. Urban100 contains more buildings with repetitive structures. We use an AMD Ryzen 1600X CPU (3.6 GHz) and 16GB of memory, and only the CNN methods are performed using a GTX1070Ti Graphics Processing Unit (GPU).

The experimental parameters are set as follows. The patch size a×a is

Conclusion

This paper proposes a self-learning SR algorithm based on NE. We have shown that a good feature extraction method is helpful to improve the SR result. We use the LBP feature to select the best patch candidate for performing coarse selection, which eliminates the time complexity and improves the SR results; and a vertical similarity prior is found in the image pyramid. We propose a random oscillation+horizontal propagation+vertical propagation strategy to obtain better matching patches. Compared

CRediT authorship contribution statement

Jian Xu: Conceptualization, Methodology, Software, Funding acquisition. Yan Gao: Formal analysis, Writing - original draft, Visualization. Jun Xing: Software. Jiulun Fan: Supervision, Project administration. Qiannan Gao: Software. Shaojie Tang: Investigation.

Declaration of Competing Interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled “Two-direction Self-learning Super-Resolution Propagation Based on Neighbor Embedding”.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant 62071380, 61601362, 41874173, 62071378, and 62071379, China Scholarship Council, Natural Science Basic Research Plan in Shannxi Province of China 2019JQ-865, New Star Team of Xi’an University of Posts and Telecommunications xyt2016-01, Innovation Foundation for Postgraduate CXJJLZ202001.

References (59)

  • C. Tian et al.

    Coarse-to-fine CNN for image super-resolution

    IEEE Trans. Multimed.

    (2021)
  • B. Niu et al.

    Single image super-resolution via a holistic attention network

    European Conference on Computer Vision

    (2020)
  • K. Zhang et al.

    Learning a single convolutional super-resolution network for multiple degradations

    IEEE Conference on Computer Vision and Pattern Recognition

    (2018)
  • Z. Wang et al.

    Variational probabilistic generative framework for single image super-resolution

    Signal Process.

    (2019)
  • Y. Cao et al.

    Fast and accurate single image super-resolution via an energy-aware improved deep residual network

    Signal Process.

    (2019)
  • W. Han et al.

    Image super-resolution via dual-state recurrent networks

    IEEE Conference on Computer Vision and Pattern Recognition

    (2018)
  • S. Anwar et al.

    Densely residual Laplacian super-resolution

    IEEE Transactions on Pattern Analysis and MachineIntelligence

    (2021)
  • J. Li et al.

    MDCN: multi-scale dense cross network for image super-resolution

    IEEE Trans. Circuitsand Systems for Video Technology

    (2021)
  • M. Zhang et al.

    Supervised pixel-wise GAN for face super-resolution

    IEEE Trans. Multimed.

    (2021)
  • C. Ma et al.

    Structure-preserving super resolution with gradient guidance

    IEEE Conference on Computer Vision and Pattern Recognition

    (2020)
  • F. Yang et al.

    Learning texture transformer network for image super-resolution

    IEEE Conference on Computer Vision and Pattern Recognition

    (2020)
  • Y. Guo et al.

    Closed-loop matters: dual regression networks for single image super-resolution

    IEEE Conference on Computer Vision and Pattern Recognition

    (2020)
  • Y. Xu et al.

    Nonlocal patch tensor sparse representation for hyperspectral image super-resolution

    IEEE Trans. Image Process.

    (2019)
  • Nikhil et al.

    Contrast enhancement of progressive visual secret sharing (PVSS) scheme for gray-scale and color images using super-resolution

    Signal Process.

    (2019)
  • W. Dong et al.

    Nonlocally centralized sparse representation for image restoration

    IEEE Trans. Image Process.

    (2013)
  • B. Hou et al.

    Adaptive super-resolution for remote sensing images based on sparse representation with global joint dictionary model

    IEEE Trans. Geosci. Remote Sens.

    (2018)
  • Z. Xu et al.

    Single color image super-resolution using sparse representation and color constraint

    J. Syst. Eng. Electron.

    (2020)
  • J. Hu et al.

    Noise robust single image super-resolution using a multiscale image pyramid

    Signal Process.

    (2019)
  • J. Huang et al.

    Learning deep analysis dictionaries for image super-resolution

    IEEE Trans. Signal Process.

    (2020)
  • View full text