Learning local Gaussian process regression for image super-resolution

Abstract

Learning based super-resolution (SR) methods, which predict the high-resolution pixel values but not directly provide an estimation of uncertainty, are typically non-probabilistic and have limited generalization ability. Gaussian processes can provide a framework for deriving regression techniques with explicit uncertainty models, but Gaussian Process Regression (GPR) has a significant drawback in being time consuming. The computational complexity of GPR is cubic in the number of training examples, which is prohibitively expensive for a large-scale training set. In this article, we have proposed learning local GPR for image SR. Two algorithms are developed to support local GPR for super resolution. A data-driven GPR based super-resolution algorithm is first developed to learn a local GPR model for every LR patch on an input oriented training dataset with moderate size. In order to further improve the running speed, a prototype based GPR algorithm is developed for super resolution. The proposed algorithm is about one-order faster than the data-driven GPR solution because it makes models for the prototypes of image patches rather than for each image patch. Thus, the local regression efforts are greatly reduced to just finding the nearest prototype for each LR image patch and applying its corresponding pre-computed projective matrix for super-resolution prediction. Our algorithms have greater robustness and usability as they provide a formularized way to automatically learn the hyper-parameters introduced for optimizing the covariance function, while most of the state-of-the-art super-resolution methods could only utilize these parameters in a cross-verification way. Moreover, our algorithms offer confidence values at the test points which benefit the pixels’ post-processing. Our algorithms are evaluated on popular datasets that are widely used in the super-resolution literature, and the experimental results have demonstrated that the efficiency and effectiveness of our proposed algorithms are comparative with several state-of-the-arts super-resolution methods.

Introduction

Image super-resolution (SR) is widely used in many practical applications such as satellite and medical imaging, where the analysis or diagnosis from low-resolution (LR) images can be difficult. Therefore, SR has become a hot topic in the field of computer vision. The goal of image SR is to generate a high-resolution (HR) image from one or multiple low-resolution (LR) input images. The SR problem is ill-posed because a low-resolution image can be generated by many different high-resolution images under different transformations. Up to now, a large number of learning based SR approaches [11], [12], [13], [14], [15], [16], [17], [18], [19] have been developed and they can yield promising results. However, the state-of-the-art learning based super-resolution methods are typically non-probabilistic and can predict the high-resolution pixel values but not directly provide an estimate of uncertainty. Thus, they are inadequate when the uncertainty is required. Moreover, the parameters used in these methods are specified a priori. In this article, we present an explicit probabilistic model for SR. We introduce Gaussian processes regression (GPR) for SR and our probabilistic formulation provides a principled way to learn hyper-parameters.

The computer vision community has paid little attention to Gaussian processes (GP) due to the fact that the Gaussian processes conventionally limit the amount of training data, because the computational complexity of GP is $O(N^3)$, which is cubic in the number of training examples and it is prohibitively expensive when the training dataset is large scale. In this article, we focus on how to make GPR work when the training dataset is large-scale. We propose an approach which learns local GPR models for the SR problem. We make a local GPR model for an LR image patch rather than make a global GPR model. It is worth noting that each query LR patch has its special local model. Two SR algorithms are developed to support the local GPR solution. In the first algorithm, a training collection is searched for an LR image patch from the training dataset, and then a GPR model is made depending on the training collection. Because each image patch has its special GPR model, the local GPR model is data-driven, and we name it data-driven Gaussian Process Regression (DDGPR). In the second algorithm, we first find prototypes for image patches and make GPR models for the prototypes of the image patches. We name it prototype-based GPR (PGPR). DDGPR focuses on the practicability of GPR and the super-resolution accuracy. And PGPR focuses on low running time.

The three main contributions of this article are: 1) an explicit probabilistic model is made for super-resolution based on Gaussian Process regression which could not only predict the high-resolution values but also give their confidence values; 2) the data-driven GPR based SR is developed, which is a local model and achieves the promising SR performance; 3) the prototype-based GPR scheme is developed, which reduces the computational complexity and is one-order faster than DDGPR. We have shown that the proposed approach can achieve state-of-the-art super-resolution results on the benchmark image set with superior scaling ability.

This article extends our previous work [23] by proposing a novel algorithm named PGPR. PGPR is different from DDGPR which is proposed in [23] in the two major ways: 1) in DDGPR, the local GPR models are made for image patches, while in PGPR, the local GPR models are made for prototypes of image patches; 2) when an HR image is generated from its LR image, we need to build an image training dataset and learn a GPR model for each image patch in DDGPR, while in PGPR we just anchor an image patch to its nearest prototype and look up the corresponding prototype model which is pre-computed and stored as a reference model. The two differences lead to much fewer GPR models and significantly less computational time of the HR image reconstruction for PGPR than for DDGPR. Thus, PGPR is about one-order faster than DDGPR without scarifying much accuracy. However, DDGPR can achieve more accurate results than PGPR. Moreover, we also add additional empirical results and making an in-depth analysis of our approach’s performance.

The remainder of this article is organized as follows. We introduce the related work in Section 2, and we give a brief overview of Gaussian Process regression in Section 3, and then we describe the local Gaussian Process regression algorithms for image super resolution in Section 4. The experimental results are shown in Section 5. In the last section, we give our conclusions.