Stochastic super-resolution image reconstruction

https://doi.org/10.1016/j.jvcir.2010.01.001Get rights and content

Abstract

The objective of super-resolution (SR) imaging is to reconstruct a single higher-resolution image based on a set of lower-resolution images that were acquired from the same scene to overcome the limitations of image acquisition process for facilitating better visualization and content recognition. In this paper, a stochastic Markov chain Monte Carlo (MCMC) SR image reconstruction approach is proposed. First, a Bayesian inference formulation, which is based on the observed low-resolution images and the prior high-resolution image model, is mathematically derived. Second, to exploit the MCMC sample-generation technique for the stochastic SR image reconstruction, three fundamental issues are observed as follows. First, since the hyperparameter value of the prior image model controls the degree of regularization and intimately affects the quality of the reconstructed high-resolution image, how to determine an optimal hyperparameter value for different low-resolution input images becomes a very challenging task. Rather than exploiting the exhaustive search, an iterative updating approach is developed in this paper by allowing the value of hyperparameter being simultaneously updated in each sample-generation iteration. Second, the samples generated during the so-called burn-in period (measured in terms of the number of samples initially generated) of the MCMC-based sample-generation process are considered unreliable and should be discarded. To determine the length of the burn-in period for each set of low-resolution input images, a time-period bound in closed form is mathematically derived. Third, image artifacts could be incurred in the reconstructed high-resolution image, if the number of samples (counting after the burn-in period) generated by the MCMC-based sample-generation process is insufficient. For that, a variation-sensitive bilateral filter is proposed as a ‘complementary’ post-processing scheme, to improve the reconstructed high-resolution image quality, when the number of samples is insufficient. Extensive simulation results have clearly shown that the proposed stochastic SR image reconstruction method consistently yields superior performance.

Introduction

The objective of super-resolution (SR) imaging is to fuse a set of similar images acquired from the same scene to produce a single higher-resolution image with more details in order to enhance visualization and/or improve content recognition ability. The SR image reconstruction problem was firstly addressed by Tsai and Huang [1], and it has become as an active research area due to its potential in overcoming hardware limitations of existing image acquisition systems [2], [3], [4], [5], [6], [7], [8], [9].

Motivated by the fact that the SR computation is, in essence, an ill-posed inverse problem [10], numerous Bayesian-inferenced methods [11], [12], [13] have been developed for addressing this issue. The basic idea of these methods is to exploit a regularization strategy for solving the ill-posed numerical issues encountered in the SR computation by incorporating the prior knowledge of higher-resolution image under reconstruction. The target high-resolution image can be estimated via some statistics (e.g., mean) based on a large number of samples generated according to its probability distribution. The distribution can be mathematically derived by applying Bayesian inference approach to exploit the information provided by the observed low-resolution images and the prior knowledge of the target high-resolution image. The major challenge of all these Bayesian-inferenced methods is that the derived distribution of the target high-resolution image is usually very complicated; thus, its statistics is quite difficult to compute analytically. To tackle this challenge, the Markov chain Monte Carlo (MCMC) method [14] is considered in this paper as a promising numerical approach, since it provides an extremely powerful and efficient way to compute the statistics of any complicated distribution.

The Monte Carlo method, which is placed among the top 10 scientific computing algorithms that have had the greatest impact on the development and practice of sciences and engineering in the 20th century [15], firstly became popular in physics [16], and it has been applied to numerous applications [14]. Despite its extensive use in the various areas in the past, the exploitation of the Monte Carlo method in signal processing is fairly recent [17], [18], [19], [20], [21], [22]. The basic idea of the Monte Carlo method is to provide an accurate estimation of the unknown target (in our case, the high-resolution image) through stochastic sample generations. By further incorporating the first-order Markov chain property (i.e., the generation of the current sample only depends on the previous one) into the Monte Carlo method [16], a sufficiently large number of reliable samples will be generated according to the target probability distribution for the MCMC-based SR process. Note that the above-mentioned reliable samples are generated after discarding those initially-generated samples (considered as unreliable ones) such that the more accurate statistical measurement (say, the mean value) of the target distribution can be obtained. All these samples are generated according to the derived probability distribution of the target high-resolution image. Finally, the high-resolution image is produced by taking the average of those reliable samples. However, there are certain fundamental issues required to be addressed for the proposed MCMC-based SR image reconstruction approach, as follows.

It is important to note that the hyperparameter controls the degree of regularization and intimately affects the quality of the reconstructed high-resolution image. Therefore, the first issue is how to determine the optimal hyperparameter value of the prior image model. To address this issue, some works have been developed to automatically determine the prior image model’s hyperparameter value during the process of estimating the high-resolution image [23], [24], [25], rather than applying a fixed value determined experimentally [26]. Tipping and Bishop [23] exploited the expectation-maximization (EM) algorithm for estimating the hyperparameter value by maximizing its marginal likelihood function. However, the EM algorithm yields tremendous computational cost, and more importantly, the EM algorithm does not always converge to the global optimum [27]. Pickup et al. [24] improved the method [23] by reducing its computational load and further modified the prior image model by considering illumination variations possibly presented in the captured low-resolution images. He and Kondi [25] proposed a method to jointly estimate the hyperparameter value and the high-resolution image by optimizing a cost function. However, this method assumes that the displacements among low-resolution images are restricted to the integer-valued shifts on the high-resolution grid; obviously, this assumption does not always match the real-world image acquisition. In this paper, the prior image model’s hyperparameter value is updated at each iteration throughout the entire MCMC-based sample-generation process. To be more specific, at each iteration, the Metropolis–Hastings (M–H) algorithm [16] is exploited to simultaneously update the hyperparameter value, along with the generation of one pixel sample using the Gibbs sampler technique [17].

The second fundamental issue is the determination of up to when the samples to be generated afterwards will be considered as reliable for producing the high-resolution image. Note that any stochastic sample-generation process starts from an arbitrarily-generated sample as the seed, and based on which a fairly large number of samples will be generated through iterations. The period before samples become reliable since the inception of sample generation is known as the burn-in period, and the samples generated during this period are considered unreliable and should be discarded. Only the follow-up generated samples (i.e., after the burn-in period) are viewed reliable and used to produce the high-resolution image. In this paper, a mathematical analysis is conducted to derive the bound for determining the length of the burn-in period of the sample-generation process.

The third fundamental issue is regarding the number of samples required to be generated. Like any stochastic simulation task, the MCMC-based sample-generation process also requires a fairly large number of samples generated for estimating the target high-resolution image. Otherwise, image artifacts could be incurred in the reconstructed high-resolution image. Depending on the image content, the number of samples generated for the MCMC-based SR process could become insufficient for those areas containing high-frequency contents, such as edges and textures. In this case, image artifacts could be introduced in the reconstructed high-resolution image. To tackle this problem, our strategy is to incorporate a post-processing method, called the variation-sensitive bilateral filter, into the proposed MCMC-based SR approach for enhancing the reconstructed high-resolution image quality. The proposed artifact removal method contains two sequential stages: (i) an artifact detection stage to check whether the phenomenon of ‘insufficient number of samples’ has incurred at each pixel position; and (ii) an artifact filtering stage to remove image artifacts from the reconstructed high-resolution image, by applying either the conventional bilateral filter [28] (for “sufficient number of samples” case) or its modified version (for “insufficient number of samples” case), according to the binary decision result obtained at the artifact detection stage. For the proposed variation-sensitive bilateral filter, a median operation is performed (as pre-processing) for replacing the original intensity values of the estimated high-resolution image.

The paper is organized as follows. In Section 2, a Bayesian-inferenced formulation for the SR process is mathematically derived first. The stochastic MCMC-based sample-generation process is then described to see how one pixel sample and the hyperparameter value are generated at each sample-generation iteration. The bound for determining the length of the burn-in period of the sample-generation process is derived in Section 3. Our proposed variation-sensitive bilateral filter is described in Section 4 as a post-processing stage for addressing the issue of image artifacts. Extensive simulation experiments are conducted to justify our major developments along the sections. Section 5 concludes the paper.

Section snippets

Observation model

Given an image, it can be viewed as the ‘ground truth’ of the high-resolution image to be constructed from a set of low-resolution images for the purpose of conducting objective performance evaluation. To generate a set of low-resolution images (or observations) for simulation experiment, each given image is individually warped, convolved with a point spread function (PSF), downsampled to a lower resolution, and finally added with a zero-mean white Gaussian noise. These operations can be

Mathematical derivation

As discussed in Section 2.3, the initially-generated T samples (where T denotes the burn-in parameter in statistics) are considered unreliable and should be discarded. Only the follow-up generated samples (i.e., from the (T + 1)th sample onwards) are viewed reliable and used for estimating the high-resolution image via (14). In this section, a closed-form formulation for determining the value of the burn-in parameter T is mathematically derived as follows.

The basic idea governing our derivation

Variation-sensitive bilateral filter: proposed post-processing method

Like any stochastic simulation task, the proposed MCMC-based SR image reconstruction approach also requires a sufficiently large number of samples generated for estimating the high-resolution image under reconstruction; otherwise, a certain form of image artifacts could be incurred in the reconstructed high-resolution image. To address this concern, a modified version of the conventional bilateral filter [28], called the variation-sensitive bilateral filter, is proposed as a post-processing

Conclusions

In this paper, the powerful, well-known MCMC method has been successfully exploited for conducting stochastic SR image reconstruction, and the proposed approach consistently achieves superior image reconstruction results when compared with other state-of-the-art methods. A Bayesian inference formulation is first mathematically derived based on the observed low-resolution images and the prior high-resolution image model. The high-resolution image is then produced based on the reliable samples

References (40)

  • S. Borman, R.L. Stevenson, Spatial1 resolution enhancement of low-resolution image sequences: a comprehensive review...
  • S. Borman, R.L. Stevenson, Super-resolution from image sequences – a review, in: Proceedings of the IEEE Midwest...
  • M. Bertero et al.

    Introduction to Inverse Problems in Imaging

    (1998)
  • R.R. Schultz et al.

    Extraction of high-resolution frames from video sequences

    IEEE Transactions on Image Processing

    (1996)
  • R.C. Hardie et al.

    Joint MAP registration and high-resolution image estimation using a sequence of undersampled images

    IEEE Transactions on Image Processing

    (1997)
  • H. He et al.

    An image super-resolution algorithm for different error levels per frame

    IEEE Transactions on Image Processing

    (2006)
  • W.R. Gilks et al.

    Markov Chain Monte Carlo in Practice

    (1996)
  • J. Dongarra et al.

    Introduction to the top 10 algorithms

    Computing in Science & Engineering

    (2000)
  • N. Metropolis et al.

    Equations of state calculations by fast computing machines

    Journal of Chemical Physics

    (1953)
  • S. Geman et al.

    Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (1984)
  • Cited by (70)

    • Fusing bi-directional global–local features for single image super-resolution

      2024, Engineering Applications of Artificial Intelligence
    • Single image detecting enhancement through scattering media based on transmission matrix with a deep learning network

      2021, Optics Communications
      Citation Excerpt :

      This means that the projecting image is a binary image. In previous research, Zhou attempted to reconstruct a high-resolution (HR) image using its down-sample TM and few subpixel displacement images, which is similar to the multi-super-resolution reconstruction in traditional lens-based imaging applications [21–23]. His method obviously applies the principle of multiple-frame SR algorithm.

    • An efficient 3D face recognition approach using Frenet feature of iso-geodesic curves

      2019, Journal of Visual Communication and Image Representation
    • Multimedia super-resolution via deep learning: A survey

      2018, Digital Signal Processing: A Review Journal
      Citation Excerpt :

      Due to their localized nature, DWT domain is better suited. Regularization methods [58,59] are useful in case of limited “number of LR images or ill-conditioned blur operators”, and try to use either deterministic or stochastic “regularization strategy to incorporate some prior knowledge of the unknown HR image” [31]. For a detailed treatment on the subject, the readers are recommended to consult [30,60,31,61–63,29].

    View all citing articles on Scopus
    View full text