Elsevier

Information Fusion

Volume 13, Issue 3, July 2012, Pages 185-195
Information Fusion

Super-resolution image reconstruction techniques: Trade-offs between the data-fidelity and regularization terms

https://doi.org/10.1016/j.inffus.2010.11.005Get rights and content

Abstract

Stochastic regularized methods are quite advantageous in super-resolution (SR) image reconstruction problems. In the particular techniques, the SR problem is formulated by means of two terms, the data-fidelity term and the regularization term. The present work examines the effect of each one of these terms on the SR reconstruction result with respect to the presence or absence of noise in the low-resolution (LR) frames. Experimentation is carried out with the widely employed L2, L1, Huber and Lorentzian estimators for the data-fidelity term. The Tikhonov and Bilateral (B) Total Variation (TV) techniques are employed for the regularization term. The extracted conclusions can, in practice, help to select an effective SR method for a given sequence of LR frames. Thus, in case that the potential methods present common data-fidelity or regularization term, and frames are noiseless, the method which employs the most robust regularization or data-fidelity term should be used. Otherwise, experimental conclusions regarding performance ranking vary with the presence of noise in frames, the noise model as well as the difference in robustness of efficiency between the rival terms. Estimators employed for the data-fidelity term or regularizations stand for the rival terms.

Research highlights

► Comparative study in super-resolution image reconstruction techniques performance. ► Super-resolution: trade-offs between the data-fidelity and regularization terms. ► Super-resolution: the L2, L1, Huber and Lorentzian estimators in the data-fidelity term. ► Super-resolution: the Tikhonov and BTV priors in the regularization term.

Introduction

Super-resolution is the term generally applied to the problem of transcending the limitations of optical imaging systems by employing image processing algorithms. Basically, in SR image reconstruction a sequence of noisy blurred LR images are fused to produce a higher-resolution image, which exhibits more high-frequency content and less noise and blur effects than any of the employed LR images. Actually, the obtained high-resolution (HR) image contains pieces of information from all the LR images. Therefore, data fusion takes place. Early works on SR reconstruction have shown that the recovery of the HR fused image is enabled by the aliasing effects that exist in the LR images, as long as there is a relative subpixel shift between the particular images [1]. The resolution enhancement results of SR image reconstruction methods [2], [3], [4] are more powerful than those of interpolation techniques [5], [6].

Several approaches to the SR image reconstruction problem have been developed [4], [7]. The stochastic regularized SR techniques are quite advantageous. In these techniques the formulation of the SR problem takes place by means of two terms, the data-fidelity term and the regularization term. In literature the L2, L1, Huber and Lorentzian estimators are commonly employed for the data-fidelity term. Additionally, the Tikhonov and bilateral TV regularization techniques are often utilized for the regularization term [3], [8], [9], [10], [11], [12], [13], [14]. Often, for a given sequence of LR frames, the most effective SR method among several potential ones has to be chosen before proceeding to the SR reconstruction task. If there is common data-fidelity or regularization term in the methods, and the LR frames are noiseless, the method which presents the most robust regularization or data-fidelity term should be employed. Nevertheless, if the frames are corrupted by noise and/or there are different data-fidelity terms as well as different regularization terms, selecting the most effective SR method is ambiguous. In literature a variety of SR reconstruction methods have been presented. Nevertheless, there has not yet been presented any work dealing with the selection of an effective SR method, among several potential ones, for a given LR sequence of frames. The present work treats this specific issue.

In the present work trade-offs between the data-fidelity and regularization terms are considered. Actually, these terms effect on the SR reconstruction result is worked out. The L2, L1, Huber and Lorentzian estimators are employed for measuring the difference between the projected estimate of the HR image and each LR frame. Regularization takes the form of the Tikhonov and Bilateral TV (BTV) priors. Eight different SR methods are formulated. Experimentation is carried out with noiseless frames as well as with frames corrupted by noise of various models. The experimental results are evaluated and conclusions are reached via grouping the SR methods. The methods are grouped per common data-fidelity term and per common regularization term. Grouping of the methods per pairs takes place as well. In the specific grouping, each pair of methods presents rival terms that exhibit different robustness of efficiency. Estimators employed for the similarity cost or regularizations stand for the rival terms. The conclusions reached can, in practice, help to select an effective SR image reconstruction method for a given sequence of LR frames. Therefore, in case that the potential SR methods present common data-fidelity term the method employing the most robust regularization should be chosen. If the methods display common regularization term and frames are noiseless, the ranking in methods performance is in accordance with the ranking in robustness of the estimators employed for assuring fidelity to the data. Nevertheless, in case that the potential methods exhibit different data-fidelity terms as well as different regularization terms, experimental conclusions regarding performance ranking vary. In the particular case the ranking in methods performance is affected by the absence or presence of noise in frames, the noise model and the difference in robustness of efficiency between the rival terms.

In Sections 2 Data-fidelity term, 3 Regularization term the data-fidelity and regularization terms that formulate the SR image reconstruction problem are discussed. The experimental procedure is presented in Section 4. The experimental results are provided in Section 5. Conclusions are drawn in Section 6.

Section snippets

Data-fidelity term

Super-resolution image reconstruction algorithms attempt to extract the HR image corrupted by the limitations of the optical imaging system. Before proceeding to solving the specific inverse problem, a forward model has to be formed. The most commonly employed forward model is linear and presents the following form:Y(t)=M(t)X(t)+V(t).The operator Y stands for the measured LR images, while X is the unknown HR image. The operator M represents the imaging system and V is the random noise inherent

The concept of regularization in mathematics

In general terms regularization is the approximation of an ill-posed problem by a family of neighbouring well-posed problems. Let it is required to approximate the best-approximate solution x = Ty of:Tx=yfor a specific right-hand side y in the situation that the exact data y are not known precisely, but that only an approximation yδ with:yδ-yδis available. yδ is called noisy data, whilst δ is called noise level. In the ill-posed case Tyδ is certainly not a good approximation of Ty due to

Experimental procedure

In this work experimentation is carried out to assess the importance of each one of the data-fidelity and regularization terms in affecting the SR image reconstruction result, with respect to the presence or absence of noise in the LR frames. Various noise models are considered. An HR image is created from a sequence of subpixel shifted, aliased LR frames. Resolution is increased by a factor of four. The employed error norms and priors are given in Table 1. The L2, L1, Huber and Lorentzian

Numerical results

Fig. 3, Fig. 4 depict SR reconstructed Lena images obtained for noiseless frames and frames corrupted by salt& pepper noise, correspondingly. Table 4, Table 5, Table 6, Table 7, Table 8 present the numerical results obtained for the tested SR methods in all cases of experimentation. The values of the Xydeas and Petrovich [26], and MSE measures are presented. Evaluation of the experimental results can take place via grouping the tested methods, along with their results.

Grouping per common data-fidelity term

Table 9 presents ranking,

Conclusions

In this work trade-offs between the data-fidelity and regularization terms, which formulate the SR image reconstruction problem in the context of stochastic regularized techniques, are discussed. Experimentation is carried out with the L2, L1, Huber and Lorentzian estimators employed for the data-fidelity term. In addition, the Tikhonov and BTV regularizers are utilized for the regularization term. Eight different SR techniques are formulated. Super-resolution image reconstruction is performed

References (26)

  • M.V.W. Zibetti et al.

    Determining the regularization parameters for super-resolution problems

    Signal Process.

    (2008)
  • R.Y. Tsai et al.

    Multiframe image restoration and registration

    ACVIP

    (1984)
  • A. Panagiotopoulou, V. Anastassopoulos, Super-resolution image reconstruction employing Kriging interpolation...
  • A. Panagiotopoulou, V. Anastassopoulos, Super-resolution reconstruction of thermal infrared images, in: 4th WSEAS Int....
  • A.K. Katsaggelos et al.

    Super Resolution of Images and Video, Synthesis Lectures on Image, Video and Multimedia Processing

    (2007)
  • V. Tsagaris, A. Panagiotopoulou, V. Anastassopoulos, Interpolation in multispectral data using neural networks, in:...
  • A. Panagiotopoulou et al.

    Scanned images resolution improvement using neural networks

    Neural Comput. Appl.

    (2008)
  • S.C. Park et al.

    Super-resolution image reconstruction: a technical overview

    IEEE SP Mag.

    (2003)
  • S. Farsiu et al.

    Fast and robust multiframe super-resolution

    IEEE Trans. Image Process.

    (2004)
  • L.C. Pickup, S.J. Roberts, A. Zisserman, A sampled texture prior for image super-resolution, Advances in NIPS Conf.,...
  • V. Patanavijit, S. Jitapunkul, A robust iterative multiframe super-resolution reconstruction using a Huber–Bayesian...
  • V. Patanavijit, S. Jitapunkul, An iterative super-resolution reconstruction of image sequences using a Bayesian...
  • V. Patanavijit, S. Jitapunkul, An iterative super-resolution reconstruction of image sequences using affine block-based...
  • Cited by (27)

    • Learning an epipolar shift compensation for light field image super-resolution

      2022, Information Fusion
      Citation Excerpt :

      Generally, the LF image presents different view information in sub-aperture images with sub-pixel shifts in a narrow baseline so that there exist strong correlations among them, which provide the redundant data used generally by super-resolution (SR) techniques. As the internal similarity performs well in depth continuous region [5], traditional methods [6–13] for LF image SR first rely on the intrinsic imaging consistency, which explores the depth information to warp or register the sub-aperture images, and then different image priors are utilized to regularize the SR reconstruction process. Obviously, the disparity estimation is crucial for these approaches, and any defect in the depth computation or the image-level wrapping operation may introduce significant artifacts.

    • Real-world single image super-resolution: A brief review

      2022, Information Fusion
      Citation Excerpt :

      With the SR techniques that reconstruct a higher resolution output from the LR observation, we can obtain images with the resolution beyond the limit of imaging systems, thereby improving visual quality and benefiting the subsequent analysis and understanding tasks such as segmentation [1–5], detection [6–8], recognition [9–11], and motion tracking [12]. In general, as presented in Fig. 1, existing SR techniques can be grouped into two categories according to the LR input and the reconstructed HR output, i.e., video super-resolution (VSR) [13–29] and image super-resolution (ISR) [30–82]. On the whole, VSR aims to improve the spatial resolution (known as spatial VSR) [13–21] or the frame rate (known as temporal VSR) [22–29] of the observed video.

    • Adaptive l<inf>q</inf>-norm constrained general nonlocal self-similarity regularizer based sparse representation for single image super-resolution

      2020, Information Fusion
      Citation Excerpt :

      Especially, the performance of the interpolation technology will be deteriorated seriously when the upsampling factor is higher than 2 [31]. The reconstruction technology hallucinates the high frequency component lost in degradation process with the aid of the image prior knowledge such as TV smoothness prior [7,8], NLM_SKR complementary prior [9], edge-directed prior [10], gradient profile prior [11], and so on. These image priors can well suppress artifacts and reconstruct fine high-frequency details in the reconstruction.

    • An iterative image super-resolution approach based on Bregman distance

      2017, Signal Processing: Image Communication
      Citation Excerpt :

      Since the construction of the prior information about the restored image is very difficult and knowing that we can only consider translation motion between LR frames, the frequency model is limited. However, many spatial domains SR reconstruction methods have been introduced, including projection onto convex sets approach [24,25] maximum a posteriori SR [26–29], learning-based approach [30,31] and regularization-based method [7,32–34]. While there is many multiframe SR techniques based on other techniques and hybrid algorithms such as the use of Non-local-Means algorithm [35,36] and Wavelet-based method [37–39].

    View all citing articles on Scopus
    View full text