Super-resolution image reconstruction techniques: Trade-offs between the data-fidelity and regularization terms
Research highlights
► Comparative study in super-resolution image reconstruction techniques performance. ► Super-resolution: trade-offs between the data-fidelity and regularization terms. ► Super-resolution: the L2, L1, Huber and Lorentzian estimators in the data-fidelity term. ► Super-resolution: the Tikhonov and BTV priors in the regularization term.
Introduction
Super-resolution is the term generally applied to the problem of transcending the limitations of optical imaging systems by employing image processing algorithms. Basically, in SR image reconstruction a sequence of noisy blurred LR images are fused to produce a higher-resolution image, which exhibits more high-frequency content and less noise and blur effects than any of the employed LR images. Actually, the obtained high-resolution (HR) image contains pieces of information from all the LR images. Therefore, data fusion takes place. Early works on SR reconstruction have shown that the recovery of the HR fused image is enabled by the aliasing effects that exist in the LR images, as long as there is a relative subpixel shift between the particular images [1]. The resolution enhancement results of SR image reconstruction methods [2], [3], [4] are more powerful than those of interpolation techniques [5], [6].
Several approaches to the SR image reconstruction problem have been developed [4], [7]. The stochastic regularized SR techniques are quite advantageous. In these techniques the formulation of the SR problem takes place by means of two terms, the data-fidelity term and the regularization term. In literature the L2, L1, Huber and Lorentzian estimators are commonly employed for the data-fidelity term. Additionally, the Tikhonov and bilateral TV regularization techniques are often utilized for the regularization term [3], [8], [9], [10], [11], [12], [13], [14]. Often, for a given sequence of LR frames, the most effective SR method among several potential ones has to be chosen before proceeding to the SR reconstruction task. If there is common data-fidelity or regularization term in the methods, and the LR frames are noiseless, the method which presents the most robust regularization or data-fidelity term should be employed. Nevertheless, if the frames are corrupted by noise and/or there are different data-fidelity terms as well as different regularization terms, selecting the most effective SR method is ambiguous. In literature a variety of SR reconstruction methods have been presented. Nevertheless, there has not yet been presented any work dealing with the selection of an effective SR method, among several potential ones, for a given LR sequence of frames. The present work treats this specific issue.
In the present work trade-offs between the data-fidelity and regularization terms are considered. Actually, these terms effect on the SR reconstruction result is worked out. The L2, L1, Huber and Lorentzian estimators are employed for measuring the difference between the projected estimate of the HR image and each LR frame. Regularization takes the form of the Tikhonov and Bilateral TV (BTV) priors. Eight different SR methods are formulated. Experimentation is carried out with noiseless frames as well as with frames corrupted by noise of various models. The experimental results are evaluated and conclusions are reached via grouping the SR methods. The methods are grouped per common data-fidelity term and per common regularization term. Grouping of the methods per pairs takes place as well. In the specific grouping, each pair of methods presents rival terms that exhibit different robustness of efficiency. Estimators employed for the similarity cost or regularizations stand for the rival terms. The conclusions reached can, in practice, help to select an effective SR image reconstruction method for a given sequence of LR frames. Therefore, in case that the potential SR methods present common data-fidelity term the method employing the most robust regularization should be chosen. If the methods display common regularization term and frames are noiseless, the ranking in methods performance is in accordance with the ranking in robustness of the estimators employed for assuring fidelity to the data. Nevertheless, in case that the potential methods exhibit different data-fidelity terms as well as different regularization terms, experimental conclusions regarding performance ranking vary. In the particular case the ranking in methods performance is affected by the absence or presence of noise in frames, the noise model and the difference in robustness of efficiency between the rival terms.
In Sections 2 Data-fidelity term, 3 Regularization term the data-fidelity and regularization terms that formulate the SR image reconstruction problem are discussed. The experimental procedure is presented in Section 4. The experimental results are provided in Section 5. Conclusions are drawn in Section 6.
Section snippets
Data-fidelity term
Super-resolution image reconstruction algorithms attempt to extract the HR image corrupted by the limitations of the optical imaging system. Before proceeding to solving the specific inverse problem, a forward model has to be formed. The most commonly employed forward model is linear and presents the following form:The operator Y stands for the measured LR images, while X is the unknown HR image. The operator M represents the imaging system and V is the random noise inherent
The concept of regularization in mathematics
In general terms regularization is the approximation of an ill-posed problem by a family of neighbouring well-posed problems. Let it is required to approximate the best-approximate solution x† = T†y of:for a specific right-hand side y in the situation that the exact data y are not known precisely, but that only an approximation yδ with:is available. yδ is called noisy data, whilst δ is called noise level. In the ill-posed case T†yδ is certainly not a good approximation of T†y due to
Experimental procedure
In this work experimentation is carried out to assess the importance of each one of the data-fidelity and regularization terms in affecting the SR image reconstruction result, with respect to the presence or absence of noise in the LR frames. Various noise models are considered. An HR image is created from a sequence of subpixel shifted, aliased LR frames. Resolution is increased by a factor of four. The employed error norms and priors are given in Table 1. The L2, L1, Huber and Lorentzian
Numerical results
Fig. 3, Fig. 4 depict SR reconstructed Lena images obtained for noiseless frames and frames corrupted by salt& pepper noise, correspondingly. Table 4, Table 5, Table 6, Table 7, Table 8 present the numerical results obtained for the tested SR methods in all cases of experimentation. The values of the Xydeas and Petrovich [26], and MSE measures are presented. Evaluation of the experimental results can take place via grouping the tested methods, along with their results.
Grouping per common data-fidelity term
Table 9 presents ranking,
Conclusions
In this work trade-offs between the data-fidelity and regularization terms, which formulate the SR image reconstruction problem in the context of stochastic regularized techniques, are discussed. Experimentation is carried out with the L2, L1, Huber and Lorentzian estimators employed for the data-fidelity term. In addition, the Tikhonov and BTV regularizers are utilized for the regularization term. Eight different SR techniques are formulated. Super-resolution image reconstruction is performed
References (26)
- et al.
Determining the regularization parameters for super-resolution problems
Signal Process.
(2008) - et al.
Multiframe image restoration and registration
ACVIP
(1984) - A. Panagiotopoulou, V. Anastassopoulos, Super-resolution image reconstruction employing Kriging interpolation...
- A. Panagiotopoulou, V. Anastassopoulos, Super-resolution reconstruction of thermal infrared images, in: 4th WSEAS Int....
- et al.
Super Resolution of Images and Video, Synthesis Lectures on Image, Video and Multimedia Processing
(2007) - V. Tsagaris, A. Panagiotopoulou, V. Anastassopoulos, Interpolation in multispectral data using neural networks, in:...
- et al.
Scanned images resolution improvement using neural networks
Neural Comput. Appl.
(2008) - et al.
Super-resolution image reconstruction: a technical overview
IEEE SP Mag.
(2003) - et al.
Fast and robust multiframe super-resolution
IEEE Trans. Image Process.
(2004) - L.C. Pickup, S.J. Roberts, A. Zisserman, A sampled texture prior for image super-resolution, Advances in NIPS Conf.,...
Cited by (27)
Learning an epipolar shift compensation for light field image super-resolution
2022, Information FusionCitation Excerpt :Generally, the LF image presents different view information in sub-aperture images with sub-pixel shifts in a narrow baseline so that there exist strong correlations among them, which provide the redundant data used generally by super-resolution (SR) techniques. As the internal similarity performs well in depth continuous region [5], traditional methods [6–13] for LF image SR first rely on the intrinsic imaging consistency, which explores the depth information to warp or register the sub-aperture images, and then different image priors are utilized to regularize the SR reconstruction process. Obviously, the disparity estimation is crucial for these approaches, and any defect in the depth computation or the image-level wrapping operation may introduce significant artifacts.
Real-world single image super-resolution: A brief review
2022, Information FusionCitation Excerpt :With the SR techniques that reconstruct a higher resolution output from the LR observation, we can obtain images with the resolution beyond the limit of imaging systems, thereby improving visual quality and benefiting the subsequent analysis and understanding tasks such as segmentation [1–5], detection [6–8], recognition [9–11], and motion tracking [12]. In general, as presented in Fig. 1, existing SR techniques can be grouped into two categories according to the LR input and the reconstructed HR output, i.e., video super-resolution (VSR) [13–29] and image super-resolution (ISR) [30–82]. On the whole, VSR aims to improve the spatial resolution (known as spatial VSR) [13–21] or the frame rate (known as temporal VSR) [22–29] of the observed video.
Adaptive l<inf>q</inf>-norm constrained general nonlocal self-similarity regularizer based sparse representation for single image super-resolution
2020, Information FusionCitation Excerpt :Especially, the performance of the interpolation technology will be deteriorated seriously when the upsampling factor is higher than 2 [31]. The reconstruction technology hallucinates the high frequency component lost in degradation process with the aid of the image prior knowledge such as TV smoothness prior [7,8], NLM_SKR complementary prior [9], edge-directed prior [10], gradient profile prior [11], and so on. These image priors can well suppress artifacts and reconstruct fine high-frequency details in the reconstruction.
An iterative image super-resolution approach based on Bregman distance
2017, Signal Processing: Image CommunicationCitation Excerpt :Since the construction of the prior information about the restored image is very difficult and knowing that we can only consider translation motion between LR frames, the frequency model is limited. However, many spatial domains SR reconstruction methods have been introduced, including projection onto convex sets approach [24,25] maximum a posteriori SR [26–29], learning-based approach [30,31] and regularization-based method [7,32–34]. While there is many multiframe SR techniques based on other techniques and hybrid algorithms such as the use of Non-local-Means algorithm [35,36] and Wavelet-based method [37–39].
A novel reconstruction model for multi-frame super-resolution image based on <sup>lmix</sup> prior
2014, Computers and Electrical Engineering