A performance comparison among different super-resolution techniques☆
Introduction
The computerized image resolution enhancement began in 1984 when Tsai and Huang [1] introduced a mathematical method for combining multiple low resolution (LR) images to obtain a single high resolution (HR) image. While initially there was little interest in this technology, over time with much theoretical and practical improvement, the technique led to the development of many tools currently available and was used in different fields such as security surveillance, biomedical applications, remote sensing, object recognition (such as face, finger print, iris, vehicle number plate and text) and video conversion [2], [3]. Resolution enhancement is one of the most rapidly growing areas of research in the field of image processing. The term resolution refers to the ability of an imaging instrument in revealing the fine details of an object. The resolution of an imaging device depends on the quality of its optics as well as its recording (sensor) and display components. The spatial resolution of an imaging instrument can be improved by modifying the hardware (sensor) in two ways. The first approach is to increase the pixel numbers. However, this approach has rather limited applications since it decreases the Signal-to-Noise Ratio (SNR) and increases the image acquisition time, and therefore, it is challenging to balance the trade-off between resolution, SNR, and acquisition time [4]. The second approach is to increase the chip size; however, a chip size necessary to capture a HR image would be very expensive [5]. An interesting alternative to both of the aforementioned approaches is to use the super-resolution (SR) techniques. SR is an off-line approach for improving the resolution of an image. SR techniques are broadly divided into multi-frame SR (classic approach) and single-frame SR. In multi-frame SR techniques a set of LR images acquired from the same scene are combined to reconstruct a single HR image. LR images can be taken by the same imaging instrument or with different instruments. The goal is to find the information missing in one LR image in other LR images. By doing so, the information contained in all LR images is pooled to obtain a HR image [5]. Several multi-frame SR techniques have been investigated in medical imaging [4]. In single frame SR technique, the missing high frequency information in the LR image during the acquisition step is estimated from a large number of training set images and added to the LR image [2].
In this paper, we present a survey of major SR techniques. Besides this, the MATLAB codes written and published by different groups of researchers were downloaded from their websites and the performance of various SR techniques were compared. The comparisons are made in terms of common image quality metrics such as peak SNR (PSNR) and Structural Similarity (SSIM) discussed in details in Section 5. We also report the execution time of the codes for each method. A number of review papers have also been published in this field [3], [5], [6], [7], [8]. While some of these papers provide a good overview of SR techniques, only [8] provides a comprehensive performance comparison in terms of image quality metrics. The survey paper [8] has provided the performance comparison in terms of objective quality metrics; however, it is limited to single-frame SR techniques. This paper is different from the previous review papers in that it provides performance comparisons of both single-frame and multi-frame SR techniques. The rest of the paper is organized as follows. Section 2 explains observation model that relates the HR image to the observed LR images. Several multi-frame SR techniques are described in Section 3. The single-frame SR techniques are described in Section 4. The image quality metrics are discussed in Section 5. Section 6 provides comprehensive performance comparisons of various SR techniques with natural images. A detailed discussion of the pros and cons of each technique is presented in Section 7, and the paper is concluded in this section.
Section snippets
Observation model
The observation model describes the way by which the observed LR images have been obtained. It models the parameters that degrade the original HR image to the observed LR images; therefore, it is also termed as forward model. A number of parameters contribute to the reduced image quality. These include: (a) the blur created either by defocus or motion of the camera; (b) sampling an object at a frequency less than the highest frequency contained in the object produces aliasing artifact on the
Super-resolution algorithms
As we discussed earlier, a HR image is reconstructed either from a single LR image or from a sequence of LR images. There are a number of different approaches for reconstructing a single HR image from LR image(s). This paper includes only the most common reconstruction approaches.
Single-frame super-resolution
Although a number of multi-frame SR algorithms have been developed to enhance the resolution of an image, they highly depend on the estimation accuracy of the registration parameters [3]. The registration methods are restricted mostly to the global motion; however, different components in the same scene may have different or complex motion in the real world applications. In such cases, multi-frame SR methods do not give good results. Sometimes, LR images are better than the super-resolved
Image quality metrics
To compare the performance of SR techniques, Peak-SNR (PSNR) and Structural Similarity (SSIM) between the super-resolved image and its original are calculated. The PSNR is calculated from the Mean Square Error (MSE), which is the average error between the original image and the super-resolved image. Given a super-resolved image and its original X(i, j), MSE and PSNR are defined as:
The SSIM index computes the similarity between
Simulations
MATLAB software (version R2008a) was used to code and/or to run the programs. The MATLAB codes were downloaded from the websites of respective authors, and the parameters of each method were set according to the values given in their corresponding papers. A computer with the operating system 64 bit version of Windows 7, Intel (R) Pentium (R) CPU G620T 2.2 GHz processor, and 4 GB RAM was used to run the simulations. The screen resolution was 1920 × 1080. Natural images Barbara, Butterfly, Lena,
Conclusion
In this paper, we provided a general survey of the existing SR techniques. We also reported a comprehensive performance comparison among different SR techniques in terms of PSNR and SSIM indices. The results showed that the Fourier-based cubic interpolation method significantly blurred the reconstructed image. The IBP, robust regularization and single image bicubic interpolation methods introduced small amount of “ringing effect”; however, they preserved most of the image features. The
Acknowledgements
We thank several authors for providing their MATLAB codes online. We also thank University of Waterloo and Ryerson University for providing lab equipment and financial support for this study.
Damber Thapa (PhD) received his PhD degree in Vision Science at the University of Waterloo in 2015. He received B.Sc degree in Physics from Tribhuvan University, Kathmandu, Nepal. His current research interests include adaptive optics of the eye, optical imaging, and biomedical image processing.
References (100)
Image super-resolution survey
Image Vis Comput
(2006)- et al.
Improving resolution by image registration
CVGIP: Graph Models Image Process
(1991) - et al.
Quantization analysis and enhancement of a VLSI gradient-based motion estimation architecture
Digit Signal Process
(2012) - et al.
Improved resolution from sub-pixel shifted pictures
CVGIP Graph Models Image Process
(1992) - et al.
Bayesian combination of sparse and non-sparse priors in image super-resolution
Digit Signal Process
(2013) - et al.
Stochastic super-resolution image reconstruction
J Vis Commun Image Represent
(2010) - et al.
Resolution enhancement via probabilistic deconvolution of multiple degraded images
Pattern Recogn Lett
(2006) - et al.
Example-based image super-resolution with class-specific predictors
J Vis Commun Image Represent
(2009) - et al.
Steerable pyramid-based face hallucination
Pattern Recogn
(2005) - et al.
Hallucinating face by position-patch
Pattern Recogn
(2010)
Super-resolution of human face image using canonical correlation analysis
Pattern Recogn
Multiframe image restoration and registration
Adv Comput Vis Image Process
Single-image super-resolution based on Markov random field and contourlet transform
J Electron Imag
Super-resolution: a comprehensive survey
Mach Vis Appl
Super-resolution methods in MRI: can they improve the trade-off between resolution, signal-to-noise ratio, and acquisition time?
Magn Reson Med
A survey on super-resolution imaging
Signal Image Video Process
Super-resolution image reconstruction: a technical overview
IEEE Signal Process Mag
An introduction to super-resolution imaging
Computer vision applied to super-resolution
IEEE Signal Process Mag
Joint MAP registration and high-resolution image estimation using a sequence of undersampled images
IEEE Trans Image Process
A computationally efficient super-resolution reconstruction algorithm based on the hybrid interpolation
J Comput
New edge-directed interpolation
IEEE Trans Image Process
An edge-guided image interpolation algorithm via directional filtering and data fusion
IEEE Trans Image Process
Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation
IEEE Trans Image Process
High-resolution image recovery from image-plane arrays, using convex projections
JOSA A
Super-resolution video reconstruction with arbitrary sampling lattices and nonzero aperture time
IEEE Trans Image Process
A frequency domain approach to registration of aliased images with application to super-resolution
EURASIP J Appl Signal Process
A noise-robust frequency domain technique for estimating planar roto-translations
IEEE Trans Signal Process
Recursive re-construction of high resolution image from noisy under-sampled multiframes
IEEE Trans Acoust Speech Signal Process
Discrete cosine transform based regularized high-resolution image reconstruction algorithm
Opt Eng
A wavelet-based interpolation-restoration method for super-resolution (wavelet super-resolution)
Circ Syst Signal Process
Regularized super-resolution reconstruction of images using wavelet fusion
Opt Eng
Simultaneous noise filtering and super-resolution with second-generation wavelets
IEEE Signal Process Lett
Robust wavelet-based super-resolution reconstruction: theory and algorithm
IEEE Trans Pattern Anal Mach Intell
Image resolution enhancement via data-driven parametric models in the wavelet space
EURASIP J Image Video Process
Super-resolution with sparse mixing estimators
IEEE Trans Image Process
Single-frame image super-resolution using learned wavelet coefficients
Int J Imag Syst Technol
Single-frame image super-resolution through contourlet learning
EURASIP J Appl Signal Process
Recursive high-resolution reconstruction of blurred multiframe images
IEEE Trans Image Process
Fast and robust multiframe super-resolution
IEEE Trans Image Process
Robust super-resolution
IEEE Comput Soc Conf Comput Vis Pattern Recogn
Cited by (30)
An efficient and robust multi-frame image super-resolution reconstruction using orthogonal Fourier-Mellin moments
2017, DisplaysCitation Excerpt :SR is an approach that extracts the information present in LR image(s) and uses this information to estimate the intensity values of HR image(s). SR has found its usage in many image processing applications [1,2] varying from satellite and aerial imaging, medical image processing, sign and number plate recognition, facial image recognition and enhancement in surveillance videos, iris recognition, fingerprint recognition, enhancement of text documents, etc. Intuitively, SR is simply a process of increasing the number of pixels in an image.
Single-image super-resolution using orthogonal rotation invariant moments
2017, Computers and Electrical EngineeringCitation Excerpt :Thus, single-frame SR cannot be treated as a special case of multi-frame SR. Single-image super-resolution has been widely used in several applications [1–4], such as medical image processing, infrared imaging, recognition of facial images, fingerprint image enhancement, signature and number plate reading, and for improvement of text documents. In this paper, we focus our attention towards single-image super-resolution and propose a novel SR approach based on orthogonal rotation invariant moments (ORIMs).
Deep Learning in Medical Image Super-Resolution: A Survey
2023, International Journal of Engineering Trends and TechnologyHitchhiker's Guide to Super-Resolution: Introduction and Recent Advances
2023, IEEE Transactions on Pattern Analysis and Machine IntelligenceMagnetic resonance image super-resolution reconstruction based on frequency-domain constraints and cross-fusion feature
2023, Chinese Journal of Medical PhysicsReview on Video Super Resolution: Methods and Metrics
2023, 2023 International Conference on Control, Communication and Computing, ICCC 2023
Damber Thapa (PhD) received his PhD degree in Vision Science at the University of Waterloo in 2015. He received B.Sc degree in Physics from Tribhuvan University, Kathmandu, Nepal. His current research interests include adaptive optics of the eye, optical imaging, and biomedical image processing.
Kaamran Raahemifar (PhD) is a professor of Department of Electrical and Computer Engineering at Ryerson University. His research interests include hardware implementation and software based approaches to signal and image processing algorithms with focus on biomedical engineering application. He is a professional engineer of Ontario and a senior member of IEEE.
William R. Bobier (PhD) is a professor of Optometry and Vision Science at the University of Waterloo. His research interests include the optics of the eye and related binocular motor development. He is primarily interested in normal and abnormal developmental patterns in infants and children.
V. Lakshminarayanan (PhD) is a professor at the University of Waterloo and has held positions at UC Irvine, Universities of Missouri and Michigan. He was a KITP Scholar at the Kavili Institute of Theoretical Physics, and is an optics advisor to the International Center for Theoretical Physics, Trieste, Italy. He is a fellow of APS, SPIE, OSA, AAAS, and IoP.
- ☆
Reviews processed and recommended for publication to the Editor-in-Chief by Associate Editor Dr. F. Sahin.