A performance comparison among different super-resolution techniques

https://doi.org/10.1016/j.compeleceng.2015.09.011Get rights and content

Highlights

  • Comprehensive survey of super-resolution algorithms.

  • Performance comparison among different super-resolution algorithms.

  • Super-resolution techniques applied to natural images.

Abstract

Improving image resolution by refining hardware is usually expensive and/or time consuming. A critical challenge is to optimally balance the trade-off among image resolution, Signal-to-Noise Ratio (SNR), and acquisition time. Super-resolution (SR), an off-line approach for improving image resolution, is free from these trade-offs. Numerous methodologies such as interpolation, frequency domain, regularization, and learning-based approaches have been developed for SR of natural images. In this paper we provide a survey of the existing SR techniques. Various approaches for obtaining a high resolution image from a single and/or multiple low resolution images are discussed. We also compare the performance of various SR methods in terms of Peak SNR (PSNR) and Structural Similarity (SSIM) index between the super-resolved image and the ground truth image. For each method, the computational time is also reported.

Introduction

The computerized image resolution enhancement began in 1984 when Tsai and Huang [1] introduced a mathematical method for combining multiple low resolution (LR) images to obtain a single high resolution (HR) image. While initially there was little interest in this technology, over time with much theoretical and practical improvement, the technique led to the development of many tools currently available and was used in different fields such as security surveillance, biomedical applications, remote sensing, object recognition (such as face, finger print, iris, vehicle number plate and text) and video conversion [2], [3]. Resolution enhancement is one of the most rapidly growing areas of research in the field of image processing. The term resolution refers to the ability of an imaging instrument in revealing the fine details of an object. The resolution of an imaging device depends on the quality of its optics as well as its recording (sensor) and display components. The spatial resolution of an imaging instrument can be improved by modifying the hardware (sensor) in two ways. The first approach is to increase the pixel numbers. However, this approach has rather limited applications since it decreases the Signal-to-Noise Ratio (SNR) and increases the image acquisition time, and therefore, it is challenging to balance the trade-off between resolution, SNR, and acquisition time [4]. The second approach is to increase the chip size; however, a chip size necessary to capture a HR image would be very expensive [5]. An interesting alternative to both of the aforementioned approaches is to use the super-resolution (SR) techniques. SR is an off-line approach for improving the resolution of an image. SR techniques are broadly divided into multi-frame SR (classic approach) and single-frame SR. In multi-frame SR techniques a set of LR images acquired from the same scene are combined to reconstruct a single HR image. LR images can be taken by the same imaging instrument or with different instruments. The goal is to find the information missing in one LR image in other LR images. By doing so, the information contained in all LR images is pooled to obtain a HR image [5]. Several multi-frame SR techniques have been investigated in medical imaging [4]. In single frame SR technique, the missing high frequency information in the LR image during the acquisition step is estimated from a large number of training set images and added to the LR image [2].

In this paper, we present a survey of major SR techniques. Besides this, the MATLAB codes written and published by different groups of researchers were downloaded from their websites and the performance of various SR techniques were compared. The comparisons are made in terms of common image quality metrics such as peak SNR (PSNR) and Structural Similarity (SSIM) discussed in details in Section 5. We also report the execution time of the codes for each method. A number of review papers have also been published in this field [3], [5], [6], [7], [8]. While some of these papers provide a good overview of SR techniques, only [8] provides a comprehensive performance comparison in terms of image quality metrics. The survey paper [8] has provided the performance comparison in terms of objective quality metrics; however, it is limited to single-frame SR techniques. This paper is different from the previous review papers in that it provides performance comparisons of both single-frame and multi-frame SR techniques. The rest of the paper is organized as follows. Section 2 explains observation model that relates the HR image to the observed LR images. Several multi-frame SR techniques are described in Section 3. The single-frame SR techniques are described in Section 4. The image quality metrics are discussed in Section 5. Section 6 provides comprehensive performance comparisons of various SR techniques with natural images. A detailed discussion of the pros and cons of each technique is presented in Section 7, and the paper is concluded in this section.

Section snippets

Observation model

The observation model describes the way by which the observed LR images have been obtained. It models the parameters that degrade the original HR image to the observed LR images; therefore, it is also termed as forward model. A number of parameters contribute to the reduced image quality. These include: (a) the blur created either by defocus or motion of the camera; (b) sampling an object at a frequency less than the highest frequency contained in the object produces aliasing artifact on the

Super-resolution algorithms

As we discussed earlier, a HR image is reconstructed either from a single LR image or from a sequence of LR images. There are a number of different approaches for reconstructing a single HR image from LR image(s). This paper includes only the most common reconstruction approaches.

Single-frame super-resolution

Although a number of multi-frame SR algorithms have been developed to enhance the resolution of an image, they highly depend on the estimation accuracy of the registration parameters [3]. The registration methods are restricted mostly to the global motion; however, different components in the same scene may have different or complex motion in the real world applications. In such cases, multi-frame SR methods do not give good results. Sometimes, LR images are better than the super-resolved

Image quality metrics

To compare the performance of SR techniques, Peak-SNR (PSNR) and Structural Similarity (SSIM) between the super-resolved image and its original are calculated. The PSNR is calculated from the Mean Square Error (MSE), which is the average error between the original image and the super-resolved image. Given a super-resolved m×n image X^(i,j) and its original X(i, j), MSE and PSNR are defined as:MSE=1mni=0m1j=0n1[X(i,j)X^(i,j)]2PSNR=20log10(LMSE)

The SSIM index computes the similarity between

Simulations

MATLAB software (version R2008a) was used to code and/or to run the programs. The MATLAB codes were downloaded from the websites of respective authors, and the parameters of each method were set according to the values given in their corresponding papers. A computer with the operating system 64 bit version of Windows 7, Intel (R) Pentium (R) CPU G620T 2.2 GHz processor, and 4 GB RAM was used to run the simulations. The screen resolution was 1920 × 1080. Natural images Barbara, Butterfly, Lena,

Conclusion

In this paper, we provided a general survey of the existing SR techniques. We also reported a comprehensive performance comparison among different SR techniques in terms of PSNR and SSIM indices. The results showed that the Fourier-based cubic interpolation method significantly blurred the reconstructed image. The IBP, robust regularization and single image bicubic interpolation methods introduced small amount of “ringing effect”; however, they preserved most of the image features. The

Acknowledgements

We thank several authors for providing their MATLAB codes online. We also thank University of Waterloo and Ryerson University for providing lab equipment and financial support for this study.

Damber Thapa (PhD) received his PhD degree in Vision Science at the University of Waterloo in 2015. He received B.Sc degree in Physics from Tribhuvan University, Kathmandu, Nepal. His current research interests include adaptive optics of the eye, optical imaging, and biomedical image processing.

References (100)

  • H. Huang et al.

    Super-resolution of human face image using canonical correlation analysis

    Pattern Recogn

    (2010)
  • R.Y. Tsai et al.

    Multiframe image restoration and registration

    Adv Comput Vis Image Process

    (1984)
  • W. Wu et al.

    Single-image super-resolution based on Markov random field and contourlet transform

    J Electron Imag

    (2011)
  • K. Nasrollahi et al.

    Super-resolution: a comprehensive survey

    Mach Vis Appl

    (2014)
  • E. Plenge et al.

    Super-resolution methods in MRI: can they improve the trade-off between resolution, signal-to-noise ratio, and acquisition time?

    Magn Reson Med

    (2012)
  • J. Tian et al.

    A survey on super-resolution imaging

    Signal Image Video Process

    (2011)
  • S.C. Park et al.

    Super-resolution image reconstruction: a technical overview

    IEEE Signal Process Mag

    (2003)
  • Borman S, Stevenson R. Spatial resolution enhancement of low-resolution image sequences-a comprehensive review with...
  • J.D. Simpkins et al.

    An introduction to super-resolution imaging

  • Keren D, Peleg S, Brada R. Image sequence enhancement using sub-pixel displacements. In: Proceedings of IEEE computer...
  • Bergen JR, Anandan P, Hanna KJ, Hingorani R. Hierarchical model-based motion estimation. In: Proceedings of 2nd ECCV...
  • D. Capel et al.

    Computer vision applied to super-resolution

    IEEE Signal Process Mag

    (2003)
  • R.C. Hardie et al.

    Joint MAP registration and high-resolution image estimation using a sequence of undersampled images

    IEEE Trans Image Process

    (1997)
  • Cambridge in colour. Digital image interpolation. <http://www.cambridgeincolour.com/tutorials/image-interpolation.htm>...
  • X. Zhang et al.

    A computationally efficient super-resolution reconstruction algorithm based on the hybrid interpolation

    J Comput

    (2010)
  • X. Li et al.

    New edge-directed interpolation

    IEEE Trans Image Process

    (2001)
  • L. Zhang et al.

    An edge-guided image interpolation algorithm via directional filtering and data fusion

    IEEE Trans Image Process

    (2006)
  • Chu J, Liu J, Qiao J, Wang X, Li Y. Gradient-based adaptive interpolation in super-resolution image restoration. In:...
  • X. Zhang et al.

    Image interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation

    IEEE Trans Image Process

    (2008)
  • H. Stark et al.

    High-resolution image recovery from image-plane arrays, using convex projections

    JOSA A

    (1989)
  • Tekalp AM, Ozkan MK, Sezan MI. High-resolution image reconstruction from lower-resolution image sequences and...
  • A.J. Patti et al.

    Super-resolution video reconstruction with arbitrary sampling lattices and nonzero aperture time

    IEEE Trans Image Process

    (1997)
  • P. Vandewalle et al.

    A frequency domain approach to registration of aliased images with application to super-resolution

    EURASIP J Appl Signal Process

    (2006)
  • L. Lucchese et al.

    A noise-robust frequency domain technique for estimating planar roto-translations

    IEEE Trans Signal Process

    (2000)
  • S. Kim et al.

    Recursive re-construction of high resolution image from noisy under-sampled multiframes

    IEEE Trans Acoust Speech Signal Process

    (1990)
  • S. Rhee et al.

    Discrete cosine transform based regularized high-resolution image reconstruction algorithm

    Opt Eng

    (1999)
  • N. Nguyen et al.

    A wavelet-based interpolation-restoration method for super-resolution (wavelet super-resolution)

    Circ Syst Signal Process

    (2000)
  • S.E. El-Khamy et al.

    Regularized super-resolution reconstruction of images using wavelet fusion

    Opt Eng

    (2005)
  • M.B. Chappalli et al.

    Simultaneous noise filtering and super-resolution with second-generation wavelets

    IEEE Signal Process Lett

    (2005)
  • Ji H, Fermüller C. Wavelet-based super-resolution reconstruction: theory and algorithm. In: ECCV; 2006. p....
  • H. Ji et al.

    Robust wavelet-based super-resolution reconstruction: theory and algorithm

    IEEE Trans Pattern Anal Mach Intell

    (2009)
  • X. Li

    Image resolution enhancement via data-driven parametric models in the wavelet space

    EURASIP J Image Video Process

    (2007)
  • S. Mallat et al.

    Super-resolution with sparse mixing estimators

    IEEE Trans Image Process

    (2010)
  • C.V. Jiji et al.

    Single-frame image super-resolution using learned wavelet coefficients

    Int J Imag Syst Technol

    (2004)
  • C.V. Jiji et al.

    Single-frame image super-resolution through contourlet learning

    EURASIP J Appl Signal Process

    (2006)
  • S.P. Kim et al.

    Recursive high-resolution reconstruction of blurred multiframe images

    IEEE Trans Image Process

    (1993)
  • S. Farsiu et al.

    Fast and robust multiframe super-resolution

    IEEE Trans Image Process

    (2004)
  • A. Zomet et al.

    Robust super-resolution

    IEEE Comput Soc Conf Comput Vis Pattern Recogn

    (2001)
  • Babacan SD, Molina R, Katsaggelos AK. Total variation super resolution using a variational approach. In: 15th IEEE...
  • Villena S, Vega M, Molina R, Katsaggelos AK. Bayesian super-resolution image reconstruction using an l1 prior. In: 6th...
  • Cited by (30)

    • An efficient and robust multi-frame image super-resolution reconstruction using orthogonal Fourier-Mellin moments

      2017, Displays
      Citation Excerpt :

      SR is an approach that extracts the information present in LR image(s) and uses this information to estimate the intensity values of HR image(s). SR has found its usage in many image processing applications [1,2] varying from satellite and aerial imaging, medical image processing, sign and number plate recognition, facial image recognition and enhancement in surveillance videos, iris recognition, fingerprint recognition, enhancement of text documents, etc. Intuitively, SR is simply a process of increasing the number of pixels in an image.

    • Single-image super-resolution using orthogonal rotation invariant moments

      2017, Computers and Electrical Engineering
      Citation Excerpt :

      Thus, single-frame SR cannot be treated as a special case of multi-frame SR. Single-image super-resolution has been widely used in several applications [1–4], such as medical image processing, infrared imaging, recognition of facial images, fingerprint image enhancement, signature and number plate reading, and for improvement of text documents. In this paper, we focus our attention towards single-image super-resolution and propose a novel SR approach based on orthogonal rotation invariant moments (ORIMs).

    • Deep Learning in Medical Image Super-Resolution: A Survey

      2023, International Journal of Engineering Trends and Technology
    • Hitchhiker's Guide to Super-Resolution: Introduction and Recent Advances

      2023, IEEE Transactions on Pattern Analysis and Machine Intelligence
    • Review on Video Super Resolution: Methods and Metrics

      2023, 2023 International Conference on Control, Communication and Computing, ICCC 2023
    View all citing articles on Scopus

    Damber Thapa (PhD) received his PhD degree in Vision Science at the University of Waterloo in 2015. He received B.Sc degree in Physics from Tribhuvan University, Kathmandu, Nepal. His current research interests include adaptive optics of the eye, optical imaging, and biomedical image processing.

    Kaamran Raahemifar (PhD) is a professor of Department of Electrical and Computer Engineering at Ryerson University. His research interests include hardware implementation and software based approaches to signal and image processing algorithms with focus on biomedical engineering application. He is a professional engineer of Ontario and a senior member of IEEE.

    William R. Bobier (PhD) is a professor of Optometry and Vision Science at the University of Waterloo. His research interests include the optics of the eye and related binocular motor development. He is primarily interested in normal and abnormal developmental patterns in infants and children.

    V. Lakshminarayanan (PhD) is a professor at the University of Waterloo and has held positions at UC Irvine, Universities of Missouri and Michigan. He was a KITP Scholar at the Kavili Institute of Theoretical Physics, and is an optics advisor to the International Center for Theoretical Physics, Trieste, Italy. He is a fellow of APS, SPIE, OSA, AAAS, and IoP.

    Reviews processed and recommended for publication to the Editor-in-Chief by Associate Editor Dr. F. Sahin.

    View full text