Single-image super-resolution using orthogonal rotation invariant moments

Highlights

• ORIMs-based single image super-resolution approach has been presented.

• The kernel functions of ORIMs have been used to estimate HR pixel values.

• Among ORIMs, ZMs, PZMs, and OFMMs have been considered in the proposed framework.

• Experiments conducted on both noise-free and noisy LR test images.

• The proposed approach is very fast as compared to NLM and NLM+ZMs/PZMs-based methods.

Abstract

In this paper, we propose an interpolation-based single-frame image super-resolution approach using orthogonal rotation invariant moments (ORIMs). The ORIMs have several useful characteristics in addition to having the property of image reconstruction. Therefore, they have been used successfully in many image processing applications. Among the various ORIMs, Zernike moments (ZMs), pseudo-Zernike moments (PZMs) and orthogonal Fourier-Mellin moments (OFMMs) have been considered in our proposed framework. The SR performance of the proposed approach has been compared with the classical interpolation-based approaches like bicubic, cubic B-spline, and Lanczos, as well as with nonlocal-means (NLM), and recently developed NLM+ZMs and NLM+PZMs-based SR approaches on twelve standard test images. The experiments have been conducted on both noise-free and noisy LR images corrupted with uniform blur and Gaussian noise. The results demonstrate the superiority of the proposed ORIMs-based approach in super-resolving both noise-free and noisy images over recently developed NLM+ORIMs-based SR approaches. Also, a comparative performance analysis, among various ORIMs (ZMs, PZMs, and OFMMs), is also presented to determine the ORIM which performs better over others under a given condition. A time complexity analysis shows that the proposed method is very fast as compared to NLM, NLM+ZMs and NLM+PZMs-based methods.

Introduction

The primary goal of a super-resolution (SR) technique is to obtain a high resolution (HR) image from one or more observations of a low resolution (LR) image. Since an HR image can offer more details that may be crucial, it has attracted a lot of demand in various applications involving security surveillance, biomedical sciences, remote sensing, object recognition (such as face, fingerprint, iris, vehicle number plate), etc. [1], [2], [3], [4], and has become one of the most active research areas in the field of image processing.

The techniques for SR can be divided into two different categories: single-frame SR and multi-frame SR. In single-image (or single-frame) SR, the HR pixel values are either interpolated or estimated using the information contained in single (degraded) LR image. Whereas, in multi-frame SR, information from all available (multiple) LR images (of the same scene) is accumulated to form a single high quality HR image. The single-image SR methods differ from the multi-frame SR methods as the latter requires more than one frame (or image) for image super-resolution which may not be available in many cases such as LR images of fingerprints reported from a crime scene, LR images of the number plate of over-speeding vehicles, LR images of old relics and documents, etc. Multi-frame methods require multiple images for their proper functioning. They are designed to extract inter-frame redundancies required to generate a high quality HR image, whereas, single-frame SR methods are designed specifically to work on one (and the only) LR image available for HR image reconstruction. Thus, single-frame SR cannot be treated as a special case of multi-frame SR. Single-image super-resolution has been widely used in several applications [1], [2], [3], [4], such as medical image processing, infrared imaging, recognition of facial images, fingerprint image enhancement, signature and number plate reading, and for improvement of text documents. In this paper, we focus our attention towards single-image super-resolution and propose a novel SR approach based on orthogonal rotation invariant moments (ORIMs).

Before going into the details of the proposed approach, we present a brief overview of various existing single SR techniques. Single SR techniques can broadly be classified into two groups. First group of the techniques are learning based-approaches that require an additional set of training images for generating an optimal HR image. These approaches estimate the missing information by determining the similarities between the given LR images and the training images present in the external dictionaries. Among the learning-based approaches, the sparse representation-based approaches [5], [6] are most predominant and provide very superior SR results. In sparse representation-based approaches, the sparse coefficients of the LR patches (extracted from the given LR image) are used to obtain the most suitable HR patch from the external HR dictionary. This external HR dictionary stores the sparse coefficients of the LR-HR patches extracted from the given set of training images. Finally, all super-resolved patches are combined to form an HR image. Different approaches vary in their methodologies to construct these HR dictionaries. Another learning-based approach which has become very popular because of its high SR performance is the example-based (EB) SR method proposed by Kim and Kwon [7]. In EB method, pairs of LR-HR image patches are collected and stored in the training stage. In the learning stage, each LR patch of the input image is then compared with the stored LR patches, and a nearest LR patch and its corresponding HR pair is selected by using sparse kernel ridge regression method. Recently, few single SR approaches [8], [9], [10] have emerged that present several ways to improve the example-based single-image super-resolution. Among them, anchored neighbor regression (ANR) [8] and adjusted anchored neighbor regression (A+) [9] are the two approaches which are faster versions of sparse representation-based approaches. A fast execution is achieved by using the neighbor-embedding and ridge-regression to learn sparse dictionaries offline and then using these neighborhoods to pre-compute projections to map LR patches onto the HR domain. The A+ is an improved variant of ANR approach. SRCNN [10] (super-resolution convolutional neural network) is another recently proposed single SR approach that uses convolutional neural networks for learning directly the end-to-end mapping between the LR and HR images. The SR performance reported by SCNN is observed to be the best among various existing state-of-the-art learning-based methods. Although the learning-based single-frame SR techniques are very effective, they are complex, computation intensive and their performance is dependent highly on the choice of training images and the training methods. Therefore, they need proper training to achieve high SR performance.

The second category of single-frame SR methods consists mainly of the interpolation-based approaches [11], [12]. These methods are very popular as they do not need any training process. Moreover, they are very simple and fast to compute. Since they do not need training, a new class of images can be super-resolved without a new set of training images. However, their performance is not as good as the learning-based methods. In interpolation-based methods, intensities in HR image are derived using information contained in LR image itself. However, artifacts like Gibb's phenomena (ringing effect), smearing near edges and stair-case effects are easily visible in the HR images super-resolved by these methods.

Attempts have been made to overcome the limitations of the interpolation-based methods and remove the effects of noise in the LR image. One such solution is to divide the entire SR process into three stages. In the first stage, interpolation methods are used to derive an initial estimate of the HR image, which then undergoes the denoising and deblurring operations in the second and the third stage to enhance further its SR quality. The bilateral-based [13] and the NLM-based [14] single-image SR methods belong to this category. These approaches are simple to implement and provide high quality HR images. The level of improvement depends largely in the manner these filters are used to denoise the HR image in the second stage because the deblurring process is handled by the standard deblurring algorithms in the third stage. Thus, stage two of the whole process is very crucial for the success of these approaches.

The NLM-based image super-resolution is very effective for images with smooth areas and repetitive textures for which the redundancy is high. However, its performance suffers due to three major issues: rare patch effect, patch jittering blur effect, and false patch detections due to rotation [15], [16]. The rare patch effect refers to the inability of the NLM method to find enough similar patches for singular structures, such as edges and corners, thus, performing insufficient denoising of the noisy image. The patch jittering blur effect causes over-smoothing of the image by averaging several pixels that do not truly belong to the same underlying texture. The presence of noise in the compared patches can lead to false detections. Moreover, when two patches are similar but rotated, the similarity value turns out to be low which prevents it from detecting many redundant regions of the image. These limitations are well addressed by the NLM-ORIMs-based image SR methods [16], [17], [18], [19]. These methods use the magnitude of ORIMs coefficients, instead of the intensity values, which are rotation invariant and robust to image noise. As a result, the NLM-ORIMs-based SR methods are very capable of determining similar patches even if the compared patches are noisy and rotated. One of the important properties of ORIMs is that they represent an image signal in various frequency components. The low frequency components represent the holistic view of an image and the high frequency components represent the fine details such as edges and corners in the image. Being orthogonal and complete, they are very useful in reconstructing the signal and hence they can be used for interpolating the signal. Being rotation-invariant, they preserve edge information in any orientation. In fact, ORIMs have been used for sub-pixel edge detection in images [20]. The moments are computed as an integration process, and hence they have been observed to be robust to image noise. Because of their useful properties, ORIMs have been very successful in several image processing applications including noise removal [21], [22] and image super-resolution [16], [17], [18], [19] which are relevant to the present work. Recently, Gao et al. [16], Kanan and Salkhordeh [17], [18], and Nayak and Patra [19] have used ZMs and PZMs, respectively, for multi-frame image super-resolution using the general framework of NLM-based image super-resolution [14]. However, no attempt has been made to use ORIMs for the single-frame image super-resolution. Therefore, in this paper, a novel idea of using ORIMs for single-image super-resolution has been presented. The functioning of the proposed ORIMs-based single SR approach is based on the three-stage strategy of SR, similar to the one adopted by [14], [16], [17], [18], [19]. However, the proposed ORIMs-based SR approach employs ORIMs kernel functions as opposed to conventional Bicubic or Lanczos interpolation methods used in NLM-based and NLM-ORIMs-based SR approaches to obtain an initial estimate of the HR image. The use of ORIMs kernel functions for interpolation in the first stage offers a good advantage over Bicubic or Lanczos interpolation in obtaining high quality HR images by preserving important image features with minimum amount of artifacts. The three ORIMs considered in this paper are: Zernike moments (ZMs), pseudo-Zernike moments (PZMs), and orthogonal Fourier-Mellin moments (OFMMs) because they have been used widely in several image processing applications.

The rest of the paper is organized as follows. Section 2 presents an introduction to the ORIMs and describes how the proposed ORIMs-based approach can estimate the desired HR image from the observed degraded LR image. Section 3 presents an overall algorithm for single-image super-resolution using various ORIMs. Section 4 presents a detailed experimental analysis for the performance evaluation of the proposed ORIMs-based approach and its comparison with other interpolation-based methods in terms of their PSNR and MSSIM values. Also, a comparative performance analysis of the proposed SR methods with recent state-of-the-art learning-based methods has been presented in Section 5. Time complexity analysis and speed enhancement techniques are discussed in Section 6. Section 7 concludes our work.