An efficient approach for image sequence denoising using Zernike moments-based nonlocal means approach

Highlights

• Image sequence denoising using NLM and Zernike moments proposed.

• Significant improvement in image sequence denoising achieved.

• Denoising results are comparable to state-of-the-art methods with much faster speed.

• Proposed method is faster by a factor lying between 11 and 12.

• Better rates of improvement achieved at higher levels of noise.

Abstract

Nonlocal means (NLM)-based denoising is an efficient and simple method for image sequence denoising which has been applied successfully in many image sequence denoising applications. We extend the method for image sequence denoising using Zernike moments (ZMs) called NLM–ZMs which provides better denoising performance as compared to NLM-based approach. In addition, the proposed method is faster because the number of computations needed for block matching and weight computation are significantly reduced. The NLM approach uses the photometric distance between two image patches for determining the similarity distance. In the proposed approach, low order ZMs are used for the block matching process, resulting in better denoising performance at a much lower computation cost. Detailed experimental results are provided to demonstrate better performance and higher speed of the proposed approach as compared to the NLM approach. The results are also compared with the state-of-the-art NLM-based image sequence denoising methods and the denoising results are observed to be competitive with higher speed.

Introduction

Image sequence denoising is an important image processing task which has wide applications in several areas such as video films, infrared imaging, X-ray imaging, ultra-sound, and confocal microscopy. Restoration of old films and videos is an important area where image sequence denoising finds an important role because of the degradation of such videos over a period of time. The image sequence denoising is a difficult task because it involves estimation of motion while performing denoising. What makes the motion estimation difficult is the movement of the local objects in a scene. Therefore, the task of the denoising process not only involves the treatment of the problem in the spatial domain but also in the temporal domain. This complexity of the problem has kept the area active and more and more researches are being carried out in this area as ever before.

Many methods have been suggested for video sequence denoising. Broadly they are divided into two classes: methods with explicit motion estimation and methods without explicit motion estimation (motion is estimated implicitly). An initial survey of image sequence denoising can be found in [1]. The survey reports that the motion compensation task enhances the performance of both spatiotemporal and temporal filters from their non-motion compensated counterparts. Also, the performance of spatiotemporal filters is reported to be better than that of temporal filters. Later, a Weiner filter based approach for denoising of image sequences using 2D parametric motion model for explicit motion estimation in a spatiotemporal filtering scheme was proposed in [2]. Another spatiotemporal filtering approach has been proposed [3] in which spatial filtering is performed by the combination of wavelet and 2D Weiner filter while for temporal filtering bi-directional block based motion compensation using enhanced predictive zonal search (EPZS) algorithm is used. In contrast to above mentioned approaches, other approaches that do not rely on explicit motion estimation can also be found in the literature. For instance, a video denoising approach has been developed in [4] which is based on 3D anisotropic diffusion equation for noise removal and deblurring of video sequences. The proposed approach has extended the 2D diffusion equations to three dimensions to derive benefit from high degree of spatial and temporal correlation between the frames and thus it is capable of suppressing the flickering effect. Also, transform domain approaches based on DCT-based filtering [5] and wavelet-based filtering [6], [7] have been extended and applied to image sequence denoising in spatiotemporal domain. Among the methods without explicit motion estimation, one of the most effective and simple approach is based on the nonlocal means (NLM) filter developed by Buades et al. [8]. This approach has been extended to image sequence denoising [9] and video-denoising [10].

In the NLM approach, denoising is achieved by replacing every pixel by a weighted sum of its neighborhood pixels. The weights are obtained after computing a similarity parameter between the block centered on the pixel being averaged and the blocks centered on the neighborhood pixels being considered for the averaging process. The weights are considered to be proportional to the probability that a pixel might have moved from its current location on an image in the sequence being denoised to the location in its neighborhood. The effectiveness of this approach has been demonstrated by Buades et al. [9], [10] and Boulanger et al. [11].

The performance of NLM-based image sequence denoising is quite satisfactory, especially on smooth areas and repetitive textures for which the redundancy is high. On singular structures where the redundancy is low, the algorithm provides poor results because it fails to detect enough similar structures [12], [13]. The presence of high noise might affect similarity between two patches because of false patch detection. The similarity measure used for neighborhood blocks in the NLM-based method is simply a Euclidean distance of pixel values between two blocks which suffers from image noise. Two visually similar blocks but affected by noise may not always yield smaller distance as compared to two slightly dissimilar blocks. Moreover, the local motion in image sequence may cause rotation of objects or a slight change in the scene. Under these situations, the NLM-based block matching and denoising process does not work well. The use of block matching under rotation for single image denoising has been analyzed by Grewenig et al. [14] where the authors have used two approaches for this purpose. The first approach uses invariant geometric moments and the second approach is based on the estimation of the rotation angle and realigning the blocks. Improvements in the quality of denoising results have been reported by these authors. Also, an adaptive regularized NLM method has been proposed in [12], [13] which combines the NLM approach with total variation (TV) minimization problem in order to reduce the over-smoothing problem of NLM approach due to a combination of multiple irrelevant candidate patches. The authors have conducted experiments on both single images as well as image sequences and have shown improved quality of denoised images/image sequences obtained with their proposed regularized-NLM method. Another major drawback of the NLM method is its high computation time. The major components contributing towards the time requirements are the process of block matching and computation of weights. Mahmoudi and Sapiro [15] have attempted to reduce the time requirements by discarding those blocks which are dissimilar using average gray values of blocks and average gradient direction and magnitude. However, it affects the denoising performance.

In this paper, we propose nonlocal means and Zernike moments (ZMs)-based image sequence denoising approach, called the NLM–ZMs, which has several advantages over the NLM-based approaches. Being orthogonal and rotation invariant, ZMs possess several useful characteristics for image description over patch-based block matching used in the NLM-based approach. The ZMs encode the singular structures in a better way as the first, second and third order ZMs can effectively encode the edges and rapid change in intensity [16]. The ZMs possess minimum information redundancy and, therefore, a few low frequency components can be used to describe an image. This facilitates the process of block matching used in NLM-based approaches. The magnitudes of ZMs are rotation invariant and hence rotation of objects in a scene can be handled effectively by ZMs. This is particularly useful for the estimation of local motion of small objects which may undergo rotation. The moments are derived using a summation process, as a consequence, the effect of noise is reduced in the block matching process. We use a small number of ZMs coefficients as compared to the large number of pixel values in the NLM-based block matching approach. This step significantly reduces the time taken for block matching and weight computation which is one of the major issues in the NLM-based image sequence denoising approaches. Another key feature of the proposed approach is the enhancement of the denoising performance which is achieved by smoothing of weights. Unlike the NLM-based approach which considers the spatial distance of pixels within a patch (intra-patch distance) for smoothing the photometric distance, we smooth weights depending on the spatial distance between a reference patch and the target patch (inter-patch distance). Thus, we improve the performance of NLM–ZMs image sequence denoising approach by incorporating the concept of geometric distance between two image blocks being matched. It is also shown that as the noise level increases, the proposed method performs much better than the NLM-based approaches because ZMs-based block matching is robust to image noise.

It is worth mentioning here that the ZMs-based NLM approach has been applied to image denoising [17] and image super-resolution [18]. In [17], the authors have normalized ZMs which sometimes become unstable due to the normalization process as explained in Section 4.3. Moreover, their approach does not take into account the geometric distance between two image blocks which greatly enhances the denoising performance. In [18], the authors have used ZMs in image super-resolution, again without the use of geometric distance being taken into consideration. Moreover, the proposed method is applied to image sequence denoising, whereas the method in [17] deals single image denoising and the method in [18] deals with image super-resolution.

The rest of the paper is organized as follows. An overview of the NLM-based image sequence denoising is described in Section 2. The proposed NLM–ZMs-based image sequence denoising is presented in Section 3. Detailed experimental results on denoising performance and time complexity analysis are demonstrated in Section 4, followed by conclusion in Section 5.