Robust blind motion deblurring using near-infrared flash image

https://doi.org/10.1016/j.jvcir.2013.09.008Get rights and content

Highlights

  • NIR-flash image captures the high-frequency details lost in visible blurred image.

  • Blind deblurring algorithm exploits the gradient correlation between the image pair.

  • The NIR gradient constraint is used as a new type of image regularization.

  • The algorithm can accurately restore the images with uniform and non-uniform blur.

Abstract

In light-limited situations, camera motion blur is one of the prime causes for poor image quality. Recovering the blur kernel and latent image from the blurred observation is an inherently ill-posed problem. In this paper, we introduce a hand-held multispectral camera to capture a pair of blurred image and Near-InfraRed (NIR) flash image simultaneously and analyze the correlation between the pair of images. To utilize the high-frequency details of the scene captured by the NIR-flash image, we exploit the NIR gradient constraint as a new type of image regularization, and integrate it into a Maximum-A-Posteriori (MAP) problem to iteratively perform the kernel estimation and image restoration. We demonstrate our method on the synthetic and real images with both spatially invariant and spatially varying blur. The experiments strongly support the effectiveness of our method to provide both accurate kernel estimation and superior latent image with more details and fewer ringing artifacts.

Introduction

When taking photographs under low-light conditions with a hand-held camera, a long shutter speed is required for adequate exposure. However, the relative motion between the camera and the scene during exposure is ineluctable and results in a blurry image. Mathematically, the blur image caused by camera shake can be considered as a linear combination of multiple clear images captured from slightly different viewpoints. Restoration of images degraded by motion blur is a hotly debated topic in both image processing and computer vision. Since the high-frequency information is lost and no prior is known about the camera shake, recovering the latent image from the blurry observation is a challenging task.

In recent years, many researchers worked on the blind camera shake removal from a single image using different kinds of priors on image statistics [1], [2], [3]. In [1], the heavy-tailed distribution of image gradients are fitted by the zero-mean mixture-of-Gaussians model, then the variational Bayesian framework is used to approximate the MAP solution efficiently. Shan et al. [2] model the high-order derivatives of image noise and propose the sparse prior for the blur kernel and latent image estimation. Jia [4] analyzes the relationship between the image degradation and the transparency on the object boundary, and estimates the blur kernel by image matting. The approach in [5] maximizes the sparsity and spatial smooth of the blur kernel under curvelet system. In [6], [7], [8], the large-scale edges are selected to perform high-quality kernel estimation. Cho and Lee [6] choose simple Gaussian priors in the optimization, which can be accelerated using Fast Fourier Transform (FFT). The blind deconvolution problem in [9] is addressed by the split Bregman iteration to obtain high-quality image with low computational cost.

Given the estimated blur kernel, the image restoration is known as the non-blind deconvolution. Some traditional algorithms, such as Wiener filter [10] and Richardson–Lucy (RL) deconvolution [11], are widely used and known to be simple and efficient. However, the estimation suffers from unpleasant noise and ringing artifacts. In [12], the edge-preserving Bilateral Richardson–Lucy (BRL) deconvolution is performed with coarse-to-fine scheme. Furthermore, various regularization techniques, such as the Total-Variation (TV) regularization [13], hyper-Laplacian prior [14] and adaptive regularization [15], are explored to suppress the undesirable artifacts.

Commonly, most of the previous works assume the blur to be spatially invariant. However, as discussed in [16], the uniform blur assumption is often violated in practice. In [17], [18], [19], [20], the authors introduce different kinds of blur image formulations to model the imaging process with non-uniform camera shake. Tai et al. [21] modify the RL algorithm with regularization term to incorporate the spatially variant blur. Since blind deblurring from a single image is highly ill-posed, all the algorithms make strong prior assumptions about the blur kernel and image. However, natural images do not always meet the strong priors and the lost of image information is inevitable in the deblurring process.

To overcome the ambiguity of blind deblurring from one single image, many computational imaging techniques capturing multiple images to enhance the capabilities of digital photography are proposed. In [22], [23], [24], the blurred image pair obtained by consecutive shooting with same exposure time are assumed to be well-aligned. However, accurate automatic alignment of images blurred by different kernels is difficult to achieve by existing methods. Yuan et al. [25] use a pair of blurred and noisy images for deblurring. The blur kernel is estimated based on the sharp edges selected from the denoised image. The authors further propose an alignment approach for a blurred/no-blurred image pair in [26]. In [20], Whyte et al. extend the method in [25] to the non-uniform blur. Although the multiple images are complementary to each other, the high-frequency information of the scene has a certain amount of loss in the blurred image and the denoised result. Recently, the method in [27] recovers sharp image with the help of visible flash image, while it does not take the non-uniform blur into consideration.

Another group of computational photography methods employ specific hardware to control the capturing process. A high-resolution still camera and a low-resolution video camera are combined to form a hybrid camera system in [28], [29]. The motion blur of high-resolution image is estimated from the image sequence by optical flow. The inertial sensors are connected to the camera in [30] to capture the camera motion during exposure. These aided hardware systems provide more information on camera shake and can handle more complex motions.

Inspired by the idea of introducing additional information to assist the blind deblurring, we propose a method to deblur a photography by using the high frequency texture of its corresponding NIR-flash image. As shown in Fig. 1(b), the NIR image is sharp and captures the high-frequency details of the scene, which brings significant benefits to solve the highly ill-posed problem. We exploit the correlation of the image pair captured at visible and NIR wavebands, and propose a blind deblurring method with the application of NIR gradient constraint. Compared with other methods, our deblurring framework is based on the general blur model, and is able to provide accurate kernel estimation and superior image restoration with both spatially invariant and spatially varying blur.

The rest of this paper is organized as follows. The main properties of NIR flash image are introduced in Section 2. In Section 3, we describe the general blur model, and then the NIR flash deblurring problem is formulated into a MAP framework. Section 4 presents the blind image deblurring algorithm using the NIR flash gradient constraint for uniform blur. The algorithm is further extended to spatially variant blur in Section 5. The deblurring performance is evaluated and compared with other methods in Section 6. Finally, the conclusions and the discussions of current limitations are drawn in Section 7.

Section snippets

NIR image gradient constraint

Near-infrared light has the wavelength range of 750 nm to 1100 nm, which can be sensed by digital camera, but invisible to the human eyes. Since the physical differences in the visible and NIR wavebands, the RGB and NIR images have different intensities and frequency contents. NIR images usually contain more texture details and have higher contrast. In recent years, the NIR images are introduced to bring benefits to computational photography and computer vision applications, e.g., image

Problem formulation

In this section, we formulate the image blur model of spatially invariant and spatially varying camera shake. We unify the NIR image gradient constraint into a MAP formulation for kernel estimation and sharp image restoration.

Uniform deblurring algorithm

In this section, we first consider the uniform deblurring for shaken images. Our algorithm is similar to [27], but has a series of changes in the formulation and optimization to effectively fuse the high-frequency information in the NIR-flash image into the blind deblurring process. We divide the problem into two parts: blur kernel estimation and image restoration. Firstly, we convert the blurred image to YUV color space, and denote its luminance (Y channel) and chrominance components (U and V

Extension to non-uniform deblurring

In this section, we extend the deblurring algorithm described above to the non-uniform camera shake. In our experiments, we found that exactly adopt the same approach as for the uniform deblurring cannot get a satisfactory result for the non-uniform case. The reason should be that the iterative optimizations are non-convex and seriously depend on the accuracy of initialization. Just good initial value can ensure the iterations converge along the right direction. Therefore, the key point of this

Experiments

In this section, we present the results of our approach and do comprehensive comparisons against some state-of-the-art algorithms for blind deblurring. The performance of the algorithm is evaluated on both synthetic and real captured images considering spatially invariant as well as spatially variant blur models.

Conclusions

In this paper, we captured a well-aligned blurred/NIR-flash image pair, and exploited the correlation of images captured at different wavelengths. To utilize the edge structure captured in the NIR-flash image, the NIR gradient constraint was introduced as an image regularizer to compensate the attenuation of high-frequency information in blind deblurring. For both spatially invariant and spatially varying blur, we evaluated our approach on the synthetic and real data. Our approach was

Acknowledgments

This research is supported by the National Basic Research Program of China (Grant No. 2010CB731800), the Foundation for Innovative Research Groups of the National Natural Science Foundation of China (Grant No. 60921001), and the Key Project of NSFC (No. 61120106003).

References (50)

  • N. Joshi, R. Szeliski, D.J. Kriegman, PSF estimation using sharp edge prediction, in: IEEE Conf. on Computer Vision and...
  • L. Xu, J. Jia, Two-phase kernel estimation for robust motion deblurring, in: Proc. of the European Conf. on Computer...
  • N. Wiener

    Extrapolation, Interpolation, and Smoothing of Stationary Time Series

    (1964)
  • W.H. Richardson

    Bayesian-based iterative method of image restoration

    J. Opt. Soc. Am.

    (1972)
  • L. Yuan et al.

    Progressive inter-scale and intra-scale non-blind image deconvolution

    ACM Trans. Graph. (Proc. SIGGRAPH)

    (2008)
  • Y. Wang et al.

    A new alternating minimization algorithm for total variation image reconstruction

    SIAM J. Imag. Sci.

    (2008)
  • D. Krishnan, R. Fergus, Fast image deconvolution using hyper-Laplacian priors, in: Advances in Neural Information...
  • A. Levin, Y. Weiss, F. Durand, W.T. Freeman, Understanding and evaluating blind deconvolution algorithms, in: IEEE...
  • A. Gupta, N. Joshi, C.L. Zitnick, M. Cohen, B. Curless, Single image deblurring using motion density functions, in:...
  • S. Harmeling, M. Hirsch, B. Schölkopf, Space-variant single-image blind deconvolution for removing camera shake, in:...
  • M. Hirsch, C.J. Schuler, S. Sra, B. Schölkopf, S. Harmeling, Fast removal of non-uniform camear shake, in:...
  • O. Whyte, J. Sivic, A. Zisserman, J. Ponce, Non-uniform deblurring for shaken images, in: IEEE Conf. on Computer Vision...
  • Y. Tai et al.

    Richardson–Lucy deblurring for scenes under projective motion path

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2011)
  • J. Cai, H. Ji, C. Liu, Z. Shen, High-quality curvelet-based motion deblurring from an image pair, in: IEEE Conf. on...
  • J. Chen, L. Yuan, C.K. Tang, L. Quan, Robust dual motion deblurring, in: IEEE Conf. on Computer Vision and Pattern...
  • Cited by (23)

    • Learn from the past – sequentially one-to-one video deblurring network

      2021, Journal of Visual Communication and Image Representation
    • Multi-purpose bi-local CAT-based guidance filter

      2018, Signal Processing: Image Communication
      Citation Excerpt :

      We apply our bi-local CAT-based guidance filter to recover a sharp image from a blurry one. Like the methods in [26–29], we use a guidance image in the deblurring process. The guidance image can be a flash image, dark flash image, NIR image, etc.

    • Near-infrared fusion via a series of transfers for noise removal

      2018, Signal Processing
      Citation Excerpt :

      Since the captured NGI is noise free, the near-infrared imaging has been considered as a solution for high-quality image acquisition in low lighting conditions. Conventional near-infrared fusion models based on the gradient difference regularization (GDR) [3–5], multiresolution (MR) [6–8], and weighted least squares (WLS) [2] have failed in removing the noise and describing the details. The first reason is that the conventional fusion models are not suitable for overcoming the serious discrepancies in the edges and brightness between the NGI and VCI.

    • Robust penalty-weighted deblurring via kernel adaption using single image

      2016, Journal of Visual Communication and Image Representation
    • Variational Bayesian blind image deconvolution: A review

      2015, Digital Signal Processing: A Review Journal
      Citation Excerpt :

      Using image pairs with different properties facilitates blur estimation and helps handle saturated pixels and other camera imperfections. For instance, in [88] a near-infrared image is captured together with a visible blurred image and, in [89,90] a low exposure sharp but noisy image is used to improve the restoration results. If video is available, techniques can take into account the motion between frames [91–93] to tackle the deblurring problem.

    View all citing articles on Scopus
    View full text