Journal of Visual Communication and Image Representation
Robust blind motion deblurring using near-infrared flash image
Introduction
When taking photographs under low-light conditions with a hand-held camera, a long shutter speed is required for adequate exposure. However, the relative motion between the camera and the scene during exposure is ineluctable and results in a blurry image. Mathematically, the blur image caused by camera shake can be considered as a linear combination of multiple clear images captured from slightly different viewpoints. Restoration of images degraded by motion blur is a hotly debated topic in both image processing and computer vision. Since the high-frequency information is lost and no prior is known about the camera shake, recovering the latent image from the blurry observation is a challenging task.
In recent years, many researchers worked on the blind camera shake removal from a single image using different kinds of priors on image statistics [1], [2], [3]. In [1], the heavy-tailed distribution of image gradients are fitted by the zero-mean mixture-of-Gaussians model, then the variational Bayesian framework is used to approximate the MAP solution efficiently. Shan et al. [2] model the high-order derivatives of image noise and propose the sparse prior for the blur kernel and latent image estimation. Jia [4] analyzes the relationship between the image degradation and the transparency on the object boundary, and estimates the blur kernel by image matting. The approach in [5] maximizes the sparsity and spatial smooth of the blur kernel under curvelet system. In [6], [7], [8], the large-scale edges are selected to perform high-quality kernel estimation. Cho and Lee [6] choose simple Gaussian priors in the optimization, which can be accelerated using Fast Fourier Transform (FFT). The blind deconvolution problem in [9] is addressed by the split Bregman iteration to obtain high-quality image with low computational cost.
Given the estimated blur kernel, the image restoration is known as the non-blind deconvolution. Some traditional algorithms, such as Wiener filter [10] and Richardson–Lucy (RL) deconvolution [11], are widely used and known to be simple and efficient. However, the estimation suffers from unpleasant noise and ringing artifacts. In [12], the edge-preserving Bilateral Richardson–Lucy (BRL) deconvolution is performed with coarse-to-fine scheme. Furthermore, various regularization techniques, such as the Total-Variation (TV) regularization [13], hyper-Laplacian prior [14] and adaptive regularization [15], are explored to suppress the undesirable artifacts.
Commonly, most of the previous works assume the blur to be spatially invariant. However, as discussed in [16], the uniform blur assumption is often violated in practice. In [17], [18], [19], [20], the authors introduce different kinds of blur image formulations to model the imaging process with non-uniform camera shake. Tai et al. [21] modify the RL algorithm with regularization term to incorporate the spatially variant blur. Since blind deblurring from a single image is highly ill-posed, all the algorithms make strong prior assumptions about the blur kernel and image. However, natural images do not always meet the strong priors and the lost of image information is inevitable in the deblurring process.
To overcome the ambiguity of blind deblurring from one single image, many computational imaging techniques capturing multiple images to enhance the capabilities of digital photography are proposed. In [22], [23], [24], the blurred image pair obtained by consecutive shooting with same exposure time are assumed to be well-aligned. However, accurate automatic alignment of images blurred by different kernels is difficult to achieve by existing methods. Yuan et al. [25] use a pair of blurred and noisy images for deblurring. The blur kernel is estimated based on the sharp edges selected from the denoised image. The authors further propose an alignment approach for a blurred/no-blurred image pair in [26]. In [20], Whyte et al. extend the method in [25] to the non-uniform blur. Although the multiple images are complementary to each other, the high-frequency information of the scene has a certain amount of loss in the blurred image and the denoised result. Recently, the method in [27] recovers sharp image with the help of visible flash image, while it does not take the non-uniform blur into consideration.
Another group of computational photography methods employ specific hardware to control the capturing process. A high-resolution still camera and a low-resolution video camera are combined to form a hybrid camera system in [28], [29]. The motion blur of high-resolution image is estimated from the image sequence by optical flow. The inertial sensors are connected to the camera in [30] to capture the camera motion during exposure. These aided hardware systems provide more information on camera shake and can handle more complex motions.
Inspired by the idea of introducing additional information to assist the blind deblurring, we propose a method to deblur a photography by using the high frequency texture of its corresponding NIR-flash image. As shown in Fig. 1(b), the NIR image is sharp and captures the high-frequency details of the scene, which brings significant benefits to solve the highly ill-posed problem. We exploit the correlation of the image pair captured at visible and NIR wavebands, and propose a blind deblurring method with the application of NIR gradient constraint. Compared with other methods, our deblurring framework is based on the general blur model, and is able to provide accurate kernel estimation and superior image restoration with both spatially invariant and spatially varying blur.
The rest of this paper is organized as follows. The main properties of NIR flash image are introduced in Section 2. In Section 3, we describe the general blur model, and then the NIR flash deblurring problem is formulated into a MAP framework. Section 4 presents the blind image deblurring algorithm using the NIR flash gradient constraint for uniform blur. The algorithm is further extended to spatially variant blur in Section 5. The deblurring performance is evaluated and compared with other methods in Section 6. Finally, the conclusions and the discussions of current limitations are drawn in Section 7.
Section snippets
NIR image gradient constraint
Near-infrared light has the wavelength range of 750 nm to 1100 nm, which can be sensed by digital camera, but invisible to the human eyes. Since the physical differences in the visible and NIR wavebands, the RGB and NIR images have different intensities and frequency contents. NIR images usually contain more texture details and have higher contrast. In recent years, the NIR images are introduced to bring benefits to computational photography and computer vision applications, e.g., image
Problem formulation
In this section, we formulate the image blur model of spatially invariant and spatially varying camera shake. We unify the NIR image gradient constraint into a MAP formulation for kernel estimation and sharp image restoration.
Uniform deblurring algorithm
In this section, we first consider the uniform deblurring for shaken images. Our algorithm is similar to [27], but has a series of changes in the formulation and optimization to effectively fuse the high-frequency information in the NIR-flash image into the blind deblurring process. We divide the problem into two parts: blur kernel estimation and image restoration. Firstly, we convert the blurred image to YUV color space, and denote its luminance (Y channel) and chrominance components (U and V
Extension to non-uniform deblurring
In this section, we extend the deblurring algorithm described above to the non-uniform camera shake. In our experiments, we found that exactly adopt the same approach as for the uniform deblurring cannot get a satisfactory result for the non-uniform case. The reason should be that the iterative optimizations are non-convex and seriously depend on the accuracy of initialization. Just good initial value can ensure the iterations converge along the right direction. Therefore, the key point of this
Experiments
In this section, we present the results of our approach and do comprehensive comparisons against some state-of-the-art algorithms for blind deblurring. The performance of the algorithm is evaluated on both synthetic and real captured images considering spatially invariant as well as spatially variant blur models.
Conclusions
In this paper, we captured a well-aligned blurred/NIR-flash image pair, and exploited the correlation of images captured at different wavelengths. To utilize the edge structure captured in the NIR-flash image, the NIR gradient constraint was introduced as an image regularizer to compensate the attenuation of high-frequency information in blind deblurring. For both spatially invariant and spatially varying blur, we evaluated our approach on the synthetic and real data. Our approach was
Acknowledgments
This research is supported by the National Basic Research Program of China (Grant No. 2010CB731800), the Foundation for Innovative Research Groups of the National Natural Science Foundation of China (Grant No. 60921001), and the Key Project of NSFC (No. 61120106003).
References (50)
- et al.
Total variation blind deconvolution employing split Bregman iteration
J. Visual Commun. Image Represent.
(2012) - et al.
High-quality non-blind image deconvolution with adaptive regularization
J. Visual Commun. Image Represent.
(2011) - et al.
Two motion-based images are better than one
Pattern Recogn. Lett.
(2005) - et al.
Total variation minimizing blind deconvolution with shock filter reference
Image Vision Comput.
(2008) - et al.
Removing camera shake from a single photograph
ACM Trans. Graph. (Proc. SIGGRAPH)
(2006) - et al.
High-quality motion deblurring from a single image
ACM Trans. Graph. (Proc. SIGGRAPH)
(2008) - D. Krishnan, R. Fergus, Blind deconvolution using a normalized sparsity measure, in: IEEE Conf. on Computer Vision and...
- J. Jia, Single image motion deblurring using transparency, in: IEEE Conf. on Computer Vision and Pattern Recognition...
- J. Cai, H. Ji, C. Liu, Z. Shen, Blind motion deblurring from a single image using sparse approximation, in: IEEE Conf....
- et al.
Fast motion deblurring
ACM Trans. Graph. (Proc. SIGGRAPH Asia)
(2009)
Extrapolation, Interpolation, and Smoothing of Stationary Time Series
Bayesian-based iterative method of image restoration
J. Opt. Soc. Am.
Progressive inter-scale and intra-scale non-blind image deconvolution
ACM Trans. Graph. (Proc. SIGGRAPH)
A new alternating minimization algorithm for total variation image reconstruction
SIAM J. Imag. Sci.
Richardson–Lucy deblurring for scenes under projective motion path
IEEE Trans. Pattern Anal. Mach. Intell.
Cited by (23)
Learn from the past – sequentially one-to-one video deblurring network
2021, Journal of Visual Communication and Image RepresentationDiagnostics measure for roller bearings based on variable moiré gratings
2020, Optics and Lasers in EngineeringMulti-purpose bi-local CAT-based guidance filter
2018, Signal Processing: Image CommunicationCitation Excerpt :We apply our bi-local CAT-based guidance filter to recover a sharp image from a blurry one. Like the methods in [26–29], we use a guidance image in the deblurring process. The guidance image can be a flash image, dark flash image, NIR image, etc.
Near-infrared fusion via a series of transfers for noise removal
2018, Signal ProcessingCitation Excerpt :Since the captured NGI is noise free, the near-infrared imaging has been considered as a solution for high-quality image acquisition in low lighting conditions. Conventional near-infrared fusion models based on the gradient difference regularization (GDR) [3–5], multiresolution (MR) [6–8], and weighted least squares (WLS) [2] have failed in removing the noise and describing the details. The first reason is that the conventional fusion models are not suitable for overcoming the serious discrepancies in the edges and brightness between the NGI and VCI.
Robust penalty-weighted deblurring via kernel adaption using single image
2016, Journal of Visual Communication and Image RepresentationVariational Bayesian blind image deconvolution: A review
2015, Digital Signal Processing: A Review JournalCitation Excerpt :Using image pairs with different properties facilitates blur estimation and helps handle saturated pixels and other camera imperfections. For instance, in [88] a near-infrared image is captured together with a visible blurred image and, in [89,90] a low exposure sharp but noisy image is used to improve the restoration results. If video is available, techniques can take into account the motion between frames [91–93] to tackle the deblurring problem.