Blind image deblurring via coupled sparse representation

doi:10.1016/j.jvcir.2014.02.003

Journal of Visual Communication and Image Representation

Volume 25, Issue 5, July 2014, Pages 814-821

https://doi.org/10.1016/j.jvcir.2014.02.003 Get rights and content

Highlights

•
A blind image deblurring algorithm proposed by learning coupled dictionaries with sparse coding.
•
Coupled sparse representation of blurry image and its latent patch investigated.
•
Unified framework for optimization and fast learning scheme proposed for learning dictionaries.
•
Efficiency of the algorithm evaluated on standard images against state-of-the-art algorithms.

Abstract

The problem of blind image deblurring is more challenging than that of non-blind image deblurring, due to the lack of knowledge about the point spread function in the imaging process. In this paper, a learning-based method of estimating blur kernel under the $ℓ_{0}$ regularization sparsity constraint is proposed for blind image deblurring. Specifically, we model the patch-based matching between the blurred image and its sharp counterpart via a coupled sparse representation. Once the blur kernel is obtained, a non-blind deblurring algorithm can be applied to the final recovery of the sharp image. Our experimental results show that the visual quality of restored sharp images is competitive with the state-of-the-art algorithms for both synthetic and real images.

Introduction

Blurred images occur when the image acquiring process is influenced by the relative movement between the camera and objects during the exposure time [20]. A clean image from a blurred image can be obtained through solving the problem of image deblurring, which is one of challenging problems in image restoration. Image deblurring has been extensively studied in recent years [4], [21].

In general, the degradation process of the blurred images is often modelled as a convolution followed by a noising process, $Y = k \otimes X + n$ where Y is a blurred image, i.e. observed image, X is a sharp image, i.e. unknown latent (clean/sharp) image, $k$ is a spatial-invariant kernel, i.e. point spread function (PSF) and $n$ is independent white Gaussian noise added to the convoluted image. The problem of image deblurring is to recover the latent image X from the observed blurred one Y, which is also regarded as a deconvolution processing.

The process of image deblurring falls into two categories. If the blur kernel $k$ is known or well estimated, then the restoration of X from Y is considered as a non-blind deconvolution problem. If, on the other hand, there is very little or no information about the blur kernel $k$ , the problem is regarded as blind deblurring. Unfortunately, in most of the real-world cases, we have no knowledge of the exact kernel $k$ for a blurred image. Thus, most of the time, we have to face the challenging blind deconvolution problems [2], [24], [26]. This results in well-known,, but ill-posed and difficult problems. In this paper, we focus on the issue of blind image deblurring.

Recently, many methods have been proposed to solve this kind of problems [5]. These methods can be roughly divided into two classes based on the ways of how blur kernels are identified or learned. In the first class, a blur kernel is identified independently from a single blurred image [14]. In the second class, both the blur kernel and latent (clean or sharp) image are estimated simultaneously via a learning process [5], [6]. In most of the methods of the second class, a prior knowledge [9], [14], [17] about both latent image and blur kernel is exploited, for example in total variation and Bayesian paradigms [2], [24]. Once the blur kernel is well estimated, the problem can be simplified into a non-blind deconvolution issue, which can be effectively solved by the popular sparse representation methods [3], [29], [31].

However, many hidden mappings between the blurred images and the latent images are not well exploited during estimating blur kernels [8], [10]. For example, a motion blurred image [10] usually retains information about motion which gives us clues to recover motion from this blurred image by parameterizing the blurring model. The sparse representation of image is more helpful in estimating an appropriate blurring kernel and its relevant latent image [8]. The advancements in sparse representation of signals [7], [12] and learning their sparsifying dictionaries [1] make it more effective in solving the image restoration problems.

In this paper, we focus on the spatially invariant blurring issue. Our goal is to recover the true blur kernel based on the assumption that there exists a coupled sparse representation between the blurred and sharp (latent) image patch pairs under their own dictionaries, respectively. This type of hidden mappings is helpful in estimating the unknown blur kernel. Then, with the estimated blur kernel, we can recover the latent image by a variety of non-blind deconvolution methods. The main contribution of this paper to the literature is summarized as the integration of the sparse representation of blurred image and its corresponding latent patch into a unified framework for optimization. Specifically, in addition to requiring that the coding coefficients of each local patch are sparse, we also enforce the compatibility of sparse representations between the blurred and latent image patch pair with respect to their own dictionaries. Once the blur kernel is learned, we restore the latent clean image by imposing a sparse prior regularization over the image derivatives as done in [18]. This regularization allows a robust recovery even for a possibly lower accurate kernel estimate and is an effective way to solve the non-blind deconvolution issue.

This paper is organized as follows. In Section 2, we briefly review the related works on the approaches of blind image delurring. The blind image deblurring based sparsity prior is proposed in Section 3. The extensive experimental results are provided in Section 4. The conclusion is drawn in Section 5.

Section snippets

Related works

Due to its inherently ill-posed problem, a blind image deblurring process needs to be regularized by image priors for better solution. We briefly review a variety of approaches on blind image delurring in the following.

Generally, natural image statistics can be used as appropriate image priors, e.g., in constraining the output of local derivative operators. Among many priors, the Gaussian smoothness penalty is the simplest one, which has been widely used in blind image deconvolution [9], [14],

Conceptual framework

Consider a patch $x$ extracted from a latent image X and assume that $x$ can be well represented as a sparse linear combination over an appropriate dictionary $D$ . This dictionary $D$ may be trained from those true sample patches extracted from training images. In general, we assume $x \approx D α, {‖α‖}_{0} ≪ m with α \in R^{m} .$

Such a sparse representation can be helpful in image processing tasks [7], [12], [30]. For example, the sparse representation approach has been widely used in image denoising [13]. The idea is based on the

Experimental results

In this section, we compare our proposed method to other blind deblurring algorithms for which there is public code available, such as Fergus et al.’s method [14], Shan et al.’s method [26] and Hu et al.’s method [16].¹

First, we use several synthesized blurred images to verify the efficiency and performance of the method proposed in Section 3. Then, several real blurred images without the ground truth are used to assess

Conclusion

In this paper, a novel deblurring method based on coupled sparse representation is proposed for blind image deblurring. Except for the use of sparse representation of image patches, we also ensure that the sparse coefficients comply with the strict compatibility between the latent image and blurred one. After obtaining the estimated blur kernel, we further recover the sharp image by imposing a sparsity inducing constraint on images so that the recovered sharp image is more robust to ringing

Acknowledgments

This work is supported under the Grant DP130100364 from the Australian Research Council (ARC). The work of the first and fourth authors is supported by NSF China under Grants No. 61201392.

References (31)

O. Bryt et al.
Compression of facial images using the K-SVD algorithm
J. Visual Commun. Image Represent.
(2008)
M. Aharon et al.
The K-SVD: an algorithm for designing of over complete dictionaries for sparse representation
IEEE Trans. Signal Process.
(2006)
S.D. Babacan et al.
Variational Bayesian blind deconvolution using a total variation prior
IEEE Trans. Image Process.
(2009)
A. Beck et al.
Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems
IEEE Trans. Image Process.
(2009)
M. Ben-Ezra et al.
Motion-based motion deblurring
IEEE Trans. Pattern Anal. Mach. Intell.
(2004)
T.E. Bishop et al.
Blind Image Deconvolution: Theory and Applications
(2007)
M. Bronstein et al.
Blind deconvolution of images using optimal sparse representations
IEEE Trans. Image Process.
(2005)
J. Cai, H. Ji, C. Liu, Z. Shen, Blind motion deblurring from a single image using sparse approximation, in: Proceedings...
S. Cho et al.
Fast motion deblurring
ACM Trans. Graphics
(2009)
S. Dai, Y. Wu, Motion from blur, in: Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR’08), IEEE...

I. Daubechies et al.

Iteratively reweighted least squares minimization for sparse recovery

Commun. Pure Appl. Math.

(2010)

D. Donoho et al.

Stable recovery of sparse overcomplete representations in the presence of noise

IEEE Trans. Inf. Theory

(2006)

M. Elad et al.

Image denoising via sparse and redundant representations over learned dictionaries

IEEE Trans. Image Process.

(2006)

R. Fergus, B. Singh, A. Hertzmann, S.T. Roweis, W. Freeman, Removing camera shake from a single photograph, in: Proc....

A. Gupta, N. Joshi, L. Zitnick, M. Cohen, B. Curless, Single image deblurring using motion density functions, in:...

Cited by (18)

Blind image deblurring using group sparse representation
2020, Digital Signal Processing: A Review Journal
Citation Excerpt :
This method assumes that sharp and blurred images have the same sparse coefficients. Yin et al. [21] matched image patches with a coupled sparse representation for blind deblurring. Tang et al. [23] combined the external patch priors described by the expected patch log likelihood and a sparse representation for kernel estimation.
We propose a blind image deblurring model based on the group sparse representation (GSR) prior. The motivation for this work is that adjacent similar patches of images and blur kernels favor clear images and ground-truth kernels over blurred images and poor kernels. Therefore, we impose GSR constraints on similar patches based on images and kernels to ensure the sparsity of the intermediate latent images and kernels. In addition, to effectively preserve the main edges in the image, we impose an $L_{0}$ -regularized gradient prior on image restoration. Moreover, a flexible non-convex weighted $L_{p}$ -norm minimization is applied to the GSR and an adaptive dictionary is used for each similar group. To optimize our model, an effective optimization algorithm based on a generalized soft-thresholding algorithm and half-quadratic splitting strategy is developed. Several analyses verify the validity of the GSR prior. Experiments show that our method achieves satisfactory deblurring effects in several specific domains and is robust in both deblurring and denoising tasks. The results of qualitative and quantitative experiments indicate that our algorithm performs favorably with respect to state-of-the-art algorithms.
Multi-frame motion deblurring using coded exposure imaging with complementary fluttering sequences
2020, Optics and Laser Technology
Citation Excerpt :
For instance, inverse filtering and Richardson-Lucy (RL) [1,2] methods are simple and classical solutions. In recent years, researchers also proposed auxiliary constraint factors, such as sparse prior [3,4], Total Variation (TV) constraint[5,6], hyper-laplacian priors [7,8] to improve the quality of restored image. On the other hand, the blind deblurring solution is to estimate the PSF and sharp image simultaneously without the information of the blur kernel, which is a more challenging task.
Motion blurring is one of the most common but unpleasant phenomenons along with the camera imaging process, resulting in image resolution degradation. Coded exposure technique is an effective computational imaging approach to handle with this problem by introducing extra shutter control within the exposure time and help the deblurring problem become well-posed. In this paper, a multi-frame motion deblurring framework is presented using coded exposure imaging with complementary fluttering sequences. A set of complementary sequences are well designed to compensate for the spatial losses due to motion blurring in each frame based on the complementary theory of Golay matrix pair. Hadamard matrix is served as the Golay sequence pair to generate complementary fluttering sequences to capture multi-frame images for the following deblurring processing. The joint modulation transfer function (MTF) of the complementary code sequences has a more flat spectrum curve with smaller variance value and higher MTF value. For multi-frame deblurring implementation, a unified multi-image deblurring framework is performed by introducing Total Variation (TV) regularization prior with each selected code sequence in the complementary set integrated in the energy function. A real imaging system is built and various groups of synthetic and real-world imaging experiments are carried out to verify the effectiveness of the method. Experimental results demonstrate that the deblurrring images obtained by the proposed approach are of high quality with improved objective image evaluation values and better negative artifacts suppression.
Single image blind deblurring based on the fractional-order differential
2019, Computers and Mathematics with Applications
Citation Excerpt :
The third method is to estimate sharp image and blur kernel simultaneously by a joint minimization problem with some regularizations/priors on both them. The regularizations/priors include total variation regularization [7–10], high-order total variation regularization [11–13], non-local regularization [14–17], fractional-order regularization [18–25], sparsity regularization [26–32] and low rank prior [33–35]. In this paper, we propose a novel blind deblurring method by integrating the fractional-order total variation of latent image and blur kernel as the regularization term.
Image deblurring is a fundamental problem in image processing. When the blur type is unknown, the problem becomes more challenging, which is called blind deblurring. In this paper, we propose a novel variational method for single image blind deblurring based on the fractional-order differential, which can overcome the staircase effect produced by the total variation regularization and alleviate the ringing artifact in deblurring. The fractional-order gradient fidelity term is added in the cost functional to improve the restoration. Moreover, we make use of the edge detect function with the fractional-order gradient to preserve sharp edges, and the Bregman iteration to reconstruct more structures. The primal–dual algorithm is developed to solve the proposed model. Numerical experiments show that our method is able to get the sharp image without the staircase effect and correctly estimate the unknown blur kernel.
Blind deblurring with sparse representation via external patch priors
2018, Digital Signal Processing: A Review Journal
Citation Excerpt :
Later, more researches are carried out under this sparse-based framework. For example, Yin et al. [3] present a coupled sparse representation to match patches between the blurred image and its sharp counterpart, aiming to enhance the intrinsic structures of the latent image. Ma et al. [4] propose an optimal strategy to refine the learned dictionary by a structured sparse model selection.
In this paper, a blind image deblurring method is proposed using sparse representation with external patch priors. Different from traditional sparse-based methods that employ only internal priors from blurred images, additional external information is adopted to reconstruct latent images. In details, the Expected Patch Log Likelihood (EPLL) is introduced as a useful tool to describe external patch priors with a pre-trained Gaussian mixture model. With a set of operations, the EPLL is subsequently incorporated as a regularization term into the existing sparse-based deblurring model. Meanwhile, the dictionary is also carefully designed for each patch of the latent image, where atoms are obtained from the covariance matrix of the corresponding Gaussian component. A deblurring framework is further presented along with our sparse-based model. The solutions are respectively given to efficiently optimize the latent image and the blur kernel with an iterative procedure. The experiments demonstrate that our proposed algorithm achieves a competitive performance compared with the state-of-the-arts. Especially, it not only can obtain more accurate kernels for the deblurring, but also outperforms in noise reduction and artifact suppression for the restored images.
Robust penalty-weighted deblurring via kernel adaption using single image
2016, Journal of Visual Communication and Image Representation
Image blind deconvolution is well known as a challenging, ill-posed problem due to the uncertainty of the blur kernel and the noise condition. Based on our observations, blind deconvolution algorithms tend to generate disconnected and noisy blur kernels, which would yield a serious ringing effect in the restored image if the input image is noisy. Therefore, there is still room for further improvement, especially for noisy images captured under poor illumination conditions. In this paper, we propose a robust blind deconvolution algorithm by adopting a penalty-weighted anisotropic diffusion prior. On one hand, the anisotropic diffusion prior effectively eliminates the discontinuity in the blur kernel caused by the noisy input image during the process of kernel estimation. On the other hand, the weighted penalizer reduces the speckle noise of the blur kernel, thus improving the quality of the restored image. The effectiveness of the proposed algorithm is verified by both synthetic and real images with defocused or motion blur.
An Efficient Image Deblurring Network with a Hybrid Architecture
2023, Sensors

View all citing articles on Scopus

View full text

Blind image deblurring via coupled sparse representation

Highlights

Abstract

Introduction

Section snippets

Related works

Conceptual framework

Experimental results

Conclusion

Acknowledgments

J. Visual Commun. Image Represent.

The K-SVD: an algorithm for designing of over complete dictionaries for sparse representation

IEEE Trans. Signal Process.

Variational Bayesian blind deconvolution using a total variation prior

IEEE Trans. Image Process.

Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems

IEEE Trans. Image Process.

Motion-based motion deblurring

IEEE Trans. Pattern Anal. Mach. Intell.

Blind Image Deconvolution: Theory and Applications

Blind deconvolution of images using optimal sparse representations

IEEE Trans. Image Process.

Fast motion deblurring

ACM Trans. Graphics

Iteratively reweighted least squares minimization for sparse recovery

Commun. Pure Appl. Math.

Stable recovery of sparse overcomplete representations in the presence of noise

IEEE Trans. Inf. Theory

Image denoising via sparse and redundant representations over learned dictionaries

IEEE Trans. Image Process.