Elsevier

Neurocomputing

Volume 466, 27 November 2021, Pages 69-79
Neurocomputing

Deep robust image deblurring via blur distilling and information comparison in latent space

https://doi.org/10.1016/j.neucom.2021.09.019Get rights and content

Abstract

Current deep deblurring methods pay main attention to learning a transferring network to transfer synthetic blurred images to clean ones. Though achieving significant performance on the training datasets, they still suffer from a weaker generalization capability from training datasets to others with different synthetic blurs, thus resulting in significantly inferior performance on testing datasets. In order to alleviate this problem, we propose a latent contrastive model, Blur Distilling and Information Reconstruction Networks (BDIRNet), to learn image prior and improve the robustness of deep deblurring. The proposed BDIRNet consists of a blur removing network (DistillNet) and a reconstruction network (RecNet). Two kinds of images with almost the same information but different qualities are input into DistillNet to extract identical structure information via contrast latent information and purify the perturbations from other unimportant information like blur. While the RecNet is utilized to reconstruct sharp images based on the extracted information. In addition, inside the DistillNet and RecNet, a statistical anti-interference distilling (SAID) and anti-interference reconstruction (SAIR) modules are proposed to further enhance the robustness of our methods, respectively. Extensive experiments on different datasets show that the proposed methods achieve improved and robust results compared to recent state-of-the-art methods.

Introduction

Images captured by cameras often suffer from unwanted blurs caused by camera shake [1], [2], object movement [3], [4] or out-of-focus [5], [6]. As one low-level problem, image deblurring has drawn considerable attention in the last decades. Traditional methods utilize a variety of constraints, assumptions and natural image priors to advance the deblurring research [7], [8], [9]. These methods often restrict to deblur images with several spatially-uniform blur kernels and involve expensive non-linear computation.

Compared with the optimization-based methods [10], [11], [12], learning-based methods have brought significant improvements recently [3], [13], [14], [15]. To mimic conventional “coarse-to-fine” optimization methods, these methods often restore sharp images at different processing levels in a pyramid. The first work based on this scheme is proposed by Nah et al. [3], who build a dynamic scene deblurring CNN with a multi-scale loss function with independent parameter for each level to demonstrate the ability of deep neural networks to remove dynamic blur. In order to improve the stability of the above model, Tao et al. [13] propose a SRN model, which modifies the above work by sharing parameters in each scale. Following the similar principle, Gao et al. [15] propose a parameter selective sharing strategy to train models, whose sub-networks both have independent and shared parameters.

Despite that significant efforts have been devoted to deep image deblurring without assuming any restricted blur kernel model, there still exists one major challenge in deep deblurring methods. As Fig. 1 (a) shows, almost all of the above variations of networks directly learn a transferring process from input synthetic blurry images to their corresponding sharp ones. However, the blurred images as well as the synthetic blurs in the testing set are often unseen during the training stage. A model learned in the training set is specialized to the blurred image of the training set and performs well in restoring their sharp images. When it is applied to differently blurred images, the trained model shows a weaker generalization capability. Therefore, methods based on the transferring scheme do not necessarily achieve robust performance on testing datasets.

In order to alleviate this problem, in this paper, a joint blur distilling and information reconstruction framework, BDIRNet, is devised to utilize the information distilling to remove the perturbations of blurs. Fig. 1 (b) illustrates of the proposed BDIRNet, which consists of two modules. The first one, DistillNet, is utilized to extract identical information from two kinds of images with different qualities. The distilling information extracted from the input is expected to retain the dominant information of the original images. The second module, RecNet, is utilized to reconstruct the sharp images with high qualities. In the training stage, the DistillNet and RecNet are finally combined into a framework to learn their weights. During the testing, BDIRNet takes only the blurred images as input to purify the information and reconstruct sharp images. Compared with previous methods, BDIRNet does not directly learn the transferring process from specially blurred image in the training stage. To improve the robustness of the proposed model, a statistical anti-interference distilling (SAID) and anti-interference reconstruction(SAIR) modules are created. Interference information are randomly added into the feature maps to improve the performance of DistillNet and reconstruction, respectively. All models are trained via a unified learning algorithm.

Overall, Our contributions are summarized as follows.

  • Firstly, we propose BDIRNet, a deep robust image deblurring framework, which contains two sub-networks to restore sharp images from the corresponding blurred ones. The first sub-network DistllNet extracts the dominant information of input. Through contrasting the latent space of image with different qualities, the proposed model purifies the information and avoid the perturbations of different blurs. The second sub-network RecNet reconstructs the sharp-version images from the extracted information.

  • Secondly, inside the DistillNet and RecNet, a statistical anti-interference distilling module, SAID, and anti-interference reconstruction module, SAIR, are proposed to further enhance the robustness of the method. In addition, a unified learning algorithm is proposed to simultaneously learn the parameters of proposed modules.

  • Thirdly, our BDIRNet outperforms the state-of-the-art models on two public datasets and real-world blurry images, which shows its superiority and robustness.

Section snippets

Related work

Approaches have been proposed for image deblurring, which can be roughly classified into two categories: geometry-based methods and deep learning methods.

BDIRNet

In this section, we first introduce the BDIRNet framework, including DistillNet and RecNet. Then two statistical anti-interference modules, SAID and SAIR, are represented. Finally, a unified learning algorithm to simultaneously learn the parameters of above modules is discussed.

Experiments

In this section, several experiments are conducted to evaluate the performance of the proposed method. We evaluate our method on the GoPro dataset [3] and Stereo blur dataset [4].

Conclusion

In this paper, we propose an end-to-end image deblurring model, BDIRNet, to restore sharp images from their blurred versions. The proposed latent contrastive model contains two sub networks, DistillNet and RecNet. In the training stage, sharp and blurred images are input into DistillNet to model deep prior information and extract similar features to represent the dominant information, which is then fed into RecNet to reconstruct sharp images. In order to extract more robust information and

CRediT authorship contribution statement

Wenjia Niu: Conceptualization, Methodology, Software, Data curation. Kaihao Zhang: Conceptualization, Methodology, Software, Data curation. Wenhan Luo: Writing - original draft. Yiran Zhong: Writing - original draft. Hongdong Li: Writing - original draft.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work is funded in part by the ARC Centre of Excellence for Robotics Vision (CE140100016), ARC-Discovery (DP 190102261) and ARC-LIEF (190100080) grants, as well as a research grant from Baidu on autonomous driving. The authors gratefully acknowledge the GPUs donated by NVIDIA Corporation. We thank all anonymous reviewers and editors for their constructive comments.

Wenjia Niu received the B.S. degree and Ph.D degree in electronic science and technology from Hebei University of Technology, Tianjin, China. She was a visiting Ph.D student at the Australian National University from 2017 to 2018. She is now waiting for an open opportunity to continue her short-term research work at the University of Melbourne. Her research interests include machine learning, computer vision, particularly neural network algorithm clustering analysis and low-level image

References (64)

  • S. Su et al.

    Deep video deblurring for hand-held cameras

  • S. Nah et al.

    Recurrent neural networks with intra-frame iterations for video deblurring

  • S. Nah et al.

    Deep multi-scale convolutional neural network for dynamic scene deblurring

  • S. Zhou et al.

    Davanet: Stereo deblurring with view aggregation

  • C. Tang et al.

    Defusionnet: defocus blur detection via recurrently fusing and refining multi-scale deep features

  • W. Zhao et al.

    Enhancing diversity of defocus blur detectors via cross-ensemble network

  • L. Xu et al.

    Two-phase kernel estimation for robust motion deblurring

    European Conference on Computer Vision (ECCV)

    (2010)
  • L. Xu et al.

    Unnatural l0 sparse representation for natural image deblurring

  • J. Pan et al.

    Deblurring text images via l0-regularized intensity and gradient prior

  • T. Hyun Kim et al.

    Dynamic scene deblurring

  • T. Hyun Kim et al.

    Segmentation-free dynamic scene deblurring

  • J. Pan et al.

    Soft-segmentation guided object motion deblurring

  • X. Tao et al.

    Scale-recurrent network for deep image deblurring

  • H. Zhang et al.

    Deep stacked hierarchical multi-patch network for image deblurring

  • H. Gao et al.

    Dynamic scene deblurring with parameter selective sharing and nested skip connections

  • Leeseungyong Chosunghyun

    Fast motion deblurring

    ACM Transactions on Graphics

    (2009)
  • T. Hyun Kim et al.

    Generalized video deblurring for dynamic scenes

  • J. Jia

    Single image motion deblurring using transparency

  • A. Levin

    Blind motion deblurring using image statistics

    Twenty-first Conference on Neural Information Processing Systems

    (2007)
  • S.K. Nayar et al.

    Motion-based motion deblurring

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2004)
  • L. Yang et al.

    A variational em framework with adaptive edge selection for blind motion deblurring

  • L. Chen et al.

    Blind image deblurring with local maximum gradient prior

  • J. Pan et al.

    Blind image deblurring using dark channel prior

  • H. Park et al.

    Joint estimation of camera pose, depth, deblurring, and super-resolution from a blurred image sequence

  • D. Lee, H. Park, I. Kyu Park, K. Mu Lee, Joint blind motion deblurring and depth estimation of light field, in:...
  • T.F. Chan et al.

    Total variation blind deconvolution

    IEEE Trans. Image Process.

    (1998)
  • Q. Shan et al.

    High-quality motion deblurring from a single image

    ACM Trans. Graphics

    (2008)
  • D. Krishnan et al.

    Fast image deconvolution using hyper-laplacian priors

    Twenty-third Conference on Neural Information Processing Systems

    (2009)
  • J. Sun et al.

    Learning a convolutional neural network for non-uniform motion blur removal

  • A. Chakrabarti, A neural approach to blind motion deblurring, in: European Conference on Computer Vision (ECCV),...
  • T.M. Nimisha, A. Kumar Singh, A.N. Rajagopalan, Blur-invariant deep learning for blind-deblurring,...
  • X. Xu et al.

    Motion blur kernel estimation via deep learning

    (2017)
  • Cited by (0)

    Wenjia Niu received the B.S. degree and Ph.D degree in electronic science and technology from Hebei University of Technology, Tianjin, China. She was a visiting Ph.D student at the Australian National University from 2017 to 2018. She is now waiting for an open opportunity to continue her short-term research work at the University of Melbourne. Her research interests include machine learning, computer vision, particularly neural network algorithm clustering analysis and low-level image enhancement.

    Kaihao Zhang is currently pursuing the Ph.D. degree with the College of Engineering and Computer Science, The Australian National University, Canberra, ACT, Australia. Prior to that, he received the M.Eng. degree in computer application technology from the University of Electronic Science and Technology of China, Chengdu, China, in 2016. He worked at the Center for Research on Intelligent Perception and Computing, National Lab of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China for two years and the Tencent AI Laboratory, Shenzhen, China for one year. His research interests focus on video analysis and facial recognition with deep learning.

    Wenhan Luo is currently working as a senior researcher in the Tencent AI Lab, China. His research interests include several topics in computer vision and machine learning, such as motion analysis (especially object tracking), image/video quality restoration, reinforcement learning. Before joining Tencent, he received the Ph.D. degree from Imperial College London, UK, 2016, M.E. degree from Institute of Automation, Chinese Academy of Sciences, China, 2012 and B.E. degree from Huazhong University of Science and Technology, China, 2009.

    Yiran Zhong received the M.Eng in information and electronics engineering in 2014 with the first class honor from The Australian National University, Canberra, Australia. After two years of research assistant, he becomes a PhD student in the College of Engineering and Computer Science, The Australian National University, Canberra, Australia and Data61, CSIRO, Canberra, Australia. He won the ICIP Best Student Paper Award in 2014. His current research interests include geometric computer vision, machine learning and deep learning.

    Hongdong Li is currently a Professor with the Computer Vision Group of ANU (Australian National University). He is also a Chief Investigator for the Australia ARC Centre of Excellence for Robotic Vision (ACRV). His research interests include 3D vision reconstruction, structure from motion, multi-view geometry, as well as applications of optimization methods in computer vision. Prior to 2010, he was with NICTA Canberra Labs working on the “Australia Bionic Eyes” project. He is an Associate Editor for IEEE T-PAMI, and served as Area Chair in recent year ICCV, ECCV and CVPR. He was a Program Chair for ACRA 2015 - Australia Conference on Robotics and Automation, and a Program Co-Chair for ACCV 2018 - Asian Conference on Computer Vision. He won a number of prestigious best paper awards in computer vision and pattern recognition, and was the receipt for the CVPR Best Paper Award in 2012 and the Marr Prize Honorable Mention in 2017.

    View full text