Abstract
Non-blind image deconvolution (NBID) is about restoring a latent sharp image from a blurred one, given an associated blur kernel. Most existing deep neural networks for NBID are trained over many ground truth (GT) images, which limits their applicability in practical applications such as microscopic imaging and medical imaging. This paper proposes an unsupervised deep learning approach for NBID which avoids accessing GT images. The challenge raised from the absence of GT images is tackled by a self-supervised reconstruction loss that approximates its supervised counterpart well. The possible errors of blur kernels are addressed by a self-supervised prediction loss based on intermediate samples as well as an ensemble inference scheme based on kernel perturbation. The experiments show that the proposed approach provides very competitive performance to existing supervised learning-based methods, no matter under accurate kernels or erroneous kernels.
Y. Quan—Is also with Pazhou Lab, Guangzhou 510335, China. He would like to thank the support in part by National Natural Science Foundation of China under Grant 61872151 and in part by Natural Science Foundation of Guangdong Province under Grant 2022A1515011755.
H. Ji—Would like thank the support in part by Singapore MOE AcRF under Grant R-146-000-315-114.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Al-Kofahi, Y., Zaltsman, A., Graves, R., Marshall, W., Rusu, M.: A deep learning-based algorithm for 2-d cell segmentation in microscopy images. BMC Bioinf. 19(1), 1–11 (2018)
Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Neural Netw. Learn. Syst. 33(5), 898–916 (2010)
Bigdeli, S.A., Jin, M., Favaro, P., Zwicker, M.: Deep mean-shift priors for image restoration. In: Advances in Neural Information Processing Systems, pp. 763–772 (2017)
Cai, J.F., Ji, H., Liu, C., Shen, Z.: High-quality curvelet-based motion deblurring from an image pair. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1566–1573 (2009)
Chen, D., Tachella, J., Davies, M.E.: Equivariant imaging: learning beyond the range space. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021)
Chen, G., Zhu, F., Ann Heng, P.: An efficient statistical method for image noise level estimation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 477–485 (2015)
Chen, M., Quan, Y., Pang, T., Ji, H.: Nonblind image deconvolution via leveraging model uncertainty in an untrained deep neural network. Int. J. Comput. Vision 130, 1770–789 (2022). https://doi.org/10.1007/s11263-022-01621-9
Cho, S., Lee, S.: Fast motion deblurring. In: Proceedings of the ACM SIGGRAPH Asia, pp. 1–8 (2009)
Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)
Dong, J., Roth, S., Schiele, B.: Deep wiener deconvolution: wiener meets deep learning for image deblurring. In: Advances in Neural Information Processing Systems, vol. 33 (2020)
Dong, J., Roth, S., Schiele, B.: Learning spatially-variant map models for non-blind image deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4886–4895 (2021)
Dong, W., Wang, P., Yin, W., Shi, G.: Denoising prior driven deep neural ketwork for image restoration. IEEE Trans. Neural Netw. Learn. Syst. 41(10), 2305–2318 (2019)
Dong, W., Zhang, L., Shi, G., Li, X.: Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process. 22(4), 1620–1630 (2012)
Eboli, T., Sun, J., Ponce, J.: End-to-end interpretable learning of non-blind image deblurring. In: Proceedings of the European Conference on Computer Vision (2020)
Folberth, J., Becker, S.: Efficient adjoint computation for wavelet and convolution operators [lecture notes]. IEEE Sig. Process. Mag. 33(6), 135–147 (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)
Hendriksen, A.A., Pelt, D.M., Batenburg, K.J.: Noise2inverse: Self-supervised deep convolutional denoising for tomography. IEEE Trans. Comput. Imaging 6, 1320–1335 (2020)
Ji, H., Wang, K.: Robust image deblurring with an inaccurate blur kernel. IEEE Trans. Image Process. 21(4), 1624–1634 (2011)
Jin, M., Roth, S., Favaro, P.: Noise-blind image deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3834–3842 (2017)
Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-laplacian priors. Adv. Neural Inf. Process. Syst. 22, 1033–1041 (2009)
Kruse, J., Rother, C., Schmidt, U.: Learning to push the limits of efficient FFT-based image deconvolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4586–4594 (2017)
Lai, W.S., Huang, J.B., Hu, Z., Ahuja, N., Yang, M.H.: A comparative study for single image blind deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1709 (2016)
Levin, A., Weiss, Y., Durand, F., Freeman, W.T.: Efficient marginal likelihood optimization in blind deconvolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2657–2664. IEEE (2011)
Li, W., Zhang, J., Dai, Q.: Exploring aligned complementary image pair for blind motion deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 273–280 (2011)
Lim, S., Park, H., Lee, S.E., Chang, S., Sim, B., Ye, J.C.: Cyclegan with a blur kernel for deconvolution microscopy: optimal transport geometry. IEEE Trans. Comput. Imaging 6, 1127–1138 (2020)
Meinhardt, T., Moller, M., Hazirbas, C., Cremers, D.: Learning proximal operators: using denoising networks for regularizing inverse imaging problems. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1781–1790 (2017)
Michaeli, T., Irani, M.: Blind deblurring using internal patch recurrence. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 783–798. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10578-9_51
Nan, Y., Ji, H.: Deep learning for handling kernel/model uncertainty in image deconvolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2388–2397 (2020)
Nan, Y., Quan, Y., Ji, H.: Variational-EM-based deep learning for noise-blind image deblurring. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3626–3635 (2020)
Nayar, S.K., Ben-Ezra, M.: Motion-based motion deblurring. IEEE Trans. Pattern Anal. Mach. Intell. 26(6), 689–698 (2004)
Pan, J., Sun, D., Pfister, H., Yang, M.H.: Blind image deblurring using dark channel prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1628–1636 (2016)
Pang, T., Quan, Y., Ji, H.: Self-supervised bayesian deep learning for image recovery with applications to compressive sensing. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12356, pp. 475–491. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58621-8_28
Pang, T., Zheng, H., Quan, Y., Ji, H.: Recorrupted-to-recorrupted: unsupervised deep learning for image denoising. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2043–2052 (2021)
Perrone, D., Favaro, P.: Total variation blind deconvolution: the devil is in the details. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2909–2916 (2014)
Pronina, V., Kokkinos, F., Dylov, D.V., Lefkimmiatis, S.: Microscopy image restoration with deep wiener-kolmogorov filters. In: Proceedings of the European Conference on Computer Vision (2020)
Quan, Y., Ji, H., Shen, Z.: Data-driven multi-scale non-local wavelet frame construction and image recovery. J. Sci. Comput. 63(2), 307–329 (2015)
Ren, D., Zuo, W., Zhang, D., Xu, J., Zhang, L.: Partial deconvolution with inaccurate blur kernel. IEEE Trans. Image Process. 27(1), 511–524 (2017)
Ren, W., et al.: Deep non-blind deconvolution via generalized low-rank approximation. Adv. Neural Inf. Process. Syst. 31, 297–307 (2018)
Schmidt, U., Jancsary, J., Nowozin, S., Roth, S., Rother, C.: Cascades of regression tree fields for image restoration. IEEE Trans. Neural Netw. Learn. Syst. 38(4), 677–689 (2015)
Soltanayev, S., Chun, S.Y.: Training deep learning based denoisers without ground truth data. In: Advances in Neural Information Processing Systems (2018)
Sun, L., Cho, S., Wang, J., Hays, J.: Edge-based blur kernel estimation using patch priors. In: Proceedings of the IEEE International Conference on Computational Photography, pp. 1–8. IEEE (2013)
Vasu, S., Maligireddy, V.R., Rajagopalan, A.: Non-blind deblurring: handling kernel uncertainty with CNNs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3272–3281 (2018)
Wang, W., Li, J., Ji, H.: Self-supervised deep image restoration via adaptive stochastic gradient langevin dynamics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1989–1998 (2022)
Wang, Z., Wang, Z., Li, Q., Bilen, H.: Image deconvolution with deep image and kernel priors. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (2019)
Whyte, O., Sivic, J., Zisserman, A.: Deblurring shaken and partially saturated images. Int. J. Comput. Vision 110(2), 185–201 (2014). https://doi.org/10.1007/s11263-014-0727-3
Xia, Z., Chakrabarti, A.: Training image estimators without image ground-truth. In: Advances in Neural Information Processing Systems (2019)
Xu, J., Zhang, L., Zuo, W., Zhang, D., Feng, X.: Patch group based nonlocal self-similarity prior learning for image denoising. In: Proceedings of the IEEE international Conference on Computer Vision, pp. 244–252 (2015)
Xu, L., Jia, J.: Two-phase kernel estimation for robust motion deblurring. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 157–170. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15549-9_12
Xu, L., Zheng, S., Jia, J.: Unnatural l0 sparse representation for natural image deblurring. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1107–1114 (2013)
Zhang, J., Pan, J., Lai, W.S., Lau, R., Yang, M.H.: Learning fully convolutional networks for iterative non-blind deconvolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6969–6977 (2017)
Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 2808–2817 (2017)
Zhang, Y., et al.: A poisson-gaussian denoising dataset with real fluorescence microscopy images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11710–11718 (2019)
Zukerman, J., Tirer, T., Giryes, R.: BP-DIP: a backprojection based deep image prior. In: Proceedings of the European Signal Processing Conference (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Quan, Y., Chen, Z., Zheng, H., Ji, H. (2022). Learning Deep Non-blind Image Deconvolution Without Ground Truths. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13666. Springer, Cham. https://doi.org/10.1007/978-3-031-20068-7_37
Download citation
DOI: https://doi.org/10.1007/978-3-031-20068-7_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20067-0
Online ISBN: 978-3-031-20068-7
eBook Packages: Computer ScienceComputer Science (R0)