Skip to main content
Springer Nature Link
Log in

Learning Discriminative Shrinkage Deep Networks for Image Deconvolution

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13679))

Included in the following conference series:

  • 3432 Accesses

Abstract

Most existing methods usually formulate the non-blind deconvolution problem into a maximum-a-posteriori framework and address it by manually designing a variety of regularization terms and data terms of the latent clear images. However, explicitly designing these two terms is quite challenging and usually leads to complex optimization problems which are difficult to solve. This paper proposes an effective non-blind deconvolution approach by learning discriminative shrinkage functions to model these terms implicitly. Most existing methods use deep convolutional neural networks (CNNs) or radial basis functions to learn the regularization term simply. In contrast, we formulate both the data term and regularization term and split the deconvolution model into data-related and regularization-related sub-problems according to the alternating direction method of multipliers. We explore the properties of the Maxout function and develop a deep CNN model with Maxout layers to learn discriminative shrinkage functions, which directly approximates the solutions of these two sub-problems. Moreover, the fast-Fourier-transform-based image restoration usually leads to ringing artifacts. At the same time, the conjugate-gradient-based approach is time-consuming; we develop the Conjugate Gradient Network to restore the latent clear images effectively and efficiently. Experimental results show that the proposed method performs favorably against the state-of-the-art methods in terms of efficiency and accuracy. Source codes, models, and more results are available at https://github.com/setsunil/DSDNet.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aljadaany, R., Pal, D.K., Savvides, M.: Douglas-rachford networks: learning both the image prior and data fidelity terms for blind image deconvolution. In: CVPR, pp. 10235–10244 (2019)

    Google Scholar 

  2. Barrett, R., et al.: Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods. SIAM (1994)

    Google Scholar 

  3. Bevilacqua, M., Roumy, A., Guillemot, C., Alberi-Morel, M.L.: Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: BMVC (2012)

    Google Scholar 

  4. Bigdeli, S.A., Zwicker, M., Favaro, P., Jin, M.: Deep mean-shift priors for image restoration. In: NeurIPS (2017)

    Google Scholar 

  5. Chakrabarti, A.: A neural approach to blind motion deblurring. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 221–235. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_14

    Chapter  Google Scholar 

  6. Chan, T.F., Wong, C.K.: Total variation blind deconvolution. IEEE TIP 7(3), 370–375 (1998)

    Google Scholar 

  7. Chen, L., Zhang, J., Pan, J., Lin, S., Fang, F., Ren, J.S.: Learning a non-blind deblurring network for night blurry images. In: CVPR, pp. 10542–10550 (2021)

    Google Scholar 

  8. Cho, S., Wang, J., Lee, S.: Handling outliers in non-blind image deconvolution. In: ICCV, pp. 495–502 (2011)

    Google Scholar 

  9. Dong, J., Pan, J., Sun, D., Su, Z., Yang, M.-H.: Learning data terms for non-blind deblurring. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 777–792. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_46

    Chapter  Google Scholar 

  10. Dong, J., Roth, S., Schiele, B.: Deep wiener deconvolution: Wiener meets deep learning for image deblurring. In: NeurIPS (2020)

    Google Scholar 

  11. Dong, J., Roth, S., Schiele, B.: Learning spatially-variant map models for non-blind image deblurring. In: CVPR, pp. 4886–4895 (2021)

    Google Scholar 

  12. Eboli, T., Sun, J., Ponce, J.: End-to-end interpretable learning of non-blind image deblurring. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12362, pp. 314–331. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58520-4_19

    Chapter  Google Scholar 

  13. Gao, H., Tao, X., Shen, X., Jia, J.: Dynamic scene deblurring with parameter selective sharing and nested skip connections. In: CVPR, pp. 3848–3856 (2019)

    Google Scholar 

  14. Geman, D., Reynolds, G.: Constrained restoration and the recovery of discontinuities. IEEE TPAMI 14(3), 367–383 (1992)

    Article  Google Scholar 

  15. Gong, D., Zhang, Z., Shi, Q., van den Hengel, A., Shen, C., Zhang, Y.: Learning deep gradient descent optimization for image deconvolution. IEEE Trans. Neural Netw. Learn. Syst. 31(12), 5468–5482 (2020)

    Article  MathSciNet  Google Scholar 

  16. Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. In: ICML, pp. 1319–1327 (2013)

    Google Scholar 

  17. Gu, S., Zhang, L., Zuo, W., Feng, X.: Weighted nuclear norm minimization with application to image denoising. In: CVPR, pp. 2862–2869 (2014)

    Google Scholar 

  18. Hadamard, J.: Lectures on Cauchy’s Problem in Linear Partial Differential Equations. Courier Corporation (2003)

    Google Scholar 

  19. Jancsary, J., Nowozin, S., Rother, C.: Loss-specific training of non-parametric image restoration models: a new state of the art. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7578, pp. 112–125. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33786-4_9

    Chapter  Google Scholar 

  20. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  21. Ko, H.C., Chang, J.Y., Ding, J.J.: Deep priors inside an unrolled and adaptive deconvolution model. In: ACCV (2020)

    Google Scholar 

  22. Kong, S., Wang, W., Feng, X., Jia, X.: Deep red unfolding network for image restoration. IEEE TIP 31, 852–867 (2021)

    Google Scholar 

  23. Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-Laplacian priors. In: NeurIPS, pp. 1033–1041 (2009)

    Google Scholar 

  24. Krishnan, D., Tay, T., Fergus, R.: Blind deconvolution using a normalized sparsity measure. In: CVPR, pp. 233–240 (2011)

    Google Scholar 

  25. Kruse, J., Rother, C., Schmidt, U.: Learning to push the limits of efficient FFT-based image deconvolution. In: ICCV, pp. 4586–4594 (2017)

    Google Scholar 

  26. Kupyn, O., Budzan, V., Mykhailych, M., Mishkin, D., Matas, J.: Deblurgan: Blind motion deblurring using conditional adversarial networks. In: CVPR, pp. 8183–8192 (2018)

    Google Scholar 

  27. Lai, W.S., Huang, J.B., Hu, Z., Ahuja, N., Yang, M.H.: A comparative study for single image blind deblurring. In: CVPR, pp. 1701–1709 (2016)

    Google Scholar 

  28. Levin, A., Fergus, R., Durand, F., Freeman, W.T.: Image and depth from a conventional camera with a coded aperture. ACM TOG. 26(3), 70-es (2007)

    Google Scholar 

  29. Levin, A., Weiss, Y.: User assisted separation of reflections from a single image using a sparsity prior. IEEE TPAMI 29(9), 1647–1654 (2007)

    Article  Google Scholar 

  30. Levin, A., Weiss, Y., Durand, F., Freeman, W.T.: Understanding and evaluating blind deconvolution algorithms. In: CVPR, pp. 1964–1971 (2009)

    Google Scholar 

  31. Levin, A., Weiss, Y., Durand, F., Freeman, W.T.: Understanding blind deconvolution algorithms. IEEE TPAMI 33(12), 2354–2367 (2011)

    Article  Google Scholar 

  32. Li, L., Pan, J., Lai, W.S., Gao, C., Sang, N., Yang, M.H.: Blind image deblurring via deep discriminative priors. IJCV 127(8), 1025–1043 (2019)

    Article  Google Scholar 

  33. Li, Y., Tofighi, M., Geng, J., Monga, V., Eldar, Y.C.: Deep algorithm unrolling for blind image deblurring. arXiv preprint arXiv:1902.03493 (2019)

  34. Liu, C.S.: Modifications of steepest descent method and conjugate gradient method against noise for ill-posed linear systems. Commun. Numer. Anal. 2012, 5 (2012)

    MathSciNet  Google Scholar 

  35. Liu, R., Jia, J.: Reducing boundary artifacts in image deconvolution. In: ICIP, pp. 505–508 (2008)

    Google Scholar 

  36. Ma, K., et al.: Waterloo exploration database: new challenges for image quality assessment models. IEEE TIP 26(2), 1004–1016 (2016)

    MathSciNet  MATH  Google Scholar 

  37. Marin, L., Háo, D.N., Lesnic, D.: Conjugate gradient-boundary element method for a Cauchy problem in the lamé system. WIT Trans. Modell. Simul. 27, 10 (2001)

    Google Scholar 

  38. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV, pp. 416–423 (2001)

    Google Scholar 

  39. Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE TIP 21(12), 4695–4708 (2012). https://doi.org/10.1109/TIP.2012.2214050

    Article  MathSciNet  MATH  Google Scholar 

  40. Nah, S., Hyun Kim, T., Mu Lee, K.: Deep multi-scale convolutional neural network for dynamic scene deblurring. In: CVPR, pp. 3883–3891 (2017)

    Google Scholar 

  41. Nan, Y., Ji, H.: Deep learning for handling kernel/model uncertainty in image deconvolution. In: CVPR, pp. 2388–2397 (2020)

    Google Scholar 

  42. Nan, Y., Quan, Y., Ji, H.: Variational-em-based deep learning for noise-blind image deblurring. In: CVPR, pp. 3626–3635 (2020)

    Google Scholar 

  43. Pan, J., Sun, D., Pfister, H., Yang, M.H.: Blind image deblurring using dark channel prior. In: CVPR, pp. 1628–1636 (2016)

    Google Scholar 

  44. Parikh, N., Boyd, S.: Proximal algorithms. Found. Trends Optim. 1(3), 127–239 (2014)

    Article  Google Scholar 

  45. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’ Alché-Buc, F., Fox, E., Garnett, R. (eds.) NeurIPS, pp. 8024–8035. Curran Associates, Inc. (2019). https://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

  46. Perrone, D., Favaro, P.: Total variation blind deconvolution: the devil is in the details. In: CVPR, pp. 2909–2916 (2014)

    Google Scholar 

  47. Qiu, H., Hammernik, K., Qin, C., Rueckert, D.: GraDIRN: learning iterative gradient descent-based energy minimization for deformable image registration. arXiv preprint arXiv:2112.03915 (2021)

  48. Ren, D., Zuo, W., Zhang, D., Zhang, L., Yang, M.H.: Simultaneous fidelity and regularization learning for image restoration. IEEE TPAMI 43, 284–299 (2019)

    Article  Google Scholar 

  49. Ren, W., Cao, X., Pan, J., Guo, X., Zuo, W., Yang, M.H.: Image deblurring via enhanced low-rank prior. IEEE TIP 25(7), 3426–3437 (2016)

    MathSciNet  MATH  Google Scholar 

  50. Richardson, W.H.: Bayesian-based iterative method of image restoration. JoSA 62(1), 55–59 (1972)

    Article  Google Scholar 

  51. Roth, S., Black, M.J.: Fields of experts: a framework for learning image priors. In: CVPR, pp. 860–867 (2005)

    Google Scholar 

  52. Rudin, L.I., Osher, S.: Total variation based image restoration with free local constraints. In: ICIP, vol. 1, pp. 31–35 (1994)

    Google Scholar 

  53. Ryabtsev, A.: The error accumulation in the conjugate gradient method for degenerate problem. arXiv preprint arXiv:2004.10242 (2020)

  54. Samuel, K.G., Tappen, M.F.: Learning optimized map estimates in continuously-valued MRF models. In: CVPR, pp. 477–484 (2009)

    Google Scholar 

  55. Schmidt, U., Jancsary, J., Nowozin, S., Roth, S., Rother, C.: Cascades of regression tree fields for image restoration. IEEE TPAMI 38(4), 677–689 (2015)

    Article  Google Scholar 

  56. Schmidt, U., Roth, S.: Shrinkage fields for effective image restoration. In: CVPR, pp. 2774–2781 (2014)

    Google Scholar 

  57. Schmidt, U., Rother, C., Nowozin, S., Jancsary, J., Roth, S.: Discriminative non-blind deblurring. In: CVPR, pp. 604–611 (2013)

    Google Scholar 

  58. Schuler, C.J., Christopher Burger, H., Harmeling, S., Scholkopf, B.: A machine learning approach for non-blind image deconvolution. In: CVPR, pp. 1067–1074 (2013)

    Google Scholar 

  59. Suin, M., Purohit, K., Rajagopalan, A.: Spatially-attentive patch-hierarchical network for adaptive motion deblurring. In: CVPR, pp. 3606–3615 (2020)

    Google Scholar 

  60. Sun, J., Cao, W., Xu, Z., Ponce, J.: Learning a convolutional neural network for non-uniform motion blur removal. In: CVPR, pp. 769–777 (2015)

    Google Scholar 

  61. Sun, L., Cho, S., Wang, J., Hays, J.: Edge-based blur kernel estimation using patch priors. In: ICCP, pp. 1–8 (2013)

    Google Scholar 

  62. Tao, X., Gao, H., Shen, X., Wang, J., Jia, J.: Scale-recurrent network for deep image deblurring. In: CVPR, pp. 8174–8182 (2018)

    Google Scholar 

  63. Tappen, M.F., Liu, C., Adelson, E.H., Freeman, W.T.: Learning gaussian conditional random fields for low-level vision. In: CVPR, pp. 1–8 (2007)

    Google Scholar 

  64. Venkatanath, N., Praneeth, D., Bh, M.C., Channappayya, S.S., Medasani, S.S.: Blind image quality evaluation using perception based features. In: National Conference on Communications (NCC), pp. 1–6 (2015)

    Google Scholar 

  65. Wang, Y., Yang, J., Yin, W., Zhang, Y.: A new alternating minimization algorithm for total variation image reconstruction. SIAM J. Imag. Sci. 1(3), 248–272 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  66. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE TIP 13(4), 600–612 (2004)

    Google Scholar 

  67. Xiang, J., Dong, Y., Yang, Y.: FISTA-net: Learning a fast iterative shrinkage thresholding network for inverse problems in imaging. IEEE TMI 40, 1329–1339 (2021)

    Google Scholar 

  68. Yang, Y., Sun, J., Li, H., Xu, Z.: Deep ADMM-net for compressive sensing MRI. In: NeurIPS, pp. 10–18 (2016)

    Google Scholar 

  69. Zhang, J., Ghanem, B.: ISTA-net: interpretable optimization-inspired deep network for image compressive sensing. In: CVPR, pp. 1828–1837 (2018)

    Google Scholar 

  70. Zhang, J., shan Pan, J., Lai, W.S., Lau, R.W.H., Yang, M.H.: Learning fully convolutional networks for iterative non-blind deconvolution. In: CVPR, pp. 6969–6977 (2017)

    Google Scholar 

  71. Zhang, K., Gool, L.V., Timofte, R.: Deep unfolding network for image super-resolution. In: CVPR, pp. 3217–3226 (2020)

    Google Scholar 

  72. Zhang, K., Li, Y., Zuo, W., Zhang, L., Van Gool, L., Timofte, R.: Plug-and-play image restoration with deep denoiser prior. IEEE TPAMI. 44, 6360–6376 (2021)

    Google Scholar 

  73. Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: CVPR, pp. 3929–3938 (2017)

    Google Scholar 

  74. Zhang, K., Zuo, W., Zhang, L.: Deep plug-and-play super-resolution for arbitrary blur kernels. In: CVPR, pp. 1671–1681 (2019)

    Google Scholar 

  75. Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: ICCV, pp. 479–486 (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graduate Institute of Electronics Engineering, National Taiwan University, Taipei, Taiwan

    Pin-Hung Kuo & Shao-Yi Chien

  2. Nanjing University of Science and Technology, Nanjing, China

    Jinshan Pan

  3. Google Research, Nanjing, China

    Ming-Hsuan Yang

  4. University of California, Merced, USA

    Ming-Hsuan Yang

  5. Yonsei University, Seoul, South Korea

    Ming-Hsuan Yang

Authors
  1. Pin-Hung Kuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinshan Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shao-Yi Chien
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ming-Hsuan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pin-Hung Kuo .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 16339 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kuo, PH., Pan, J., Chien, SY., Yang, MH. (2022). Learning Discriminative Shrinkage Deep Networks for Image Deconvolution. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13679. Springer, Cham. https://doi.org/10.1007/978-3-031-19800-7_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19800-7_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19799-4

  • Online ISBN: 978-3-031-19800-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation