Skip to main content
SpringerLink
Log in

Pixel-Level and Perceptual-Level Regularized Adversarial Learning for Joint Motion Deblurring and Super-Resolution

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

This paper aims to restore a clear image at high resolution from a low-resolution and motion-blurred image. To this end, we propose an end-to-end neural network named P2GAN containing deblurring and super-resolution modules with pixel-level and perceptual-level regularization terms. In the deblurring module, we propose an Asymmetric Residual Encoder-Decoder architecture, which enlarges the receptive field by stacking the Residual In Residual (RIR) structure with different numbers of convolutional filters to address different sizes of motion blur kernels. In particular, we extend RIR with a Channel Attention Block (CAB) to reweight the deblurring channel features. Similarly, CAB is integrated into the Residual In Residual Dense (RIRD) structure, denoted as RIRD-CA, to reweight channelwise features from input images and the deblurring module in the super-resolution module. RIRD-CA establishes a continuous-memory mechanism that preserves the features extracted from earlier layers along with the network. In particular, pixelwise loss, adversarial loss, and contextual loss have been incorporated into P2GAN from pixel-level and perceptual-level perspectives. Adversarial loss is used to generate rich textures and high-frequency details, while contextual loss eliminates unrealistic textures. Extensive experiments conducted on a publicly available dataset show the effectiveness of the proposed model which outperforms the state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Zhang H, Yang J, Zhang Y, Nasrabadi NM, Huang TS (2011) Close the loop: Joint blind image restoration and recognition with sparse representation prior. In: Computer Vision (ICCV), IEEE international conference, pp. 770–777

  2. Li H, Li Y, Porikli F (2016) Convolutional neural net bagging for online visual tracking. Comput Vis Image Underst 153:120–129

    Article  Google Scholar 

  3. Zou WW, Yuen PC (2012) Very low resolution face recognition problem. IEEE Trans Image Process 21(1):327–340

    Article  MathSciNet  MATH  Google Scholar 

  4. Kupyn O, Budzan V, Mykhailych M, Mishkin D, Matas J (2018) Deblurgan, Blind motion deblurring using conditional adversarial networks. In: IEEE Conference on computer vision and pattern recognition, pp. 8183–8192

  5. Cho SJ, Ji SW, Hong JP, Jung SW, Ko SJ (2021) Rethinking coarse-to-fine approach in single image deblurring. In Proceedings of the IEEE/CVF international conference on computer vision. pp. 4641–4650

  6. Wen Y, Chen J, Sheng B, Chen Z, Li P, Tan P, Lee TY (2021) Structure-aware motion deblurring using multi-adversarial optimized cyclegan. IEEE Trans Image Process 30:6142–6155

    Article  Google Scholar 

  7. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Shi W (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE conference on computer vision and pattern recognition, pp. 4681–4690

  8. Al Ismaeil K, Aouada D, Mirbach B, Ottersten B (2016) Enhancement of dynamic depth scenes by upsampling for precise super-resolution (UP-SR). Comput Vis Image Underst 147:38–49

    Article  Google Scholar 

  9. Yin M, Gao J, Cai S (2015) Image super-resolution via 2D tensor regression learning. Comput Vis Image Underst 132:12–23

    Article  Google Scholar 

  10. Xu X, Sun D, Pan J, Zhang Y, Pfister H, Yang MH (2017) Learning to super-resolve blurry face and text images. In: IEEE Conference on computer vision, pp. 251–260

  11. Zhang X, Wang F, Dong H, Guo Y (2018) A deep encoder-decoder networks for joint deblurring and super-resolution. In: 2018 IEEE International conference on acoustics, speech and signal processing (ICASSP), pp. 1448–1452

  12. Zhang X, Dong H, Hu Z, Lai WS, Wang F, Yang, MH (2018) Gated fusion network for joint image deblurring and super-resolution. arXiv preprint arXiv:1807.10806

  13. Li Y, Yang Z, Mao X, Wang Y, Li Q, Liu W, Wang Y (2019) GAN with pixel and perceptual regularizations for photo-realistic joint deblurring and super-resolution. In: Computer graphics international conference. pp. 395–401. Springer, Cham

  14. Zhang Y, Li K, Li K, Wang L Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: European conference on computer vision, pp. 286–301

  15. Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Loy CC (2018) Esrgan: Enhanced super-resolution generative adversarial networks. In: European conference on computer vision, pp. 63–79. Springer, Cham

  16. Blau Y, Michaeli T (2017) The perception-distortion tradeoff. In: IEEE Conference on computer vision and pattern recognition. pp. 6228–6237

  17. https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio

  18. Mechrez R, Talmi I, Zelnik-Manor L (2018) The contextual loss for image transformation with non-aligned data. In: European conference on computer vision, pp. 768–783

  19. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  20. Sim H, Kim M (2019) A deep motion deblurring network based on per-pixel adaptive kernels with residual down-up and up-down modules. In: IEEE Conference on computer vision and pattern recognition workshops. pp. 0–0

  21. Richardson WH (1972) Bayesian-based iterative method of image restoration. Josa 62(1):55–59

    Article  Google Scholar 

  22. Lucy LB (1974) An iterative technique for the rectification of observed distributions. Astron J 79:745

    Article  Google Scholar 

  23. Helstrom CW (1967) Image restoration by the method of least squares. Josa 57(3):297–303

    Article  Google Scholar 

  24. Dines K, Kak A (1977) Constrained least squares filtering. IEEE Trans Acoust Speech Signal Process 25(4):346–350

    Article  MATH  Google Scholar 

  25. Li L, Wang J, Lu W, Tan S (2017) Simultaneous tumor segmentation, image restoration, and blur kernel estimation in PET using multiple regularizations. Comput Vis Image Underst 155:173–194

    Article  Google Scholar 

  26. Seibold C, Hilsmann A, Eisert P (2017) Model-based motion blur estimation for the improvement of motion tracking. Comput Vis Image Underst 160:45–56

    Article  Google Scholar 

  27. Tran P, Tran A, Phung Q, Hoai M (2021) Explore image deblurring via blur kernel space. arXiv preprint arXiv:2104.00317

  28. Li Y, Tofighi M, Geng J, Monga V, Eldar YC (2019) Deep algorithm unrolling for blind image deblurring. arXiv preprint arXiv:1902.03493

  29. Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873

    Article  MathSciNet  MATH  Google Scholar 

  30. Arun PV, Porwal A (2019) Spatial-spectral feature based approach towards convolutional sparse coding of hyperspectral images. Comput Vis Image Underst 188:102797

    Article  Google Scholar 

  31. Timofte R, De Smet V, Van Gool L (2014) A+, Adjusted anchored neighborhood regression for fast super-resolution. In: Asian conference on computer vision, pp. 111–126, Springer, Cham

  32. Chang H, Yeung DY, Xiong Y (2004) Super-resolution through neighbor embedding. In: 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR, Vol. 1, pp. I–I

  33. Guo Y, Chen J, Wang J, Chen Q, Cao J, Deng Z, Tan M (2020) Closed-loop matters: dual regression networks for single image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5407–5416

  34. Zuo Y, Wu Q, Fang Y, An P, Huang L, Chen Z (2019) Multi-scale frequency reconstruction for guided depth map super-resolution via deep residual network. IEEE Trans Circuits Syst Video Technol. https://doi.org/10.1109/TCSVT.2018.2890271

    Article  Google Scholar 

  35. Liang J, Zhang K, Gu S, Van Gool L, Timofte R (2021). Flow-based kernel prior with application to blind super-resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10601–10610

  36. Sajjadi MS, Schölkopf B, Hirsch M (2017) Enhancenet: Single image super-resolution through automated texture synthesis. In: Computer Vision (ICCV), IEEE international conference, pp. 4501–4510

  37. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680

  38. Michaeli T, Irani M (2013) Nonparametric blind super-resolution. In: IEEE International conference on computer vision, pp. 945–952

  39. Ma C, Yang CY, Yang X, Yang MH (2017) Learning a no-reference quality metric for single-image super-resolution. Comput Vis Image Underst 158:1–16

    Article  Google Scholar 

  40. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  41. Cheon M, Kim JH, Choi JH, Lee JS (2018) Generative adversarial network-based image super-resolution using perceptual content losses. In: European conference on computer vision (ECCV), pp. 0–0

  42. Hui Z, Wang X, Deng L, Gao X (2018) Perception-preserving convolutional networks for image enhancement on smartphones. In: European conference on computer vision (ECCV), pp. 0–0

  43. Choi J-H et al (2019) Deep learning-based image super-resolution considering quantitative and perceptual quality. Neurocomputing. https://doi.org/10.1016/j.neucom.2019.06.103

    Article  Google Scholar 

  44. Zhang R, Isola P, Efros AA, Shechtman E, Wang O (2018) The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE Conference on computer vision and pattern recognition, pp. 586–595

  45. Mechrez R, Talmi I, Shama F, Zelnik-Manor L (2018) Maintaining natural image statistics with the contextual loss. In: Asian Conference on computer vision (pp. 427–443). Springer, Cham

  46. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE conference on computer vision and pattern recognition. pp. 770–778

  47. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: IEEE conference on computer vision and pattern recognition, pp. 4700–4708

  48. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: IEEE conference on computer vision and pattern recognition, pp. 2472–2481

  49. Nah S, Hyun Kim T, Mu Lee K (2017) Deep multi-scale convolutional neural network for dynamic scene deblurring. In: IEEE conference on computer vision and pattern recognition. pp. 3883–3891

  50. Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC (2017) Improved training of wasserstein gans. In: Advances in neural information processing systems, pp. 5767–5777

  51. Liu Z, Luo P, Wang X, Tang X, (2015) Deep learning face attributes in the wild. In: IEEE International conference on computer vision, pp. 3730–3738

  52. Zhang H, Dai Y, Li H, Koniusz P (2019) Deep stacked hierarchical multi-patch network for image deblurring. In: IEEE Conference on computer vision and pattern recognition, pp. 5978–5986

  53. Li J, Fang F, Mei K, Zhang G (2018) Multi-scale residual network for image super-resolution. In: European conference on computer vision (ECCV), pp.517–532

  54. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in neural information processing systems, pp. 6626–6637

  55. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. In: IEEE conference on computer vision and pattern recognition, pp. 815–823

Download references

Author information

Authors and Affiliations

  1. South China Normal University, Guangzhou, China

    Yong Li & Tianyong Hao

  2. Guangdong University of Technology, Guangzhou, China

    Zhenguo Yang & Wenyin Liu

  3. The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China

    Qing Li

Authors
  1. Yong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenguo Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tianyong Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wenyin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Zhenguo Yang or Wenyin Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Yang, Z., Hao, T. et al. Pixel-Level and Perceptual-Level Regularized Adversarial Learning for Joint Motion Deblurring and Super-Resolution. Neural Process Lett 55, 905–926 (2023). https://doi.org/10.1007/s11063-022-10913-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-022-10913-7

Keywords

Search

Navigation