Skip to main content
SpringerLink
Log in

High-resolution image de-raining using conditional GAN with sub-pixel upscaling

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

High-quality image de-raining is a challenging task that has been given considerable importance in recent times. To begin with, this problem is modeled as an image decomposition task where a rainy image is decomposed into the rain-free background and the associated rain streak map. Most of the existing methods have been successful in removing the rain-streaks but fails to restore the image quality, which is degraded due to noise removal. This paper proposes a novel architecture called High-Resolution Image De-Raining using Conditional Generative Adversarial Networks (HRID-GAN) to generate a de-rained image with minimal artifacts and better visual quality. Extensive experiments on publicly available synthetic as well as real-world datasets show a substantial improvement over the state-of-the-art methods SPANet (Wang et al. 2019) by ∼ 2.43% in PSNR and, DID-MDN (Zhang and Patel 2018) by ∼ 2.43%, ∼ 10.12% and ID-CGAN (Zhang et al. 2017) by ∼ 11.80%, ∼ 34.70% in SSIM and PSNR respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. Transpose convolution is also referred as Deconvolution

  2. https://github.com/hezhangsprinter/DIDMDN

References

  1. Abadi M, Agarwal A (2015) TensorFlow: Large-scale machine learning on heterogeneous systems. http://tensorflow.org/. Software available from tensorflow.org

  2. Chang Y, Yan L, Zhong S (2017) Transformed low-rank model for line pattern noise removal. In: 2017 IEEE International conference on computer vision (ICCV), pp 1735–1743. https://doi.org/10.1109/ICCV.2017.191

  3. Chen DY, Chen CC, Kang LW (2014) Visual depth guided color image rain streaks removal using sparse coding. IEEE Trans Circ Syst Vid Technol 24(8):1430–1455. https://doi.org/10.1109/TCSVT.2014.2308627

    Article  Google Scholar 

  4. Chen Q, Yi X, Ni B, Shen Z, Yang X (2017) Rain removal via residual generation cascading. In: 2017 IEEE Visual communications and image processing (VCIP), pp 1–4. https://doi.org/10.1109/VCIP.2017.8305092

  5. Fu X, Huang J, Ding X, Liao Y, Paisley J (2017) Clearing the skies: a deep network architecture for single-image rain removal. IEEE Trans Image Process 26(6):2944–2956. https://doi.org/10.1109/TIP.2017.2691802

    Article  MathSciNet  MATH  Google Scholar 

  6. Fu X, Huang J, Zeng D, Huang Y, Ding X, Paisley J (2017) Removing rain from single images via a deep detail network. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR), pp 1715–1723. https://doi.org/10.1109/CVPR.2017.186

  7. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Advances in Neural Information Processing Systems 27, Curran Associates, Inc., pp 2672–2680. http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf

  8. Greff K, Srivastava RK, Koutník J, Steunebrink BR, Schmidhuber J (2017) Lstm: a search space odyssey. IEEE Trans Neural Netw Learn Syst 28(10):2222–2232. https://doi.org/10.1109/TNNLS.2016.2582924

    Article  MathSciNet  Google Scholar 

  9. Gu S, Meng D, Zuo W, Zhang L (2017) Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: 2017 IEEE International conference on computer vision (ICCV), pp 1717–1725. https://doi.org/10.1109/ICCV.2017.189

  10. Gu S, Meng D, Zuo W, Zhang L (2017) Joint convolutional analysis and synthesis sparse representation for single image layer separation. In: 2017 IEEE International conference on computer vision (ICCV), pp 1717–1725. https://doi.org/10.1109/ICCV.2017.189

  11. Haar A (1910) Zur theorie der orthogonalen funktionensysteme. Math Ann 69(3):331–371. https://doi.org/10.1007/BF01456326

    Article  MathSciNet  MATH  Google Scholar 

  12. He K, Sun J, Tang X (2011) Single image haze removal using dark channel prior. IEEE Trans Patt Anal Mach Intel 33(12):2341–2353. https://doi.org/10.1109/TPAMI.2010.168

    Article  Google Scholar 

  13. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  14. Huang DA, Kang LW, Wang YCF, Lin CW (2014) Self-learning based image decomposition with applications to single image denoising. IEEE Trans Multimedia 16(1):83–93. https://doi.org/10.1109/TMM.2013.2284759

    Article  Google Scholar 

  15. Huang G, Liu Z, Weinberger KQ (2016) Densely connected convolutional networks. arXiv:abs/1608.06993

  16. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Bach F, Blei D (eds) Proceedings of the 32nd International Conference on Machine Learning, Proceedings of Machine Learning Research. http://proceedings.mlr.press/v37/ioffe15.html, vol 37. PMLR, Lille, pp 448–456

  17. Johnson J, Alahi A, Li F (2016) Perceptual losses for real-time style transfer and super-resolution. arXiv:abs/1603.08155

  18. Kang LW, Lin CW, Fu YH (2012) Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans Image Process 21 (4):1742–1755. https://doi.org/10.1109/TIP.2011.2179057

    Article  MathSciNet  MATH  Google Scholar 

  19. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: International conference on learning representations (ICLR)

  20. Kopf J, Cohen MF, Lischinski D, Uyttendaele M (2007) Joint bilateral upsampling. ACM Trans Graph 26(3):96–es. https://doi.org/10.1145/1276377.1276497

    Article  Google Scholar 

  21. Kumar Sharma P, Jain P, Sur A (2020) Scale-aware conditional generative adversarial network for image dehazing. In: 2020 IEEE Winter conference on applications of computer vision (WACV), pp 2344–2354

  22. Kupyn O, Budzan V, Mykhailych M, Mishkin D, Matas J (2018) Deblurgan: Blind motion deblurring using conditional adversarial networks. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  23. Li R, Pan J, Li Z, Tang J (2018) Single image dehazing via conditional generative adversarial network. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  24. Li Y, Tan RT, Guo X, Lu J, Brown MS (2016) Rain streak removal using layer priors. In: 2016 IEEE Conference on computer vision and pattern recognition (CVPR), pp 2736–2744. https://doi.org/10.1109/CVPR.2016.299

  25. Li Y, Tan RT, Guo X, Lu J, Brown MS (2017) Single image rain streak decomposition using layer priors. IEEE Trans Image Process 26 (8):3874–3885. https://doi.org/10.1109/TIP.2017.2708841

    Article  MathSciNet  MATH  Google Scholar 

  26. Lin T, Goyal P, Girshick R, He K, Dollaŕ P (2017) Focal loss for dense object detection. In: 2017 IEEE International conference on computer vision (ICCV), pp 2999–3007

  27. Lu X, Guo Y, Liu N, Wan L, Fang T (2018) Non-convex joint bilateral guided depth upsampling. Multimed Tools Appl 77(12):15521–15544. https://doi.org/10.1007/s11042-017-5131-x

    Article  Google Scholar 

  28. Lu X, Ma C, Ni B, Yang X, Reid I, Yang MH (2018) Deep regression tracking with shrinkage loss. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y (eds) Computer vision – ECCV 2018. Springer International Publishing, Cham, pp 369–386

  29. Lu X, Wang W, Ma C, Shen J, Shao L, Porikli F (2019) See more, know more: Unsupervised video object segmentation with co-attention siamese networks. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  30. Mao X, Shen C, Yang YB (2016) Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Lee DD, Sugiyama M, Luxburg UV, Guyon I, Garnett R (eds) Advances in Neural Information Processing Systems 29, pp 2802–2810. Curran Associates, Inc. http://papers.nips.cc/paper/6172-image-restoration-using-very-deep-convolutional-encoder-decoder-networks-with-symmetric-skip-connections.pdf

  31. Nash JF (1950) Equilibrium points in n-person games. Proceedings of the National Academy of Sciences 36(1):48–49. https://doi.org/10.1073/pnas.36.1.48

    Article  MathSciNet  MATH  Google Scholar 

  32. Odena A, Dumoulin V, Olah C (2016) Deconvolution and checkerboard artifacts. Distill. http://distill.pub/2016/deconv-checkerboard/

  33. Park K, Yu S, Jeong J (2018) A contrast restoration method for effective single image rain removal algorithm. In: 2018 International workshop on advanced image technology (IWAIT), pp 1–4. https://doi.org/10.1109/IWAIT.2018.8369644

  34. Qian R, Tan RT, Yang W, Su J, Liu J (2018) Attentive generative adversarial network for raindrop removal from a single image. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  35. Ren W, Tian J, Han Z, Chan A, Tang Y (2017) Video desnowing and deraining based on matrix decomposition. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR), pp 2838–2847. https://doi.org/10.1109/CVPR.2017.303

  36. Sharma PK, Jain P, Sur A (2019) Dual-domain single image de-raining using conditional generative adversarial network. In: 2019 IEEE International conference on image processing (ICIP), pp 2796–2800. https://doi.org/10.1109/ICIP.2019.8803353

  37. Sheikh HR, Bovik AC (2006) Image information and visual quality. IEEE Trans Image Process 15(2):430–444. https://doi.org/10.1109/TIP.2005.859378

    Article  Google Scholar 

  38. Shelhamer E, Long J, Darrell T (2017) Fully convolutional networks for semantic segmentation. IEEE Trans Patt Anal Mach Intel 39(4):640–651. https://doi.org/10.1109/TPAMI.2016.2572683

    Article  Google Scholar 

  39. Shen L, Yue Z, Chen Q, Feng F, Ma J (2018) Deep joint rain and haze removal from single images. arXiv:abs/1801.06769

  40. Shi W, Caballero J, Huszar F, Totz J, Aitken AP, Bishop R, Rueckert D, Wang Z (2016) Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  41. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Advances in Neural Information Processing Systems 27, Curran Associates, Inc., pp 3104–3112. http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf

  42. Wang J, Li X, Yang J (2018) Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  43. Wang T, Yang X, Xu K, Chen S, Zhang Q, Lau RW (2019) Spatial attentive single-image deraining with a high quality real rain dataset. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  44. Wang Y, Chen C, Zhu S, Zeng B (2016) A framework of single-image deraining method based on analysis of rain characteristics. In: 2016 IEEE International conference on image processing (ICIP), pp 4087–4091

  45. Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9(3):81–84. https://doi.org/10.1109/97.995823

    Article  Google Scholar 

  46. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  47. Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: The thrity-seventh asilomar conference on signals, systems computers, 2003, vol 2, pp 1398–1402. https://doi.org/10.1109/ACSSC.2003.1292216

  48. Yang W, Tan RT, Feng J, Liu J, Guo Z, Yan S (2017) Deep joint rain detection and removal from a single image. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR), pp 1685–1694. https://doi.org/10.1109/CVPR.2017.183

  49. Yeh CH, Liu PH, Yu CE, Lin CY (2015) Single image rain removal based on part-based model. In: 2015 IEEE International conference on consumer electronics - taiwan, pp 462–463. https://doi.org/10.1109/ICCE-TW.2015.7216999

  50. Yu S, Ou W, You X, Mou Y, Jiang X, Tang Y (2015) Single image rain streaks removal based on self-learning and structured sparse representation. In: 2015 IEEE China Summit and international conference on signal and information processing (chinaSIP), pp 215–219. https://doi.org/10.1109/ChinaSIP.2015.7230394

  51. Zhang H, Patel VM (2017) Convolutional sparse and low-rank coding-based rain streak removal. In: 2017 IEEE Winter conference on applications of computer vision (WACV), pp 1259–1267. https://doi.org/10.1109/WACV.2017.145

  52. Zhang H, Patel VM (2018) Density-aware single image de-raining using a multi-stream dense network. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  53. Zhang H, Sindagi V, Patel VM (2017) Image de-raining using a conditional generative adversarial network. arXiv:abs/1701.05957

  54. Zhang H, Sindagi V, Patel VM (2019) Image de-raining using a conditional generative adversarial network. IEEE Transactions on Circuits and Systems for Video Technology, pp 1–1

  55. Zhu L, Fu CW, Lischinski D, Heng PA (2017) Joint bi-layer optimization for single-image rain streak removal. In: 2017 IEEE International conference on computer vision (ICCV), pp 2545–2553. https://doi.org/10.1109/ICCV.2017.276

Download references

Acknowledgments

Authors would like to thank the anonymous reviewers for their insightful comments and suggestions. Authors would also like to acknowledge the funding agency, Ministry of Human Resource Development, Government of India.

Author information

Authors and Affiliations

  1. Indian Institute of Technology Guwahati, Guwahati, India

    Prasen Kumar Sharma, Sathisha Basavaraju & Arijit Sur

Authors
  1. Prasen Kumar Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sathisha Basavaraju
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arijit Sur
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Prasen Kumar Sharma.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, P.K., Basavaraju, S. & Sur, A. High-resolution image de-raining using conditional GAN with sub-pixel upscaling. Multimed Tools Appl 80, 1075–1094 (2021). https://doi.org/10.1007/s11042-020-09642-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09642-7

Keywords

Search

Navigation