Skip to main content
Springer Nature Link
Log in

ART-SS: An Adaptive Rejection Technique for Semi-supervised Restoration for Adverse Weather-Affected Images

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13678))

Included in the following conference series:

Abstract

In recent years, convolutional neural network-based single image adverse weather removal methods have achieved significant performance improvements on many benchmark datasets. However, these methods require large amounts of clean-weather degraded image pairs for training, which is often difficult to obtain in practice. Although various weather degradation synthesis methods exist in the literature, the use of synthetically generated weather degraded images often results in sub-optimal performance on the real weatherdegraded images due to the domain gap between synthetic and real world images. To deal with this problem, various semi-supervised restoration (SSR) methods have been proposed for deraining or dehazing which learn to restore clean image using synthetically generated datasets while generalizing better using unlabeled real-world images. The performance of a semi-supervised method is essentially based on the quality of the unlabeled data. In particular, if the unlabeled data characteristics are very different from that of the labeled data, then the performance of a semi-supervised method degrades significantly. We theoretically study the effect of unlabeled data on the performance of an SSR method and develop a technique that rejects the unlabeled images that degrade the performance. Extensive experiments and ablation study show that the proposed sample rejection method increases the performance of existing SSR deraining and dehazing methods significantly. Code is available at: https://github.com/rajeevyasarla/ART-SS.

This work was supported by an ARO grant W911NF–21–1–0135.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Berman, D., et al.: Non-local image dehazing. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1674–1682 (2016)

    Google Scholar 

  2. Cai, B., Xu, X., Jia, K., Qing, C., Tao, D.: Dehazenet: an end-to-end system for single image haze removal. IEEE Trans. Image Process. 25(11), 5187–5198 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  3. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 833–851. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_49

    Chapter  Google Scholar 

  4. Deng, S., et al.: Detail-recovery image deraining via context aggregation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14560–14569 (2020)

    Google Scholar 

  5. Dong, H., et al.: Multi-scale boosted dehazing network with dense feature fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2157–2167 (2020)

    Google Scholar 

  6. Fattal, R.: Dehazing using color-lines. ACM Trans. Graph. (TOG) 34(1), 1–14 (2014)

    Article  Google Scholar 

  7. Fu, X., Huang, J., Zeng, D., Huang, Y., Ding, X., Paisley, J.: Removing rain from single images via a deep detail network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3855–3863 (2017)

    Google Scholar 

  8. Fu, X., Qi, Q., Zha, Z.J., Zhu, Y., Ding, X.: Rain streak removal via dual graph convolutional network. In: Proceedings AAAI Conference on Artificial Intelligence, pp. 1–9 (2021)

    Google Scholar 

  9. Gao, S., Cheng, M.M., Zhao, K., Zhang, X.Y., Yang, M.H., Torr, P.H.: Res2net: a new multi-scale backbone architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019)

    Google Scholar 

  10. He, K., Sun, J., Tang, X.: Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2010)

    Google Scholar 

  11. Hu, X., Fu, C.W., Zhu, L., Heng, P.A.: Depth-attentional features for single-image rain removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8022–8031 (2019)

    Google Scholar 

  12. Huang, D.A., Kang, L.W., Wang, Y.C.F., Lin, C.W.: Self-learning based image decomposition with applications to single image denoising. IEEE Trans. Multimedia 16(1), 83–93 (2013)

    Article  Google Scholar 

  13. Huang, H., Yu, A., He, R.: Memory oriented transfer learning for semi-supervised image deraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7732–7741 (2021)

    Google Scholar 

  14. Jiang, K., et al.: Multi-scale progressive fusion network for single image deraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8346–8355 (2020)

    Google Scholar 

  15. Kang, L.W., Lin, C.W., Fu, Y.H.: Automatic single-image-based rain streaks removal via image decomposition. IEEE Trans. Image Process. 21(4), 1742–1755 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  16. Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? arXiv preprint arXiv:1703.04977 (2017)

  17. Li, B., et al.: Benchmarking single-image dehazing and beyond. IEEE Trans. Image Process. 28(1), 492–505 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  18. Li, L., et al.: Semi-supervised image dehazing. IEEE Trans. Image Process. 29, 2766–2779 (2019)

    Article  MATH  Google Scholar 

  19. Li, P., Tian, J., Tang, Y., Wang, G., Wu, C.: Deep retinex network for single image dehazing. IEEE Trans. Image Process. 30, 1100–1115 (2020)

    Article  Google Scholar 

  20. Li, R., Pan, J., Li, Z., Tang, J.: Single image dehazing via conditional generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8202–8211 (2018)

    Google Scholar 

  21. Li, R., Cheong, L.F., Tan, R.T.: Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1633–1642 (2019)

    Google Scholar 

  22. Li, R., Tan, R.T., Cheong, L.F.: All in one bad weather removal using architectural search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3175–3185 (2020)

    Google Scholar 

  23. Li, X., Wu, J., Lin, Z., Liu, H., Zha, H.: Recurrent squeeze-and-excitation context aggregation net for single image deraining. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 262–277. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_16

    Chapter  Google Scholar 

  24. Li, Y., Tan, R.T., Guo, X., Lu, J., Brown, M.S.: Rain streak removal using layer priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2736–2744 (2016)

    Google Scholar 

  25. Lin, H., Li, Y., Fu, X., Ding, X., Huang, Y., Paisley, J.: Rain o’er me: Synthesizing real rain to derain with data distillation. IEEE Trans. Image Process. 29, 7668–7680 (2020)

    Article  MATH  Google Scholar 

  26. Luo, Y., Xu, Y., Ji, H.: Removing rain from a single image via discriminative sparse coding. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3397–3405 (2015)

    Google Scholar 

  27. Qu, Y., Chen, Y., Huang, J., Xie, Y.: Enhanced pix2pix dehazing network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8160–8168 (2019)

    Google Scholar 

  28. Ren, D., Zuo, W., Hu, Q., Zhu, P., Meng, D.: Progressive image deraining networks: a better and simpler baseline. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3937–3946 (2019)

    Google Scholar 

  29. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems 28 (2015)

    Google Scholar 

  30. Ren, W., Liu, S., Zhang, H., Pan, J., Cao, X., Yang, M.-H.: Single image dehazing via multi-scale convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 154–169. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_10

    Chapter  Google Scholar 

  31. Ren, W., et al.: Gated fusion network for single image dehazing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3253–3261 (2018)

    Google Scholar 

  32. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  33. Shao, Y., Li, L., Ren, W., Gao, C., Sang, N.: Domain adaptation for image dehazing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2808–2817 (2020)

    Google Scholar 

  34. Tu, Z., et al.: Maxim: Multi-axis MLP for image processing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5769–5780 (2022)

    Google Scholar 

  35. Wang, G., Sun, C., Sowmya, A.: ERL-Net: entangled representation learning for single image de-raining. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5644–5652 (2019)

    Google Scholar 

  36. Wang, H., Xie, Q., Zhao, Q., Meng, D.: A model-driven deep neural network for single image rain removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3103–3112 (2020)

    Google Scholar 

  37. Wang, H., Yue, Z., Xie, Q., Zhao, Q., Zheng, Y., Meng, D.: From rain generation to rain removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14791–14801 (2021)

    Google Scholar 

  38. Wang, T., Yang, X., Xu, K., Chen, S., Zhang, Q., Lau, R.W.: Spatial attentive single-image deraining with a high quality real rain dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12270–12279 (2019)

    Google Scholar 

  39. Wang, Y., Song, Y., Ma, C., Zeng, B.: Rethinking image deraining via rain streaks and vapors. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12362, pp. 367–382. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58520-4_22

    Chapter  Google Scholar 

  40. Wei, W., Meng, D., Zhao, Q., Xu, Z., Wu, Y.: Semi-supervised transfer learning for image rain removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3877–3886 (2019)

    Google Scholar 

  41. Wu, H., et al.: Contrastive learning for compact single image dehazing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10551–10560 (2021)

    Google Scholar 

  42. Yang, T., Priebe, C.E.: The effect of model misspecification on semi-supervised classification. IEEE Trans. Pattern Anal. Mach. Intell. 33(10), 2093–2103 (2011)

    Article  Google Scholar 

  43. Yang, W., Tan, R.T., Feng, J., Liu, J., Guo, Z., Yan, S.: Deep joint rain detection and removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1357–1366 (2017)

    Google Scholar 

  44. Yasarla, R., Patel, V.M.: Uncertainty guided multi-scale residual learning-using a cycle spinning CNN for single image de-raining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8405–8414 (2019)

    Google Scholar 

  45. Yasarla, R., Patel, V.M.: Confidence measure guided single image de-raining. IEEE Trans. Image Process. 29, 4544–4555 (2020)

    Article  MATH  Google Scholar 

  46. Yasarla, R., Patel, V.M.: Learning to restore images degraded by atmospheric turbulence using uncertainty. In: 2021 IEEE International Conference on Image Processing (ICIP), pp. 1694–1698. IEEE (2021)

    Google Scholar 

  47. Yasarla, R., Patel, V.M.: CNN-based restoration of a single face image degraded by atmospheric turbulence. Behavior, and Identity Science, IEEE Transactions on Biometrics (2022)

    Google Scholar 

  48. Yasarla, R., Sindagi, V.A., Patel, V.M.: Syn2real transfer learning for image deraining using gaussian processes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  49. Yasarla, R., Sindagi, V.A., Patel, V.M.: Unsupervised restoration of weather-affected images using deep gaussian process-based cyclegan. arXiv preprint arXiv:2204.10970 (2022)

  50. Yasarla, R., Valanarasu, J.M.J., Patel, V.M.: Exploring overcomplete representations for single image deraining using CNNs. IEEE J. Select. Top. Sig. Process. 15(2), 229–239 (2020)

    Article  Google Scholar 

  51. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5728–5739 (2022)

    Google Scholar 

  52. Zamir, S.W., et al.: Multi-stage progressive image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14821–14831 (2021)

    Google Scholar 

  53. Zhang, H., Patel, V.M.: Densely connected pyramid dehazing network. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 3194–3203 (2018)

    Google Scholar 

  54. Zhang, H., Patel, V.M.: Density-aware single image de-raining using a multi-stream dense network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 695–704 (2018)

    Google Scholar 

  55. Zhang, H., Sindagi, V., Patel, V.M.: Image de-raining using a conditional generative adversarial network. IEEE Trans. Circuits Syst. Video Technol. 30(11), 3943–3956 (2019)

    Article  Google Scholar 

  56. Zhang, H., Sindagi, V., Patel, V.M.: Joint transmission map estimation and dehazing using deep networks. IEEE Trans. Circuits Syst. Video Technol. 30(7), 1975–1986 (2019)

    Google Scholar 

  57. Zhang, J., et al.: Hierarchical density-aware dehazing network. IEEE Transactions on Cybernetics (2021)

    Google Scholar 

  58. Zhou, M., et al.: Image de-raining via continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4907–4916 (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Johns Hopkins University, Baltimore, MD, 21218, USA

    Rajeev Yasarla, Carey E. Priebe & Vishal M. Patel

Authors
  1. Rajeev Yasarla
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carey E. Priebe
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vishal M. Patel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rajeev Yasarla .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 7221 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yasarla, R., Priebe, C.E., Patel, V.M. (2022). ART-SS: An Adaptive Rejection Technique for Semi-supervised Restoration for Adverse Weather-Affected Images. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13678. Springer, Cham. https://doi.org/10.1007/978-3-031-19797-0_40

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19797-0_40

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19796-3

  • Online ISBN: 978-3-031-19797-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics