Skip to main content
Springer Nature Link
Log in

Uncertainty-Based Thin Cloud Removal Network via Conditional Variational Autoencoders

  • Conference paper
  • First Online:
Computer Vision – ACCV 2022 (ACCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13843))

Included in the following conference series:

  • 1036 Accesses

Abstract

Existing thin cloud removal methods treat this image restoration task as a point estimation problem, and produce a single cloud-free image following a deterministic pipeline. In this paper, we propose a novel thin cloud removal network via Conditional Variational Autoencoders (CVAE) to generate multiple reasonable cloud-free images for each input cloud image. We analyze the image degradation process with a probabilistic graphical model and design the network in an encoder-decoder fashion. Since the diversity in sampling from the latent space, the proposed method can avoid the shortcoming caused by the inaccuracy of a single estimation. With the uncertainty analysis, we can generate a more accurate clear image based on these multiple predictions. Furthermore, we create a new benchmark dataset with cloud and clear image pairs from real-world scenes, overcoming the problem of poor generalization performance caused by training on synthetic datasets. Quantitative and qualitative experiments show that the proposed method significantly outperforms state-of-the-art methods on real-world cloud images. The source code and dataset are available at https://github.com/haidong-Ding/Cloud-Removal.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Li, B., et al.: Benchmarking single-image dehazing and beyond. IEEE Trans. Image Process. 28(1), 492–505 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  2. Zhang, Z., Zhao, L., Liu, Y., Zhang, S., Yang, J.: Unified density-aware image dehazing and object detection in real-world hazy scenes. In: Proceedings of the Asian Conference on Computer Vision (2020)

    Google Scholar 

  3. Li, B., Peng, X., Wang, Z., Xu, J., Feng, D.: End-to-end united video dehazing and detection. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)

    Google Scholar 

  4. Tu, Z., Chen, X., Yuille, A.L., Zhu, S.C.: Image parsing: unifying segmentation, detection, and recognition. Int. J. Comput. Vis. 63(2), 113–140 (2005)

    Article  Google Scholar 

  5. Tarel, J.P., Hautiere, N., Cord, A., Gruyer, D., Halmaoui, H.: Improved visibility of road scene images under heterogeneous fog. In: 2010 IEEE Intelligent Vehicles Symposium, pp. 478–485. IEEE (2010)

    Google Scholar 

  6. Sakaridis, C., Dai, D., Van Gool, L.: Semantic foggy scene understanding with synthetic data. Int. J. Comput. Vis. 126(9), 973–992 (2018)

    Article  Google Scholar 

  7. Richter, R.: A spatially adaptive fast atmospheric correction algorithm. Int. J. Remote Sens. 17(6), 1201–1214 (1996)

    Article  Google Scholar 

  8. Vermote, E.F., Tanré, D., Deuze, J.L., Herman, M., Morcette, J.J.: Second simulation of the satellite signal in the solar spectrum, 6s: an overview. IEEE Trans. Geosci. Remote Sens. 35(3), 675–686 (1997)

    Article  Google Scholar 

  9. He, K., Sun, J., Tang, X.: Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2010)

    Google Scholar 

  10. Xu, M., Deng, F., Jia, S., Jia, X., Plaza, A.J.: Attention mechanism-based generative adversarial networks for cloud removal in landsat images. Remote Sens. Environ. 271, 112902 (2022)

    Article  Google Scholar 

  11. Li, W., Li, Y., Chen, D., Chan, J.C.W.: Thin cloud removal with residual symmetrical concatenation network. ISPRS J. Photogramm. Remote. Sens. 153, 137–150 (2019)

    Article  Google Scholar 

  12. Zhou, Y., Jing, W., Wang, J., Chen, G., Scherer, R., Damaševičius, R.: MSAR-DefogNet: lightweight cloud removal network for high resolution remote sensing images based on multi scale convolution. IET Image Proc. 16(3), 659–668 (2022)

    Article  Google Scholar 

  13. Zi, Y., Xie, F., Zhang, N., Jiang, Z., Zhu, W., Zhang, H.: Thin cloud removal for multispectral remote sensing images using convolutional neural networks combined with an imaging model. IEEE J. Sel. Top. Appl. Earth Observations Remote Sens. 14, 3811–3823 (2021)

    Article  Google Scholar 

  14. Cai, B., Xu, X., Jia, K., Qing, C., Tao, D.: DehazeNet: an end-to-end system for single image haze removal. IEEE Trans. Image Process. 25(11), 5187–5198 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  15. Li, B., Peng, X., Wang, Z., Xu, J., Feng, D.: AOD-Net: all-in-one dehazing network. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4770–4778 (2017)

    Google Scholar 

  16. Qin, M., Xie, F., Li, W., Shi, Z., Zhang, H.: Dehazing for multispectral remote sensing images based on a convolutional neural network with the residual architecture. IEEE J. Sel. Top. Appl. Earth Observations Remote Sens. 11(5), 1645–1655 (2018)

    Article  Google Scholar 

  17. Zheng, J., Liu, X.Y., Wang, X.: Single image cloud removal using U-net and generative adversarial networks. IEEE Trans. Geosci. Remote Sens. 59(8), 6371–6385 (2020)

    Article  Google Scholar 

  18. Chavez, P.S., Jr.: An improved dark-object subtraction technique for atmospheric scattering correction of multispectral data. Remote Sens. Environ. 24(3), 459–479 (1988)

    Article  Google Scholar 

  19. Fattal, R.: Dehazing using color-lines. ACM Trans. Graph. (TOG) 34(1), 1–14 (2014)

    Article  Google Scholar 

  20. Berman, D., Avidan, S., et al.: Non-local image dehazing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1674–1682 (2016)

    Google Scholar 

  21. Xu, M., Pickering, M., Plaza, A.J., Jia, X.: Thin cloud removal based on signal transmission principles and spectral mixture analysis. IEEE Trans. Geosci. Remote Sens. 54(3), 1659–1669 (2015)

    Article  Google Scholar 

  22. Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Advances in Neural Information Processing Systems, vol. 29 (2016)

    Google Scholar 

  23. Singh, P., Komodakis, N.: Cloud-GAN: cloud removal for sentinel-2 imagery using a cyclic consistent generative adversarial networks. In: 2018 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2018, pp. 1772–1775. IEEE (2018)

    Google Scholar 

  24. Ren, W., Liu, S., Zhang, H., Pan, J., Cao, X., Yang, M.-H.: Single image dehazing via multi-scale convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 154–169. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_10

    Chapter  Google Scholar 

  25. Makarau, A., Richter, R., Müller, R., Reinartz, P.: Haze detection and removal in remotely sensed multispectral imagery. IEEE Trans. Geosci. Remote Sens. 52(9), 5895–5905 (2014)

    Article  Google Scholar 

  26. Narasimhan, S.G., Nayar, S.K.: Vision and the atmosphere. Int. J. Comput. Vis. 48(3), 233–254 (2002)

    Article  MATH  Google Scholar 

  27. Dosovitskiy, A., et al.: An image is worth \(16 \times 16\) words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  28. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

    Google Scholar 

  29. Pan, H.: Cloud removal for remote sensing imagery via spatial attention generative adversarial network. arXiv preprint arXiv:2009.13015 (2020)

  30. Zhang, X., Wang, T., Wang, J., Tang, G., Zhao, L.: Pyramid channel-based feature attention network for image dehazing. Comput. Vis. Image Underst. 197, 103003 (2020)

    Article  Google Scholar 

  31. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

    Google Scholar 

  32. Yu, W., Zhang, X., Pun, M.O., Liu, M.: A hybrid model-based and data-driven approach for cloud removal in satellite imagery using multi-scale distortion-aware networks. In: 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pp. 7160–7163. IEEE (2021)

    Google Scholar 

  33. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  34. Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind’’ image quality analyzer. IEEE Signal Process. Lett. 20(3), 209–212 (2012)

    Article  Google Scholar 

  35. Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 21(12), 4695–4708 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  36. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

Download references

Acknowledgements

This work was supported in part by the National Key Research and Development Program of China under Grant 2019YFC1510905 and in part by the National Natural Science Foundation of China under Grant 61871011.

Author information

Authors and Affiliations

  1. School of Astronautics, Beihang University, Beijing, 100191, China

    Haidong Ding, Yue Zi & Fengying Xie

Authors
  1. Haidong Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yue Zi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fengying Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fengying Xie .

Editor information

Editors and Affiliations

  1. University of Wollongong, Wollongong, NSW, Australia

    Lei Wang

  2. University of Bonn, Bonn, Germany

    Juergen Gall

  3. University of Adelaide, Adelaide, SA, Australia

    Tat-Jun Chin

  4. National Institute of Informatics, Tokyo, Japan

    Imari Sato

  5. Johns Hopkins University, Baltimore, MD, USA

    Rama Chellappa

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1026 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ding, H., Zi, Y., Xie, F. (2023). Uncertainty-Based Thin Cloud Removal Network via Conditional Variational Autoencoders. In: Wang, L., Gall, J., Chin, TJ., Sato, I., Chellappa, R. (eds) Computer Vision – ACCV 2022. ACCV 2022. Lecture Notes in Computer Science, vol 13843. Springer, Cham. https://doi.org/10.1007/978-3-031-26313-2_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-26313-2_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-26312-5

  • Online ISBN: 978-3-031-26313-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics