Skip to main content
Log in

Remote sensing image cloud removal based on multi-scale spatial information perception

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Remote sensing imagery is indispensable in diverse domains, including geographic information systems, climate monitoring, agricultural planning, and disaster management. Nonetheless, cloud cover can drastically degrade the utility and quality of these images. Current deep learning-based cloud removal methods rely on convolutional neural networks to extract features at the same scale, which can overlook detailed and global information, resulting in suboptimal cloud removal performance. To overcome these challenges, we develop a method for cloud removal that leverages multi-scale spatial information perception. Our technique employs convolution kernels of various sizes, enabling the integration of both global semantic information and local detail information. An attention mechanism enhances this process by targeting key areas within the images, and dynamically adjusting channel weights to improve feature reconstruction. We compared our method with current popular cloud removal methods across three datasets, and the results show that our proposed method improves metrics such as PSNR, SSIM, and cosine similarity, verifying the effectiveness of our method in cloud removal.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

The dataset in this paper is derived from the publicly available dataset RICE from https://github.com/BUPTLdy/RICE_DATASET and SingleImage from https://doi.org/10.7910/DVN/BSETKZ. Aircraft images for the application section are sourced from https://www.kaggle.com/datasets/airbusgeo/airbus-aircrafts-sample-dataset and https://eod-grss-ieee.com/dataset-detail/cEgweVFERDB2S0lqL1pvTUdlMnVzUT09.

References

  1. Sishodia, R.P., Ray, R.L., Singh, S.K.: Applications of remote sensing in precision agriculture: a review. Remote Sens. 12(19), 3136 (2020)

    Article  Google Scholar 

  2. Shahtahmassebi, A.R., Li, C., Fan, Y., Wu, Y., Gan, M., Wang, K., Malik, A., Blackburn, G.A., et al.: Remote sensing of urban green spaces: a review. Urban For. Urban Green. 57, 126946 (2021)

    Article  Google Scholar 

  3. Wan, H., Tang, Y., Jing, L., Li, H., Qiu, F., Wu, W.: Tree species classification of forest stands using multisource remote sensing data. Remote Sens. 13(1), 144 (2021)

    Article  Google Scholar 

  4. Li, Y., Chen, W., Zhang, Y., Tao, C., Xiao, R., Tan, Y.: Accurate cloud detection in high-resolution remote sensing imagery by weakly supervised deep learning. Remote Sens. Environ. 250, 112045 (2020)

    Article  Google Scholar 

  5. Jin, M., Wang, P., Li, Y.: Hya-gan: remote sensing image cloud removal based on hybrid attention generation adversarial network. Int. J. Remote Sens. 45(6), 1755–1773 (2024)

    Article  Google Scholar 

  6. Pan, H.: Cloud removal for remote sensing imagery via spatial attention generative adversarial network. arXiv preprint arXiv:2009.13015 (2020)

  7. Ye, Y., Zhang, J., Zhou, L., Li, J., Ren, X., Fan, J.: Optical and sar image fusion based on complementary feature decomposition and visual saliency features. IEEE Trans. Geosci. Remote Sens. 62, 1–15 (2024)

    Google Scholar 

  8. Czerkawski, M., Atkinson, R., Michie, C., Tachtatzis, C.: Satellitecloudgenerator: controllable cloud and shadow synthesis for multi-spectral optical satellite images. Remote Sens. 15(17), 4138 (2023)

    Article  Google Scholar 

  9. Li, Y., Wei, F., Zhang, Y., Chen, W., Ma, J.: Hs2p: Hierarchical spectral and structure-preserving fusion network for multimodal remote sensing image cloud and shadow removal. Inf. Fusion 94, 215–228 (2023)

    Article  Google Scholar 

  10. Wang, J.-L., Zhao, X.-L., Li, H.-C., Cao, K.-X., Miao, J., Huang, T.-Z.: Unsupervised domain factorization network for thick cloud removal of multi-temporal remotely sensed images. IEEE Trans. Geosci. Remote Sens. 1, 7 (2023). https://doi.org/10.1109/TGRS.2023.3303169

    Article  Google Scholar 

  11. Long, C., Li, X., Jing, Y., Shen, H., et al.: Bishift networks for thick cloud removal with multitemporal remote sensing images. Int. J. Intell. Syst. 2023(1), 9953198 (2023)

    Google Scholar 

  12. Ebel, P., Garnot, V.S.F., Schmitt, M., Wegner, J.D., Zhu, X.X.: Uncrtaints: Uncertainty quantification for cloud removal in optical satellite time series. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2086–2096 (2023)

  13. Ji, T.-Y., Yokoya, N., Zhu, X.X., Huang, T.-Z.: Nonlocal tensor completion for multitemporal remotely sensed images’ inpainting. IEEE Trans. Geosci. Remote Sens. 56(6), 3047–3061 (2018)

    Article  Google Scholar 

  14. Xu, F., Shi, Y., Ebel, P., Yu, L., Xia, G.-S., Yang, W., Zhu, X.X.: Glf-cr: Sar-enhanced cloud removal with global-local fusion. ISPRS J. Photogramm. Remote Sens. 192, 268–278 (2022)

    Article  Google Scholar 

  15. Zheng, W.-J., Zhao, X.-L., Zheng, Y.-B., Lin, J., Zhuang, L., Huang, T.-Z.: Spatial-spectral-temporal connective tensor network decomposition for thick cloud removal. ISPRS J. Photogramm. Remote Sens. 199, 182–194 (2023)

    Article  Google Scholar 

  16. Cheng, Q., Shen, H., Zhang, L., Yuan, Q., Zeng, C.: Cloud removal for remotely sensed images by similar pixel replacement guided with a spatio-temporal mrf model. ISPRS J. Photogramm. Remote Sens. 92, 54–68 (2014)

    Article  Google Scholar 

  17. Shen, H., Li, H., Qian, Y., Zhang, L., Yuan, Q.: An effective thin cloud removal procedure for visible remote sensing images. ISPRS J. Photogramm. Remote Sens. 96, 224–235 (2014)

    Article  Google Scholar 

  18. Meraner, A., Ebel, P., Zhu, X.X., Schmitt, M.: Cloud removal in sentinel-2 imagery using a deep residual neural network and sar-optical data fusion. ISPRS J. Photogramm. Remote Sens. 166, 333–346 (2020)

    Article  Google Scholar 

  19. Ji, S., Dai, P., Lu, M., Zhang, Y.: Simultaneous cloud detection and removal from bitemporal remote sensing images using cascade convolutional neural networks. IEEE Trans. Geosci. Remote Sens. 59(1), 732–748 (2020)

    Article  Google Scholar 

  20. Zhang, Q., Yuan, Q., Li, Z., Sun, F., Zhang, L.: Combined deep prior with low-rank tensor svd for thick cloud removal in multitemporal images. ISPRS J. Photogramm. Remote Sens. 177, 161–173 (2021)

    Article  Google Scholar 

  21. Qin, M., Xie, F., Li, W., Shi, Z., Zhang, H.: Dehazing for multispectral remote sensing images based on a convolutional neural network with the residual architecture. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 11(5), 1645–1655 (2018)

    Article  Google Scholar 

  22. Li, W., Li, Y., Chen, D., Chan, J.C.-W.: Thin cloud removal with residual symmetrical concatenation network. ISPRS J. Photogramm. Remote Sens. 153, 137–150 (2019)

    Article  Google Scholar 

  23. Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

  24. Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

  25. Qian, R., Tan, R.T., Yang, W., Su, J., Liu, J.: Attentive generative adversarial network for raindrop removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2482–2491 (2018)

  26. Enomoto, K., Sakurada, K., Wang, W., Fukui, H., Matsuoka, M., Nakamura, R., Kawaguchi, N.: Filmy cloud removal on satellite imagery with multispectral conditional generative adversarial nets. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 48–56 (2017)

  27. Engin, D., Genç, A., Kemal Ekenel, H.: Cycle-dehaze: Enhanced cyclegan for single image dehazing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 825–833 (2018)

  28. Wu, Z., Li, J., Wang, Y., Hu, Z., Molinier, M.: Self-attentive generative adversarial network for cloud detection in high resolution remote sensing images. IEEE Geosci. Remote Sens. Lett. 17(10), 1792–1796 (2019)

    Article  Google Scholar 

  29. Xu, M., Deng, F., Jia, S., Jia, X., Plaza, A.J.: Attention mechanism-based generative adversarial networks for cloud removal in landsat images. Remote Sens. Environ. 271, 112902 (2022)

    Article  Google Scholar 

  30. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Advances in neural information processing systems 27 (2014)

  31. Sarukkai, V., Jain, A., Uzkent, B., Ermon, S.: Cloud removal from satellite images using spatiotemporal generator networks. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1796–1805 (2020)

  32. Zhao, Y., Shen, S., Hu, J., Li, Y., Pan, J.: Cloud removal using multimodal gan with adversarial consistency loss. IEEE Geosci. Remote Sens. Lett. 19, 1–5 (2021)

    Google Scholar 

  33. Ma, X., Huang, Y., Zhang, X., Pun, M.-O., Huang, B.: Cloud-egan: rethinking cyclegan from a feature enhancement perspective for cloud removal by combining cnn and transformer. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 16, 4999–5012 (2023)

    Article  Google Scholar 

  34. Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)

  35. He, Y., He, N., Zhang, R., Yan, K., Yu, H.: Multi-scale feature balance enhancement network for pedestrian detection. Multimed. Syst. 28(3), 1135–1145 (2022)

    Article  Google Scholar 

  36. Yang, Y., Xia, T., Li, D., Zhang, Z., Xie, G.: A multi-scale feature fusion spatial-channel attention model for background subtraction. Multimed. Syst. 29(6), 3609–3623 (2023)

    Article  Google Scholar 

  37. Qiu, C., Song, Y., Liu, Z., Yin, J., Han, K., Liu, Y.: Cmfcunet: cascaded multi-scale feature calibration unet for pancreas segmentation. Multimed. Syst. 29(2), 871–886 (2023)

    Article  Google Scholar 

  38. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 801–818 (2018)

  39. Jiang, H., Lu, N.: Multi-scale residual convolutional neural network for haze removal of remote sensing images. Remote Sens. 10(6), 945 (2018)

    Article  Google Scholar 

  40. Yeh, C.-H., Huang, C.-H., Kang, L.-W.: Multi-scale deep residual learning-based single image haze removal via image decomposition. IEEE Trans. Image Process. 29, 3153–3167 (2019)

    Article  Google Scholar 

  41. Sun, H., Luo, Z., Ren, D., Hu, W., Du, B., Yang, W., Wan, J., Zhang, L.: Partial Siamese with multiscale bi-codec networks for remote sensing image haze removal. IEEE Trans. Geosci. Remote Sens. 1, 7 (2023)

    Google Scholar 

  42. Lin, D., Xu, G., Wang, X., Wang, Y., Sun, X., Fu, K.: A remote sensing image dataset for cloud removal. arXiv preprint arXiv:1901.00600 (2019)

  43. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 694–711). Springer (2016)

  44. Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., et al.: Speed/accuracy trade-offs for modern convolutional object detectors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7310–7311 (2017)

  45. Airbusgeo: Airbus Aircraft Detection. https://www.kaggle.com/datasets/airbusgeo/airbus-aircrafts-sample-dataset. Accessed 14 Mar 2024 (2024)

  46. Shermeyer, J., Hossler, T., Van Etten, A., Hogan, D., Lewis, R., Kim, D.: Rareplanes: Synthetic data takes flight. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 207–217 (2021)

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 61671480 and Grant 62372468; in part by the Natural Science Foundation of Shandong Province, China, under Grant ZR2019MF073 and Grant ZR2023MF008; in part by the Qingdao Natural Science Foundation under Grant 23-2-1-161-zyyd-jch; in part by the Major Scientific and Technological Projects of CNPC under Grant ZD2019-183-008; and in part by the Major Basic Research Projects in Shandong Province under Grant ZR2023ZD32.

Funding

The National Natural Science Foundation of China (61671480), the Natural Science Foundation of Shandong Province, China (ZR2023MF008, ZR2019MF073), the Qingdao Natural Science Foundation (23-2-1-161-zyyd-jch), the Major Scientific and Technological Projects of CNPC (ZD2019-183-008), the Major Basic Research Projects in Shandong Province (ZR2023ZD32).

Author information

Authors and Affiliations

Authors

Contributions

Dou, A. and Hao, Y. proposed the idea, designed and performed the experiments, and wrote the paper. Other authors propose revisions to the paper. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Baodi Liu.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Communicated by Haojie Li.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dou, A., Hao, Y., Liu, W. et al. Remote sensing image cloud removal based on multi-scale spatial information perception. Multimedia Systems 30, 249 (2024). https://doi.org/10.1007/s00530-024-01442-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00530-024-01442-5

Keywords

Navigation