Abstract
Remote sensing imagery is indispensable in diverse domains, including geographic information systems, climate monitoring, agricultural planning, and disaster management. Nonetheless, cloud cover can drastically degrade the utility and quality of these images. Current deep learning-based cloud removal methods rely on convolutional neural networks to extract features at the same scale, which can overlook detailed and global information, resulting in suboptimal cloud removal performance. To overcome these challenges, we develop a method for cloud removal that leverages multi-scale spatial information perception. Our technique employs convolution kernels of various sizes, enabling the integration of both global semantic information and local detail information. An attention mechanism enhances this process by targeting key areas within the images, and dynamically adjusting channel weights to improve feature reconstruction. We compared our method with current popular cloud removal methods across three datasets, and the results show that our proposed method improves metrics such as PSNR, SSIM, and cosine similarity, verifying the effectiveness of our method in cloud removal.






Similar content being viewed by others
Data availability
The dataset in this paper is derived from the publicly available dataset RICE from https://github.com/BUPTLdy/RICE_DATASET and SingleImage from https://doi.org/10.7910/DVN/BSETKZ. Aircraft images for the application section are sourced from https://www.kaggle.com/datasets/airbusgeo/airbus-aircrafts-sample-dataset and https://eod-grss-ieee.com/dataset-detail/cEgweVFERDB2S0lqL1pvTUdlMnVzUT09.
References
Sishodia, R.P., Ray, R.L., Singh, S.K.: Applications of remote sensing in precision agriculture: a review. Remote Sens. 12(19), 3136 (2020)
Shahtahmassebi, A.R., Li, C., Fan, Y., Wu, Y., Gan, M., Wang, K., Malik, A., Blackburn, G.A., et al.: Remote sensing of urban green spaces: a review. Urban For. Urban Green. 57, 126946 (2021)
Wan, H., Tang, Y., Jing, L., Li, H., Qiu, F., Wu, W.: Tree species classification of forest stands using multisource remote sensing data. Remote Sens. 13(1), 144 (2021)
Li, Y., Chen, W., Zhang, Y., Tao, C., Xiao, R., Tan, Y.: Accurate cloud detection in high-resolution remote sensing imagery by weakly supervised deep learning. Remote Sens. Environ. 250, 112045 (2020)
Jin, M., Wang, P., Li, Y.: Hya-gan: remote sensing image cloud removal based on hybrid attention generation adversarial network. Int. J. Remote Sens. 45(6), 1755–1773 (2024)
Pan, H.: Cloud removal for remote sensing imagery via spatial attention generative adversarial network. arXiv preprint arXiv:2009.13015 (2020)
Ye, Y., Zhang, J., Zhou, L., Li, J., Ren, X., Fan, J.: Optical and sar image fusion based on complementary feature decomposition and visual saliency features. IEEE Trans. Geosci. Remote Sens. 62, 1–15 (2024)
Czerkawski, M., Atkinson, R., Michie, C., Tachtatzis, C.: Satellitecloudgenerator: controllable cloud and shadow synthesis for multi-spectral optical satellite images. Remote Sens. 15(17), 4138 (2023)
Li, Y., Wei, F., Zhang, Y., Chen, W., Ma, J.: Hs2p: Hierarchical spectral and structure-preserving fusion network for multimodal remote sensing image cloud and shadow removal. Inf. Fusion 94, 215–228 (2023)
Wang, J.-L., Zhao, X.-L., Li, H.-C., Cao, K.-X., Miao, J., Huang, T.-Z.: Unsupervised domain factorization network for thick cloud removal of multi-temporal remotely sensed images. IEEE Trans. Geosci. Remote Sens. 1, 7 (2023). https://doi.org/10.1109/TGRS.2023.3303169
Long, C., Li, X., Jing, Y., Shen, H., et al.: Bishift networks for thick cloud removal with multitemporal remote sensing images. Int. J. Intell. Syst. 2023(1), 9953198 (2023)
Ebel, P., Garnot, V.S.F., Schmitt, M., Wegner, J.D., Zhu, X.X.: Uncrtaints: Uncertainty quantification for cloud removal in optical satellite time series. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2086–2096 (2023)
Ji, T.-Y., Yokoya, N., Zhu, X.X., Huang, T.-Z.: Nonlocal tensor completion for multitemporal remotely sensed images’ inpainting. IEEE Trans. Geosci. Remote Sens. 56(6), 3047–3061 (2018)
Xu, F., Shi, Y., Ebel, P., Yu, L., Xia, G.-S., Yang, W., Zhu, X.X.: Glf-cr: Sar-enhanced cloud removal with global-local fusion. ISPRS J. Photogramm. Remote Sens. 192, 268–278 (2022)
Zheng, W.-J., Zhao, X.-L., Zheng, Y.-B., Lin, J., Zhuang, L., Huang, T.-Z.: Spatial-spectral-temporal connective tensor network decomposition for thick cloud removal. ISPRS J. Photogramm. Remote Sens. 199, 182–194 (2023)
Cheng, Q., Shen, H., Zhang, L., Yuan, Q., Zeng, C.: Cloud removal for remotely sensed images by similar pixel replacement guided with a spatio-temporal mrf model. ISPRS J. Photogramm. Remote Sens. 92, 54–68 (2014)
Shen, H., Li, H., Qian, Y., Zhang, L., Yuan, Q.: An effective thin cloud removal procedure for visible remote sensing images. ISPRS J. Photogramm. Remote Sens. 96, 224–235 (2014)
Meraner, A., Ebel, P., Zhu, X.X., Schmitt, M.: Cloud removal in sentinel-2 imagery using a deep residual neural network and sar-optical data fusion. ISPRS J. Photogramm. Remote Sens. 166, 333–346 (2020)
Ji, S., Dai, P., Lu, M., Zhang, Y.: Simultaneous cloud detection and removal from bitemporal remote sensing images using cascade convolutional neural networks. IEEE Trans. Geosci. Remote Sens. 59(1), 732–748 (2020)
Zhang, Q., Yuan, Q., Li, Z., Sun, F., Zhang, L.: Combined deep prior with low-rank tensor svd for thick cloud removal in multitemporal images. ISPRS J. Photogramm. Remote Sens. 177, 161–173 (2021)
Qin, M., Xie, F., Li, W., Shi, Z., Zhang, H.: Dehazing for multispectral remote sensing images based on a convolutional neural network with the residual architecture. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 11(5), 1645–1655 (2018)
Li, W., Li, Y., Chen, D., Chan, J.C.-W.: Thin cloud removal with residual symmetrical concatenation network. ISPRS J. Photogramm. Remote Sens. 153, 137–150 (2019)
Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)
Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)
Qian, R., Tan, R.T., Yang, W., Su, J., Liu, J.: Attentive generative adversarial network for raindrop removal from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2482–2491 (2018)
Enomoto, K., Sakurada, K., Wang, W., Fukui, H., Matsuoka, M., Nakamura, R., Kawaguchi, N.: Filmy cloud removal on satellite imagery with multispectral conditional generative adversarial nets. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 48–56 (2017)
Engin, D., Genç, A., Kemal Ekenel, H.: Cycle-dehaze: Enhanced cyclegan for single image dehazing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 825–833 (2018)
Wu, Z., Li, J., Wang, Y., Hu, Z., Molinier, M.: Self-attentive generative adversarial network for cloud detection in high resolution remote sensing images. IEEE Geosci. Remote Sens. Lett. 17(10), 1792–1796 (2019)
Xu, M., Deng, F., Jia, S., Jia, X., Plaza, A.J.: Attention mechanism-based generative adversarial networks for cloud removal in landsat images. Remote Sens. Environ. 271, 112902 (2022)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Advances in neural information processing systems 27 (2014)
Sarukkai, V., Jain, A., Uzkent, B., Ermon, S.: Cloud removal from satellite images using spatiotemporal generator networks. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1796–1805 (2020)
Zhao, Y., Shen, S., Hu, J., Li, Y., Pan, J.: Cloud removal using multimodal gan with adversarial consistency loss. IEEE Geosci. Remote Sens. Lett. 19, 1–5 (2021)
Ma, X., Huang, Y., Zhang, X., Pun, M.-O., Huang, B.: Cloud-egan: rethinking cyclegan from a feature enhancement perspective for cloud removal by combining cnn and transformer. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 16, 4999–5012 (2023)
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)
He, Y., He, N., Zhang, R., Yan, K., Yu, H.: Multi-scale feature balance enhancement network for pedestrian detection. Multimed. Syst. 28(3), 1135–1145 (2022)
Yang, Y., Xia, T., Li, D., Zhang, Z., Xie, G.: A multi-scale feature fusion spatial-channel attention model for background subtraction. Multimed. Syst. 29(6), 3609–3623 (2023)
Qiu, C., Song, Y., Liu, Z., Yin, J., Han, K., Liu, Y.: Cmfcunet: cascaded multi-scale feature calibration unet for pancreas segmentation. Multimed. Syst. 29(2), 871–886 (2023)
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 801–818 (2018)
Jiang, H., Lu, N.: Multi-scale residual convolutional neural network for haze removal of remote sensing images. Remote Sens. 10(6), 945 (2018)
Yeh, C.-H., Huang, C.-H., Kang, L.-W.: Multi-scale deep residual learning-based single image haze removal via image decomposition. IEEE Trans. Image Process. 29, 3153–3167 (2019)
Sun, H., Luo, Z., Ren, D., Hu, W., Du, B., Yang, W., Wan, J., Zhang, L.: Partial Siamese with multiscale bi-codec networks for remote sensing image haze removal. IEEE Trans. Geosci. Remote Sens. 1, 7 (2023)
Lin, D., Xu, G., Wang, X., Wang, Y., Sun, X., Fu, K.: A remote sensing image dataset for cloud removal. arXiv preprint arXiv:1901.00600 (2019)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 694–711). Springer (2016)
Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., Guadarrama, S., et al.: Speed/accuracy trade-offs for modern convolutional object detectors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7310–7311 (2017)
Airbusgeo: Airbus Aircraft Detection. https://www.kaggle.com/datasets/airbusgeo/airbus-aircrafts-sample-dataset. Accessed 14 Mar 2024 (2024)
Shermeyer, J., Hossler, T., Van Etten, A., Hogan, D., Lewis, R., Kim, D.: Rareplanes: Synthetic data takes flight. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 207–217 (2021)
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 61671480 and Grant 62372468; in part by the Natural Science Foundation of Shandong Province, China, under Grant ZR2019MF073 and Grant ZR2023MF008; in part by the Qingdao Natural Science Foundation under Grant 23-2-1-161-zyyd-jch; in part by the Major Scientific and Technological Projects of CNPC under Grant ZD2019-183-008; and in part by the Major Basic Research Projects in Shandong Province under Grant ZR2023ZD32.
Funding
The National Natural Science Foundation of China (61671480), the Natural Science Foundation of Shandong Province, China (ZR2023MF008, ZR2019MF073), the Qingdao Natural Science Foundation (23-2-1-161-zyyd-jch), the Major Scientific and Technological Projects of CNPC (ZD2019-183-008), the Major Basic Research Projects in Shandong Province (ZR2023ZD32).
Author information
Authors and Affiliations
Contributions
Dou, A. and Hao, Y. proposed the idea, designed and performed the experiments, and wrote the paper. Other authors propose revisions to the paper. All authors have read and agreed to the published version of the manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Additional information
Communicated by Haojie Li.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Dou, A., Hao, Y., Liu, W. et al. Remote sensing image cloud removal based on multi-scale spatial information perception. Multimedia Systems 30, 249 (2024). https://doi.org/10.1007/s00530-024-01442-5
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s00530-024-01442-5