Skip to main content
SpringerLink
Log in

Cloud Imputation for Multi-sensor Remote Sensing Imagery with Style Transfer

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases: Applied Data Science and Demo Track (ECML PKDD 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14175))

  • 658 Accesses

Abstract

Widely used optical remote sensing images are often contaminated by clouds. The missing or cloud-contaminated data leads to incorrect predictions by the downstream machine learning tasks. However, the availability of multi-sensor remote sensing imagery has great potential for improving imputation under clouds. Existing cloud imputation methods could generally preserve the spatial structure in the imputed regions, however, the spectral distribution does not match the target image due to differences in sensor characteristics and temporal differences. In this paper, we present a novel deep learning-based multi-sensor imputation technique inspired by the computer vision-based style transfer. The proposed deep learning framework consists of two modules: (i) cluster-based attentional instance normalization (CAIN), and (ii) adaptive instance normalization (AdaIN). The combined module, CAINA, exploits the style information from cloud-free regions. These regions (land cover) were obtained through clustering to reduce the style differences between the target and predicted image patches. We have conducted extensive experiments and made comparisons against the state-of-the-art methods using a benchmark dataset with images from Landsat-8 and Sentinel-2 satellites. Our experiments show that the proposed CAINA is at least 24.49% better on MSE and 18.38% better on cloud MSE as compared to state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://landsat.gsfc.nasa.gov/satellites/landsat-8/.

  2. 2.

    https://sentinel.esa.int/web/sentinel/missions/sentinel-2.

  3. 3.

    https://github.com/YifanZhao0822/CAINA.

References

  1. Cresson, R., Ienco, D., Gaetano, R., Ose, K., Minh, D.H.T.: Optical image gap filling using deep convolutional autoencoder from optical and radar images. In: IGARSS 2019–2019 IEEE International Geoscience and Remote Sensing Symposium, pp. 218–221. IEEE (2019)

    Google Scholar 

  2. Cresson, R., et al.: Comparison of convolutional neural networks for cloudy optical images reconstruction from single or multitemporal joint SAR and optical images. arXiv preprint arXiv:2204.00424 (2022)

  3. Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style. arXiv preprint arXiv:1610.07629 (2016)

  4. Ebel, P., Meraner, A., Schmitt, M., Zhu, X.X.: Multisensor data fusion for cloud removal in global and all-season sentinel-2 imagery. IEEE Trans. Geosci. Remote Sens. 59(7), 5866–5878 (2020)

    Article  Google Scholar 

  5. Ebel, P., Xu, Y., Schmitt, M., Zhu, X.X.: SEN12MS-CR-TS: a remote-sensing data set for multimodal multitemporal cloud removal. IEEE Trans. Geosci. Remote Sens. 60, 1–14 (2022)

    Article  Google Scholar 

  6. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016)

    Google Scholar 

  7. Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369. IEEE (2010)

    Google Scholar 

  8. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1501–1510 (2017)

    Google Scholar 

  9. Kang, S.H., Choi, Y., Choi, J.Y.: Restoration of missing patterns on satellite infrared sea surface temperature images due to cloud coverage using deep generative inpainting network. J. Mar. Sci. Eng. 9(3), 310 (2021)

    Article  Google Scholar 

  10. King, M.D., Platnick, S., Menzel, W.P., Ackerman, S.A., Hubanks, P.A.: Spatial and temporal distribution of clouds observed by MODIS onboard the terra and aqua satellites. IEEE Trans. Geosci. Remote Sens. 51(7), 3826–3852 (2013)

    Article  Google Scholar 

  11. Kruse, F.A., et al.: The spectral image processing system (SIPS)–interactive visualization and analysis of imaging spectrometer data. Remote Sens. Environ. 44(2–3), 145–163 (1993)

    Article  Google Scholar 

  12. Li, X., Guo, Q., Lin, D., Li, P., Feng, W., Wang, S.: MISF: multi-level interactive siamese filtering for high-fidelity image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1869–1878 (2022)

    Google Scholar 

  13. Ma, W., et al.: A novel adaptive hybrid fusion network for multiresolution remote sensing images classification. IEEE Trans. Geosci. Remote Sens. 60, 1–17 (2021)

    Google Scholar 

  14. Meraner, A., Ebel, P., Zhu, X.X., Schmitt, M.: Cloud removal in sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion. ISPRS J. Photogramm. Remote. Sens. 166, 333–346 (2020)

    Article  Google Scholar 

  15. Park, T., Efros, A.A., Zhang, R., Zhu, J.-Y.: Contrastive learning for unpaired image-to-image translation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 319–345. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_19

    Chapter  Google Scholar 

  16. Qu, J., Shi, Y., Xie, W., Li, Y., Wu, X., Du, Q.: MSSL: hyperspectral and panchromatic images fusion via multiresolution spatial-spectral feature learning networks. IEEE Trans. Geosci. Remote Sens. 60, 1–13 (2021)

    Google Scholar 

  17. Requena-Mesa, C., Benson, V., Reichstein, M., Runge, J., Denzler, J.: Earthnet 2021: a large-scale dataset and challenge for earth surface forecasting as a guided video prediction task. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1132–1142 (2021)

    Google Scholar 

  18. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  19. Roy, D.P., et al.: Multi-temporal MODIS-Landsat data fusion for relative radiometric normalization, gap filling, and prediction of landsat data. Remote Sens. Environ. 112(6), 3112–3130 (2008)

    Article  Google Scholar 

  20. Rudner, T.G., et al.: Multi3Net: segmenting flooded buildings via fusion of multiresolution, multisensor, and multitemporal satellite imagery. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 702–709 (2019)

    Google Scholar 

  21. Singh, P., Komodakis, N.: Cloud-Gan: cloud removal for sentinel-2 imagery using a cyclic consistent generative adversarial networks. In: IGARSS 2018–2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 1772–1775. IEEE (2018)

    Google Scholar 

  22. Stock, A., et al.: Comparison of cloud-filling algorithms for marine satellite data. Remote Sens. 12(20), 3313 (2020)

    Article  Google Scholar 

  23. Sun, Z., Zhou, W., Ding, C., Xia, M.: Multi-resolution transformer network for building and road segmentation of remote sensing image. ISPRS Int. J. Geo Inf. 11(3), 165 (2022)

    Article  Google Scholar 

  24. Ulyanov, D., Vedaldi, A., Lempitsky, V.: Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6924–6932 (2017)

    Google Scholar 

  25. Varshney, D., Persello, C., Gupta, P.K., Nikam, B.R.: Multiresolution fully convolutional networks to detect clouds and snow through optical satellite images. arXiv preprint arXiv:2201.02350 (2022)

  26. Wang, L., Weng, L., Xia, M., Liu, J., Lin, H.: Multi-resolution supervision network with an adaptive weighted loss for desert segmentation. Remote Sens. 13(11), 2054 (2021)

    Article  Google Scholar 

  27. Wang, L., Zhang, C., Li, R., Duan, C., Meng, X., Atkinson, P.M.: Scale-aware neural network for semantic segmentation of multi-resolution remote sensing images. Remote Sens. 13(24), 5015 (2021)

    Article  Google Scholar 

  28. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  29. Weiss, D.J., Atkinson, P.M., Bhatt, S., Mappin, B., Hay, S.I., Gething, P.W.: An effective approach for gap-filling continental scale remotely sensed time-series. ISPRS J. Photogramm. Remote. Sens. 98, 106–118 (2014)

    Article  Google Scholar 

  30. Yadav, J., Sharma, M.: A review of k-mean algorithm. Int. J. Eng. Trends Technol. 4(7), 2972–2976 (2013)

    Google Scholar 

  31. Yang, X., Zhao, Y., Vatsavai, R.R.: Deep residual network with multi-image attention for imputing under clouds in satellite imagery. In: 2022 27th International Conference on Pattern Recognition (ICPR). IEEE (2022)

    Google Scholar 

  32. Yu, W., Zhang, X., Pun, M.O., Liu, M.: A hybrid model-based and data-driven approach for cloud removal in satellite imagery using multi-scale distortion-aware networks. In: 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pp. 7160–7163. IEEE (2021)

    Google Scholar 

  33. Zhang, Q., Yuan, Q., Zeng, C., Li, X., Wei, Y.: Missing data reconstruction in remote sensing image with a unified spatial-temporal-spectral deep convolutional neural network. IEEE Trans. Geosci. Remote Sens. 56(8), 4274–4288 (2018)

    Article  Google Scholar 

  34. Zhao, Y., Yang, X., Vatsavai, R.R.: Multi-stream deep residual network for cloud imputation using multi-resolution remote sensing imagery. In: 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 97–104. IEEE (2022)

    Google Scholar 

  35. Zhao, Y., Shen, S., Hu, J., Li, Y., Pan, J.: Cloud removal using multimodal GAN with adversarial consistency loss. IEEE Geosci. Remote Sens. Lett. 19, 1–5 (2021)

    Google Scholar 

  36. Zhu, H., Ma, W., Li, L., Jiao, L., Yang, S., Hou, B.: A dual-branch attention fusion deep network for multiresolution remote-sensing image classification. Inf. Fusion 58, 116–131 (2020)

    Article  Google Scholar 

  37. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

    Google Scholar 

Download references

Acknowledgments

This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract #2021-21040700001. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. We would like to thank Benjamin Raskob at ARA for useful feedback on this project.

Author information

Authors and Affiliations

  1. Computer Science Department, North Carolina State University, Raleigh, USA

    Yifan Zhao, Xian Yang & Ranga Raju Vatsavai

Authors
  1. Yifan Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xian Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ranga Raju Vatsavai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ranga Raju Vatsavai .

Editor information

Editors and Affiliations

  1. CENTAI, Turin, Italy

    Gianmarco De Francisci Morales

  2. NYU and Two Sigma, New York, NY, USA

    Claudia Perlich

  3. Netflix, Los Angeles, CA, USA

    Natali Ruchansky

  4. Telefonica Research, Barcelona, Spain

    Nicolas Kourtellis

  5. Politecnico di Torino, Turin, Italy

    Elena Baralis

  6. CENTAI, Turin, Italy

    Francesco Bonchi

Ethical Statement

Ethical Statement

Our proposed method improves cloud imputation performance. Remote sensing imagery has been widely used in applications ranging from land-use land-cover mapping to national security. By improving the imputation performance, we are directly improving the downstream applications such as assessing damages due to natural disasters, forest fires, and climate impacts. Our work does not have direct ethical implications or adverse impacts on humans.

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhao, Y., Yang, X., Vatsavai, R.R. (2023). Cloud Imputation for Multi-sensor Remote Sensing Imagery with Style Transfer. In: De Francisci Morales, G., Perlich, C., Ruchansky, N., Kourtellis, N., Baralis, E., Bonchi, F. (eds) Machine Learning and Knowledge Discovery in Databases: Applied Data Science and Demo Track. ECML PKDD 2023. Lecture Notes in Computer Science(), vol 14175. Springer, Cham. https://doi.org/10.1007/978-3-031-43430-3_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43430-3_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43429-7

  • Online ISBN: 978-3-031-43430-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation