Skip to main content
Springer Nature Link
Log in

Multi-spectral Gradient Residual Network for Haze Removal in Multi-sensor Remote Sensing Imagery

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track (ECML PKDD 2024)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14950))

Abstract

Remote sensing imagery is widely used for various Earth surface monitoring applications. However, the quality of these observations can be degraded by clouds during image acquisition. Haze, a type of thin cloud, commonly causes atmospheric absorption and scattering of visible light, resulting in partially obscured regions. Haze removal is an active research area with two main approaches: physics-driven computer vision and end-to-end data-driven machine learning. To leverage both approaches, we propose a deep neural network framework that utilizes large-scale multi-sensor data and geometric knowledge from image physics. This is achieved through a multi-spectral gradient residual network. This network transfers structural details from near-infrared (NIR) images, which have better haze penetration, to the visible (RGB) bands. During training, we incorporate a soft constraint using the partially available information under haze conditions. This constraint helps the model maintain atmospheric consistency, a concept commonly used in physical haze models. We validated our model’s performance on a multi-sensor benchmark dataset containing Landsat-8 and Sentinel-2 satellite images. Comparisons with state-of-the-art methods demonstrate significant improvements. Our model achieves a minimum of 18.71% improvement on MSE, 25.9% on SSIM, and 6.25% on MS-SSIM compared to the next best method. It also shows advancements in LPIPS (14.43%) and SAM (8.47%) measures.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Berman, D., Avidan, S., et al.: Non-local image dehazing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1674–1682 (2016)

    Google Scholar 

  2. Cresson, R., Ienco, D., Gaetano, R., Ose, K., Minh, D.H.T.: Optical image gap filling using deep convolutional autoencoder from optical and radar images. In: IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, pp. 218–221. IEEE (2019)

    Google Scholar 

  3. Cresson, R., et al.: Comparison of convolutional neural networks for cloudy optical images reconstruction from single or multitemporal joint SAR and optical images (2022). arXiv preprint arXiv:2204.00424

  4. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009). https://doi.org/10.1109/CVPR.2009.5206848

  5. Ebel, P., Schmitt, M., Zhu, X.X.: Cloud removal in unpaired sentinel-2 imagery using cycle-consistent GAN and SAR-optical data fusion. In: IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium, pp. 2065–2068. IEEE (2020)

    Google Scholar 

  6. Enomoto, K., et al.: Filmy cloud removal on satellite imagery with multispectral conditional generative adversarial nets. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 48–56 (2017)

    Google Scholar 

  7. Fattal, R.: Single image dehazing. ACM Trans. Graph. (TOG) 27(3), 1–9 (2008)

    Article  Google Scholar 

  8. Feng, C., Zhuo, S., Zhang, X., Shen, L., Süsstrunk, S.: Near-infrared guided color image dehazing. In: 2013 IEEE International Conference on Image Processing, pp. 2363–2367. IEEE (2013)

    Google Scholar 

  9. He, K., Sun, J., Tang, X.: Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2010)

    Google Scholar 

  10. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  11. Huang, A.J., Agarwal, S.: Physics informed deep learning for traffic state estimation. In: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1–6. IEEE (2020)

    Google Scholar 

  12. Jia, X., et al.: Physics-guided machine learning for scientific discovery: an application in simulating lake temperature profiles. ACM/IMS Trans. Data Sci. 2(3), 1–26 (2021)

    Article  Google Scholar 

  13. Kang, S.H., Choi, Y., Choi, J.Y.: Restoration of missing patterns on satellite infrared sea surface temperature images due to cloud coverage using deep generative inpainting network. J. Mar. Sci. Eng. 9(3), 310 (2021)

    Article  Google Scholar 

  14. Kanopoulos, N., Vasanthavada, N., Baker, R.L.: Design of an image edge detection filter using the Sobel operator. IEEE J. Solid-State Circ. 23(2), 358–367 (1988)

    Article  Google Scholar 

  15. Kim, S.Y., et al.: Zoom-to-Inpaint: image inpainting with high-frequency details. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 477–487 (2022)

    Google Scholar 

  16. Kruse, F.A., et al.: The spectral image processing system (SIPS)-interactive visualization and analysis of imaging spectrometer data. Remote Sens. Environ. 44(2–3), 145–163 (1993)

    Article  Google Scholar 

  17. Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)

    Google Scholar 

  18. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

    Google Scholar 

  19. Meraner, A., Ebel, P., Zhu, X.X., Schmitt, M.: Cloud removal in sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion. ISPRS J. Photogramm. Remote. Sens. 166, 333–346 (2020)

    Article  Google Scholar 

  20. Pawar, S., San, O., Aksoylu, B., Rasheed, A., Kvamsdal, T.: Physics guided machine learning using simplified theories. Phys. Fluids 33(1), 011701 (2021)

    Article  Google Scholar 

  21. Schaul, L., Fredembach, C., Süsstrunk, S.: Color image dehazing using the near-infrared. In: 2009 16th IEEE International Conference on Image Processing (ICIP), pp. 1629–1632. IEEE (2009)

    Google Scholar 

  22. Shen, H., et al.: Missing information reconstruction of remote sensing data: a technical review. IEEE Geosci. Remote Sens. Mag. 3(3), 61–85 (2015)

    Article  Google Scholar 

  23. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv preprint arXiv:1409.1556

  24. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-ResNet and the impact of residual connections on learning. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

    Google Scholar 

  25. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  26. Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003. vol. 2, pp. 1398–1402. IEEE (2003)

    Google Scholar 

  27. Wu, P., Yin, Z., Yang, H., Wu, Y., Ma, X.: Reconstructing geostationary satellite land surface temperature imagery based on a multiscale feature connected convolutional neural network. Remote Sens. 11(3), 300 (2019)

    Article  Google Scholar 

  28. Yang, X., Zhao, Y., Vatsavai, R.R.: Deep residual network with multi-image attention for imputing under clouds in satellite imagery. In: 26th International Conference on Pattern Recognition, ICPR 2022, Montreal, QC, Canada, August 21-25, 2022, pp. 643–649. IEEE (2022). https://doi.org/10.1109/ICPR56361.2022.9956166

  29. Yang, X., Zhao, Y., Vatsavai, R.R.: Harmonization-guided deep residual network for imputing under clouds with multi-sensor satellite imagery. In: Proceedings of the 18th International Symposium on Spatial and Temporal Data, pp. 151–160 (2023)

    Google Scholar 

  30. Yu, J., Fan, Y., Huang, T.: Wide activation for efficient image and video super-resolution. In: 30th British Machine Vision Conference, BMVC 2019 (2020)

    Google Scholar 

  31. Yu, J., et al.: Wide activation for efficient and accurate image super-resolution (2018). arXiv preprint arXiv:1808.08718

  32. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5505–5514 (2018)

    Google Scholar 

  33. Yu, Y., Yao, H., Liu, Y.: Structural dynamics simulation using a novel physics-guided machine learning method. Eng. Appl. Artif. Intell. 96, 103947 (2020)

    Article  Google Scholar 

  34. Zhang, Q., Yuan, Q., Zeng, C., Li, X., Wei, Y.: Missing data reconstruction in remote sensing image with a unified spatial-temporal-spectral deep convolutional neural network. IEEE Trans. Geosci. Remote Sens. 56(8), 4274–4288 (2018)

    Article  Google Scholar 

  35. Zhao, Y., Yang, X., Vatsavai, R.R.: Multi-stream deep residual network for cloud imputation using multi-resolution remote sensing imagery. In: 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 97–104. IEEE (2022)

    Google Scholar 

  36. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

    Google Scholar 

Download references

Acknowledgements

This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract #2021-21040700001. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. We would like to thank Dr. Benjamin Raskob at ARA for all the support and feedback on this project. We would like to thank Dr. Yifan Zho and other STAC Lab members at NCSU for various suggestions.

Author information

Authors and Affiliations

  1. North Carolina State University, Raleigh, NC, 27695, USA

    Xian Yang & Ranga Raju Vatsavai

Authors
  1. Xian Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ranga Raju Vatsavai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ranga Raju Vatsavai .

Editor information

Editors and Affiliations

  1. LTCI, Télécom Paris, Palaiseau Cedex, France

    Albert Bifet

  2. Faculty of Informatics, Vytautas Magnus University, Akademija, Lithuania

    Tomas Krilavičius

  3. Stockholm University, Kista, Sweden

    Ioanna Miliou

  4. School of Information Technology, Halmstad University, Halmstad, Sweden

    Slawomir Nowaczyk

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 2765 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yang, X., Vatsavai, R.R. (2024). Multi-spectral Gradient Residual Network for Haze Removal in Multi-sensor Remote Sensing Imagery. In: Bifet, A., Krilavičius, T., Miliou, I., Nowaczyk, S. (eds) Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track. ECML PKDD 2024. Lecture Notes in Computer Science(), vol 14950. Springer, Cham. https://doi.org/10.1007/978-3-031-70381-2_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-70381-2_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-70380-5

  • Online ISBN: 978-3-031-70381-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships