Xian Yang
Ranga Raju Vatsavai
ABSTRACT
Multi-sensor spatiotemporal satellite images have become crucial for monitoring the geophysical characteristics of the Earth’s environment. However, clouds often obstruct the view from the optical sensors mounted on satellites and therefore degrade the quality of spectral, spatial, and temporal information. Though cloud imputation with the rise of deep learning research has provided novel ways to reconstruct the cloud-contaminated regions, many learning-based methods still lack the capability of harmonizing the differences between similar spectral bands across multiple sensors. To cope with the inter-sensor inconsistency of overlapping bands in different optical sensors, we propose a novel harmonization-guided residual network to impute the areas under clouds. We present a knowledge-guided harmonization model that maps the reflectance response from one satellite collection to another based on the spectral distribution of the cloud-free pixels. The harmonized cloud-free image is subsequently exploited in the intermediate layers as an additional input, paired with a custom loss function that considers image reconstruction quality and inter-sensor consistency jointly during training. To demonstrate the performance of our model, we conducted extensive experiments on a multi-sensor remote sensing imagery benchmark dataset consisting of widely used Landsat-8 and Sentinel-2 images. Compared to the state-of-the-art methods, results show at least a 22.35% improvement in MSE.
- Robert Chastain, Ian Housman, Joshua Goldstein, Mark Finco, and Karis Tenneson. 2019. Empirical cross sensor comparison of Sentinel-2A and 2B MSI, Landsat-8 OLI, and Landsat-7 ETM+ top of atmosphere spectral characteristics over the conterminous United States. Remote sensing of environment 221 (2019), 274–285.Google Scholar
- Rémi Cresson, Dino Ienco, Raffaele Gaetano, Kenji Ose, and D Ho Tong Minh. 2019. Optical image gap filling using deep convolutional autoencoder from optical and radar images. In IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 218–221.Google ScholarCross Ref
- Rémi Cresson, Nicolas Narçon, Raffaele Gaetano, Aurore Dupuis, Yannick Tanguy, Stéphane May, and Benjamin Commandre. 2022. Comparison of convolutional neural networks for cloudy optical images reconstruction from single or multitemporal joint SAR and optical images. arXiv preprint arXiv:2204.00424 (2022).Google Scholar
- Patrick Ebel, Michael Schmitt, and Xiao Xiang Zhu. 2020. Cloud Removal in Unpaired Sentinel-2 Imagery Using Cycle-Consistent GAN and SAR-Optical Data Fusion. In IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 2065–2068.Google ScholarCross Ref
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.Google ScholarCross Ref
- Collin H Homer, Joyce A Fry, Christopher A Barnes, 2012. The national land cover database. US geological survey fact sheet 3020, 4 (2012), 1–4.Google Scholar
- Xiaowei Jia, Jared Willard, Anuj Karpatne, Jordan S Read, Jacob A Zwart, Michael Steinbach, and Vipin Kumar. 2021. Physics-guided machine learning for scientific discovery: An application in simulating lake temperature profiles. ACM/IMS Transactions on Data Science 2, 3 (2021), 1–26.Google ScholarDigital Library
- Song-Hee Kang, Youngjin Choi, and Jae Young Choi. 2021. Restoration of Missing Patterns on Satellite Infrared Sea Surface Temperature Images Due to Cloud Coverage Using Deep Generative Inpainting Network. Journal of Marine Science and Engineering 9, 3 (2021), 310.Google ScholarCross Ref
- Michael D. King, Steven Platnick, W. Paul Menzel, Steven A. Ackerman, and Paul A. Hubanks. 2013. Spatial and Temporal Distribution of Clouds Observed by MODIS Onboard the Terra and Aqua Satellites. IEEE Transactions on Geoscience and Remote Sensing 51, 7 (2013), 3826–3852.Google ScholarCross Ref
- Fred A Kruse, AB Lefkoff, JW Boardman, KB Heidebrecht, AT Shapiro, PJ Barloon, and AFH Goetz. 1993. The spectral image processing system (SIPS)—interactive visualization and analysis of imaging spectrometer data. Remote sensing of environment 44, 2-3 (1993), 145–163.Google Scholar
- Charis Lanaras, José Bioucas-Dias, Silvano Galliani, Emmanuel Baltsavias, and Konrad Schindler. 2018. Super-resolution of Sentinel-2 images: Learning a globally applicable deep neural network. ISPRS Journal of Photogrammetry and Remote Sensing 146 (2018), 305–319.Google ScholarCross Ref
- Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 136–144.Google ScholarCross Ref
- Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. 2018. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV). 85–100.Google ScholarDigital Library
- Xiaofeng Liu, Chaehwa Yoo, Fangxu Xing, Hyejin Oh, Georges El Fakhri, Je-Won Kang, Jonghye Woo, 2022. Deep unsupervised domain adaptation: A review of recent advances and perspectives. APSIPA Transactions on Signal and Information Processing 11, 1 (2022).Google ScholarCross Ref
- Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3431–3440.Google ScholarCross Ref
- Andrea Meraner, Patrick Ebel, Xiao Xiang Zhu, and Michael Schmitt. 2020. Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion. ISPRS Journal of Photogrammetry and Remote Sensing 166 (2020), 333–346.Google ScholarCross Ref
- Suraj Pawar, Omer San, Burak Aksoylu, Adil Rasheed, and Trond Kvamsdal. 2021. Physics guided machine learning using simplified theories. Physics of Fluids 33, 1 (2021), 011701.Google ScholarCross Ref
- Christian Requena-Mesa, Vitus Benson, Markus Reichstein, Jakob Runge, and Joachim Denzler. 2021. EarthNet2021: A large-scale dataset and challenge for Earth surface forecasting as a guided video prediction task.. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1132–1142.Google ScholarCross Ref
- Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234–241.Google ScholarCross Ref
- Tim GJ Rudner, Marc Rußwurm, Jakub Fil, Ramona Pelich, Benjamin Bischke, Veronika Kopačková, and Piotr Biliński. 2019. Multi3net: segmenting flooded buildings via fusion of multiresolution, multisensor, and multitemporal satellite imagery. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 702–709.Google ScholarDigital Library
- Daniel Scheffler, David Frantz, and Karl Segl. 2020. Spectral harmonization and red edge prediction of Landsat-8 to Sentinel-2 using land cover optimized multivariate regressors. Remote Sensing of Environment 241 (2020), 111723.Google ScholarCross Ref
- Rong Shang and Zhe Zhu. 2019. Harmonizing Landsat 8 and Sentinel-2: A time-series-based reflectance adjustment approach. Remote Sensing of Environment 235 (2019), 111439.Google ScholarCross Ref
- Huanfeng Shen, Xinghua Li, Qing Cheng, Chao Zeng, Gang Yang, Huifang Li, and Liangpei Zhang. 2015. Missing information reconstruction of remote sensing data: A technical review. IEEE Geoscience and Remote Sensing Magazine 3, 3 (2015), 61–85.Google ScholarCross Ref
- Andy Stock, Ajit Subramaniam, Gert L Van Dijken, Lisa M Wedding, Kevin R Arrigo, Matthew M Mills, Mary A Cameron, and Fiorenza Micheli. 2020. Comparison of cloud-filling algorithms for marine satellite data. Remote Sensing 12, 20 (2020), 3313.Google ScholarCross Ref
- Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. 2017. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence.Google ScholarDigital Library
- Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600–612.Google ScholarDigital Library
- Penghai Wu, Zhixiang Yin, Hui Yang, Yanlan Wu, and Xiaoshuang Ma. 2019. Reconstructing geostationary satellite land surface temperature imagery based on a multiscale feature connected convolutional neural network. Remote Sensing 11, 3 (2019), 300.Google ScholarCross Ref
- Michael A Wulder, Txomin Hermosilla, Joanne C White, Geordie Hobart, and Jeffrey G Masek. 2021. Augmenting Landsat time series with Harmonized Landsat Sentinel-2 data products: Assessment of spectral correspondence. Science of Remote Sensing 4 (2021), 100031.Google ScholarCross Ref
- Xian Yang, Yifan Zhao, and Ranga Raju Vatsavai. 2022. Deep Residual Network with Multi-Image Attention for Imputing Under Clouds in Satellite Imagery. In 2022 27th International Conference on Pattern Recognition (ICPR). IEEE.Google Scholar
- Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2018. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5505–5514.Google ScholarCross Ref
- Yang Yu, Houpu Yao, and Yongming Liu. 2020. Structural dynamics simulation using a novel physics-guided machine learning method. Engineering Applications of Artificial Intelligence 96 (2020), 103947.Google ScholarCross Ref
- Qiang Zhang, Qiangqiang Yuan, Chao Zeng, Xinghua Li, and Yancong Wei. 2018. Missing data reconstruction in remote sensing image with a unified spatial–temporal–spectral deep convolutional neural network. IEEE Transactions on Geoscience and Remote Sensing 56, 8 (2018), 4274–4288.Google ScholarCross Ref
- Yifan Zhao, Xian Yang, and Ranga Raju Vatsavai. 2022. Multi-stream Deep Residual Network for Cloud Imputation Using Multi-resolution Remote Sensing Imagery. In 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, 97–104.Google Scholar
- Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision. 2223–2232.Google ScholarCross Ref
Index Terms
- Harmonization-guided deep residual network for imputing under clouds with multi-sensor satellite imagery
Recommendations
Normalization method for multi-sensor high spatial and temporal resolution satellite imagery with radiometric inconsistencies
Highlights- Method to normalize CubeSat time series preserving spatial and temporal resolution.
AbstractDistributed systems of small satellites are generating a new type of remote sensing data: multi-spectral data with high spatial and temporal resolution and near-daily global coverage. This data is proving to be valuable for monitoring ...
Seasonally-managed wetland footprint delineation using Landsat ETM+ satellite imagery
One major challenge in water resource management is the estimation of evapotranspiration losses from seasonally managed wetlands. Quantifying these losses is complicated by the dynamic nature of the wetlands' areal footprint during the periods of flood-...
Multimodal Deep Learning Based Crop Classification Using Multispectral and Multitemporal Satellite Imagery
KDD '20: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data MiningThe Food and Agriculture Organization (FAO) of the United Nations predicts that in order to meet the needs of the expected 3 billion population growth by 2050, food production has to increase by 60%. Therefore, monitoring and mapping crops accurately is ...
Comments