Abstract
Pan-sharpening of remote sensing images is an effective method to get high spatial resolution multi-spectral (HRMS) images by fusing low spatial resolution multi-spectral (LRMS) images and high spatial resolution panchromatic (PAN) images. Recently, many remote sensing images pan-sharpening methods based on convolutional neural networks (CNN) have been proposed and achieved excellent performance. However, two drawbacks still exist. On the one hand, since there are no ideal HRMS images as targets for learning, most existing methods require an extra effort to produce the simulated data for training. On the other hand, these methods ignore the local features of the original images. To address these issues, we propose an unsupervised multi-scale generative adversarial network method, which can train directly on the full-resolution images without down-sampling. Firstly, a multi-scale dense generator network is proposed to extract features from the original images to generate HRMS images. Secondly, two discriminators are used to protect the spectral information of LRMS images and spatial information of PAN images, respectively. Finally, to improve the quality of the fused image and implement training under the unsupervised setting, a new loss function is proposed. Experimental results based on QuickBird and GaoFen-2 data sets demonstrate that the proposed method can obtain much better fusion results for the full-resolution images.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Tian, X., Chen, Y., Yang, C., et al.: Variational pansharpening by exploiting cartoon-texture similarities. IEEE Trans. Geosci. Remote Sens. 60, 1–6 (2021)
Luo, S., Zhou, S., Feng, Y., Xie, J.: Pansharpening via unsupervised convolutional neural networks. IEEE J. Select. Top. Appl. Earth Observ. Remote Sens. 13, 4295–4310 (2020)
Liu, X., Wang, Y., Liu, Q.: Remote sensing image fusion based on two-stream fusion network. In: Schoeffmann, K., et al. (eds.) MMM 2018. LNCS, vol. 10704, pp. 428–439. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73603-7_35
Garzelli, A., Nencini, F., Capobianco, L.: Optimal MMSE PanSharpening of very high resolution multispectral images. IEEE Trans. Geosci. Remote Sens. 46(1), 228–236 (2008)
Shahdoosti, H.R., Ghassemian, H.: Combining the spectral PCA and spatial PCA fusion methods by an optimal filter. Inf. Fusion. 27, 150–160 (2016)
Vivone, G., et al.: A critical comparison among pansharpening algorithms. IEEE Trans. Geosci. Remote Sens. 53(5), 2565–2586 (2014)
Jinju, J., Santhi, N., Ramar, K., Bama, B.S.: Spatial frequency discrete wavelet transform image fusion technique for remote sensing applications. Eng. Sci. Technol. Int. J. 22(3), 715–726 (2019)
Aiazzi, B., Alparone, L., Barducci, A., Baronti, S., Pippi, I.: Multispectral fusion of multisensor image data by the generalized Laplacian pyramid. IEEE Int. Geosci. Remote Sens. Symp. 2, 1183–1185 (1999)
Ballester, C., Caselles, V., Igual, L., Verdera, J., Rougé, B.: A variational model for P+ XS image fusion. Int. J. Comput. Vision 69(1), 43–58 (2006)
Yu, X., Gao, G., Xu, J., Wang, G.: Remote sensing image fusion based on sparse representation. In: IEEE Geoscience and Remote Sensing Symposium, pp. 2858–2861 (2014)
Masi, G., Cozzolino, D., Verdoliva, L., Scarpa, G.: Pansharpening by convolutional neural networks. Remote Sens. 8(7), 594 (2016)
Yang, J., Fu, X., Hu, Y., Huang, Y., Ding, X., Paisley, J.: PanNet: a deep network architecture for pan-sharpening. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5449–5457 (2017)
Wei, J., Xu, Y., Cai, W., Wu, Z., et al.: A two-stream multiscale deep learning architecture for pan-sharpening. IEEE J. Select. Top. Appl. Earth Observ. Remote Sens. 13, 5455–5465 (2020)
Ma, J., Yu, W., Chen, C., Liang, P., Guo, X., Jiang, J.: Pan-GAN: an unsupervised pan-sharpening method for remote sensing image fusion. Inf. Fusion 62, 110–120 (2020)
Peng, J., Liu, L., Wang, J., Zhang, E., Zhu, X., et al.: PSMD-Net: a novel Pan-Sharpening method based on a multiscale dense network. IEEE Trans. Geosci. Remote Sens. 59(6), 4957–4971 (2021)
Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A., Nencini, F., Selva, M.: Multispectral and panchromatic data fusion assessment without reference. Photogramm. Eng. Remote. Sens. 74(2), 193–200 (2008)
Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2794–802. (2017)
Goodfellow, I.J., et al.: Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014)
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International Conference on Machine Learning, pp. 214–223 (2017)
Radford, A., Metz, L., Chintala, S.: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv preprint. arXiv:1511.6434 (2020)
Chen, J., Pan, Y., Chen, Y.: Remote sensing image fusion based on Bayesian GAN. arXiv:2009.09465 (2020)
Yuhas, R.H., Goetz, A.F., Boardman, J.W.: Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm. In: Proceedings of the Summaries 3rd Annual JPL Airborne Geoscience Workshop, vol. 1, pp. 147–149 (1992)
Dong, C., Loy, C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Aiazzi, B., Alparone, L., Baronti, S., Garzelli, A., Selva, M.: MTF-tailored multiscale fusion of high-resolution MS and Pan imagery. Photogramm. Eng. Remote. Sens. 72(5), 591–596 (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, Y., Xie, Y., Wu, Y., Liang, K., Qiao, J. (2022). An Unsupervised Multi-scale Generative Adversarial Network for Remote Sensing Image Pan-Sharpening. In: Þór Jónsson, B., et al. MultiMedia Modeling. MMM 2022. Lecture Notes in Computer Science, vol 13142. Springer, Cham. https://doi.org/10.1007/978-3-030-98355-0_30
Download citation
DOI: https://doi.org/10.1007/978-3-030-98355-0_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-98354-3
Online ISBN: 978-3-030-98355-0
eBook Packages: Computer ScienceComputer Science (R0)