Skip to main content
SpringerLink
Log in

A generative adversarial network with structural enhancement and spectral supplement for pan-sharpening

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Pan-sharpening aims to obtain high-resolution multi-spectral images by fusing panchromatic images and low-resolution multi-spectral images though reasonable rules. This paper proposed a novel generative adversarial network for pan-sharpening, which utilizes the supplemented spectral information from low-resolution multi-spectral images and the enhanced structural information from panchromatic images to generate high-resolution multi-spectral images. Firstly, the forward differential operator is used to extract the spatial structural information of the panchromatic image both in the horizontal and vertical directions. Secondly, an architecture of generative adversarial network is designed. The enhanced structural information generated by the accumulation of the structural information of the two directions is added to the image fusion process in generator and the discriminating process in discriminator, and a new optimization objective is designed accordingly. What is more, the low-resolution multi-spectral image is added to the convolution process in the generator as a supplement to the spectral information. Finally, in order to obtain better image generation effect, a special objective function of the generator is designed, which adds a unique relationship to reduce the loss of spatial structural information and spectral information of fused images. Experiments on QuickBird and WorldView-3 satellites datasets show that the proposed method can generate high quality fused images and is better than most advanced methods in both objective indicators and intuitive observations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Thomas C, Ranchin T, Wald L, Chanussot J (2008) Synthesis of multispectral images to high spatial resolution: a critical review of fusion methods based on remote sensing physics. IEEE Trans Geosci Remote Sens 46(5):1301–1312

    Article  Google Scholar 

  2. Dogra A, Goyal B, Agrawal S (2017) From multi-scale decomposition to non-multi-scale decomposition methods: a comprehensive survey of image fusion techniques and its applications. IEEE Access 5:16040–16067

    Article  Google Scholar 

  3. Dong C, Loy CC, He K, Tang X (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  4. Sreehari S, Venkatakrishnan SV, Bouman Katherine L, Simmons Jeffrey P, Drummy Lawrence F, Bouman Charles A (2017) Multi-resolution data fusion for super-resolution electron microscopy. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 88–96

  5. Ma J, Wei Y, Liang P, Li C, Jiang J (2019) Fusiongan: A generative adversarial network for infrared and visible image fusion. Inform Fusion 48:11–26

    Article  Google Scholar 

  6. Zhang Q, Wang Y, Liu Q, Liu X, Wang W (2016) Cnn based suburban building detection using monocular high resolution google earth images. In: 2016 IEEE international geoscience and remote sensing symposium (IGARSS). IEEE, pp 661–664

  7. Zhiliang W, Huang Y, Zhang K (2018) Remote sensing image fusion method based on pca and curvelet transform. J Indian Soc Remote Sens 46(5):687–695

    Article  Google Scholar 

  8. Rahmani S, Strait M, Merkurjev D, Moeller M, Wittman T (2010) An adaptive ihs pan-sharpening method. IEEE Geosci Remote Sens Lett 7(4):746–750

    Article  Google Scholar 

  9. Li X, Zhang Y, Gao Y, Yue S (2016) Using guided filtering to improve gram-schmidt based pansharpening method for geoeye-1 satellite images. In: 4th International conference on information systems and computing technology. Atlantis Press

  10. Shensa MJ (1992) The discrete wavelet transform: wedding the a trous and mallat algorithms. IEEE Trans Signal Process 40(10):2464–2482

    Article  Google Scholar 

  11. Otazu X, González-Audícana M, Fors O, Núñez J (2005) Introduction of sensor spectral response into image fusion methods. application to wavelet-based methods. IEEE Trans Geosci Remote Sens 43(10):2376–2385

    Article  Google Scholar 

  12. Chen C, Li Y, Liu W, Huang J (2014) Image fusion with local spectral consistency and dynamic gradient sparsity. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2760–2765

  13. Zeng D, Yuwen H, Huang Y, Zhiliang X, Ding X (2016) Pan-sharpening with structural consistency and \(\ell _{1/2}\) gradient prior. Remote sens lett 7(12):1170–1179

    Article  Google Scholar 

  14. Ding X, Jiang Y, Huang Y, Paisley J (2014) Pan-sharpening with a bayesian nonparametric dictionary learning model. In: Artificial intelligence and statistics, pp 176–184

  15. Liu Y, Chen X, Wang Z, Wang ZJ, Ward RK, Wang X (2018) Deep learning for pixel-level image fusion: Recent advances and future prospects. Inform Fusion 42:158–173

    Article  Google Scholar 

  16. Liao W, Huang X, Van Coillie F, Gautama S, Pižurica A, Philips W, Liu H, Zhu T, Shimoni M, Moser G et al (2015) Processing of multiresolution thermal hyperspectral and digital color data: Outcome of the 2014 ieee grss data fusion contest. IEEE J Select Top Appl Earth Observa Remote Sens 8(6):2984–2996

    Article  Google Scholar 

  17. Zhang L, Zhang J (2017) A new saliency-driven fusion method based on complex wavelet transform for remote sensing images. IEEE Geosci Remote Sens Lett 14(12):2433–2437

    Article  Google Scholar 

  18. Alparone L, Wald L, Chanussot J, Thomas C, Gamba P, Bruce LM (2007) Comparison of pansharpening algorithms: outcome of the 2006 grs-s data-fusion contest. IEEE Trans Geosci Remote Sens 45(10):3012–3021

    Article  Google Scholar 

  19. Garzelli A, Nencini F, Capobianco L (2007) Optimal mmse pan sharpening of very high resolution multispectral images. IEEE Trans Geosci Remote Sens 46(1):228–236

    Article  Google Scholar 

  20. Masi G, Cozzolino D, Verdoliva L, Scarpa G (2016) Pansharpening by convolutional neural networks. Remote Sens 8(7):594

    Article  Google Scholar 

  21. Yang J, Fu X, Hu Y, Huang Y, Ding X, Paisley J (2017) Pannet: a deep network architecture for pan-sharpening. In: Proceedings of the IEEE international conference on computer vision, pp 5449–5457

  22. Liu X, Wang Y, Liu Q (2018) Psgan: a generative adversarial network for remote sensing image pan-sharpening. In: 2018 25th IEEE international conference on image processing (ICIP). IEEE, pp 873–877

  23. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680

  24. Isola P, Zhu J-Y, Zhou T, Efros Alexei A (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  25. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z et al (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681–4690

  26. Zhang H, Sun Y, Liu L, Wang X, Li L, Liu W (2018) Clothingout: a category-supervised gan model for clothing segmentation and retrieval. In: Neural computing and applications, pp 1–12

  27. Linlin L, Haijun Z, Yuzhu J, QM Jonathan W (2019) Toward ai fashion design: An attribute-gan model for clothing match. Neurocomputing 341:156–167

  28. Boyd S, Parikh N, Chu E, Peleato B, Eckstein J et al (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends® Mach Learn 3(1):1–122

  29. Aiazzi B, Alparone L, Baronti S, Garzelli A, Selva M (2015) Mtf-tailored multiscale fusion of high-resolution ms and pan imagery. Photogram Eng Remote Sens 72(5):591–596

    Article  Google Scholar 

  30. Maas Andrew L, Hannun Awni Y, Ng Andrew Y (2013) Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of ICML, vol 30, p 3

  31. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, New York, pp 234–241

  32. Kingma Diederik P, Ba J (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980

  33. Lucien W (2002) Data fusion: definitions and architectures: fusion of images of different spatial resolutions. Presses des MINES

  34. Ranchin T, Wald L (2000) Fusion of high spatial and spectral resolution images: The arsis concept and its implementation. Photogram Eng Remote Sens 66(1):49–61

    Google Scholar 

  35. Garzelli A, Nencini F (2009) Hypercomplex quality assessment of multi/hyperspectral images. IEEE Geosci Remote Sens Lett 6(4):662–665

    Article  Google Scholar 

  36. Vivone G, Alparone L, Chanussot J, Dalla Mura M, Garzelli A, Licciardi GA, Restaino R, Wald L (2014) A critical comparison among pansharpening algorithms. IEEE Trans Geosci Remote Sens 53(5):2565–2586

    Article  Google Scholar 

  37. Zhou J, Civco DL, Silander JA (1998) A wavelet transform method to merge landsat tm and spot panchromatic data. Int J Remote Sens 19(4):743–757

    Article  Google Scholar 

  38. Alparone L, Aiazzi B, Baronti S, Garzelli A, Nencini F, Selva M (2008) Multispectral and panchromatic data fusion assessment without reference. Photogramm Eng Remote Sens 74(2):193–200

    Article  Google Scholar 

Download references

Acknowledgements

This work is partially supported by the National Natural Science Foundation of China [Nos. 61472055, 61802148 and U1401252], Chongqing Outstanding Youth Fund [No. cstc2014jcyjjq40001], Doctoral Innovative Talents Project of Chongqing University of Posts and Telecommunications [No. BYJS2017009].

Author information

Authors and Affiliations

  1. Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China

    Liping Zhang & Weisheng Li

  2. College of Computer, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China

    Liping Zhang, Ce Zhang & Dajiang Lei

  3. College of Software Engineering, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China

    Liping Zhang

Authors
  1. Liping Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weisheng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ce Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dajiang Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Liping Zhang or Weisheng Li.

Ethics declarations

Conflict of interest

We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled, “A generative adversarial network with structural enhancement and spectral supplement for pan-sharpening”.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (docx 71 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, L., Li, W., Zhang, C. et al. A generative adversarial network with structural enhancement and spectral supplement for pan-sharpening. Neural Comput & Applic 32, 18347–18359 (2020). https://doi.org/10.1007/s00521-020-04973-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-04973-w

Keywords

Search

Navigation