Skip to main content
SpringerLink
Log in

FDPPGAN: remote sensing image fusion based on deep perceptual patchGAN

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Remote sensing satellites can simultaneously capture high spatial resolution panchromatic (PAN) images and low spatial resolution multispectral (MS) images. Pan-sharpening in the fusion of remote sensing images aims to generate high-resolution MS images by integrating the spatial information of PAN images and the spectral characteristics of MS images. In this study, a novel deep perceptual patch generative adversarial network (FDPPGAN) was proposed to solve the pan-sharpening problem. First, a perception generator was constructed, it included, a matching module, which can process as input images of different resolutions, a fusion module, a reconstruction module based on the residual structure, and a module for the extracting perceptual features. Second, patch discriminator was utilized to convert the dichotomy of the sample into that multiple partial images of the same size to ensure that the generated results can retain more detailed features. Finally, the loss function of FDPPGAN comprised perceptual feature loss, content loss, generator loss, and discriminator loss. Experiments on the QuickBird and WorldView datasets demonstrated that the proposed algorithm is superior to state-of-the-art algorithms in subjective and objective indexes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://www.digitalglobe.com/.

References

  1. Zhang P, Gong M, Su L, Liu J, Li Z (2016) Change detection based on deep feature representation and mapping transformation for multi-spatial resolution remote sensing images. ISPRS J Photogramm Remote Sens 116:24–41

    Article  Google Scholar 

  2. Sousa D, Davis FW (2020) Scalable mapping and monitoring of Mediterranean-climate oak landscapes with temporal mixture models. Remote Sens Environ 247:111937

    Article  Google Scholar 

  3. Imani M, Ghassemian H (2020) An overview on spectral and spatial information fusion for hyperspectral image classification: current trends and challenges. Inf Fusion 59:59–83

    Article  Google Scholar 

  4. Chavez P, Sides SC, Anderson JA (1991) Comparison of three different methods to merge multiresolution and multispectral data—Landsat TM and SPOT panchromatic. Photogramm Eng Remote Sens 57(3):295–303

    Google Scholar 

  5. Tu TM, Su SC, Shyu HC et al (2001) A new look at IHS-like image fusion methods. Inf Fusion 2(3):177–186

    Article  Google Scholar 

  6. Tu TM, Lee YC, Chang CP et al (2005) Adjustable intensity–hue–saturation and Brovey transform fusion technique for IKONOS/QuickBird imagery. Opt Eng 44(11):116201

    Article  Google Scholar 

  7. Tu T-M, Huang PS, Hung C-L, Chang C-P (2004) A fast intensity-hue-saturation fusion technique with spectral adjustment for IKONOS imagery. IEEE Geosci Remote Sens Lett 1(4):309–312

    Article  Google Scholar 

  8. Ranchin T, Wald L (2000) Fusion of high spatial and spectral resolution images: the arsis concept and its implementation. Photogramm Eng Remote Sens 66(1):49–61

    Google Scholar 

  9. Starck J-L, Candes EJ, Donoho DL (2002) The curvelet transform for image denoising, IEEE Trans Image Process 11(6): 670–684.

  10. Da Cunha AL, Zhou J, Do MN (2006) The nonsubsampled Contourlet transform: theory, design, and applications. IEEE Trans Image Process 15(10):3089–3101

    Article  Google Scholar 

  11. Zheng S, Shi W, Liu J, Tian J (2008) Remote sensing image fusion using multiscale mapped LS-SVM. IEEE Trans Geosci Remote Sens 46(5):1313–1322

    Article  Google Scholar 

  12. Zhu XX, Bamler R (2013) A sparse image fusion algorithm with application to pan-sharpening. IEEE Trans Geosci Remote Sens 51(5):2827–2836

    Article  Google Scholar 

  13. Wang W, Jiao L, Yang S (2014) Fusion of multispectral and panchromatic images via sparse representation and local autoregressive model. Inf Fusion 20:73–87

    Article  Google Scholar 

  14. Moonon AU, Hu J, Li S (2015) Remote sensing image fusion method based on nonsubsampled shearlet transform and sparse representation. Sens Imag 16(1):23

    Article  Google Scholar 

  15. Masi G, Cozzolino D, Verdoliva L, Scarpa G (2016) Pansharpening by convolutional neural networks. Remote Sens 8(7):594

    Article  Google Scholar 

  16. Shao Z, Cai J (2018) Remote sensing image fusion with deep convolutional neural network. IEEE J Selected Topics Appl Earth Observ Remote Sens 11(5):1656–1669

    Article  Google Scholar 

  17. Rao Y, He L, Zhu J (2017) A residual convolutional neural network for pan-shaprening. In: IEEE 2017 International Workshop on Remote Sensing with Intelligent Processing, pp 1–4

  18. Liu X, Liu Q, Wang Y (2020) Remote sensing image fusion based on two-stream fusion network. Inf Fusion 55:1–15

    Article  Google Scholar 

  19. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv Neural Inf Process Syst 27:2672–2680

  20. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv:1511.06434v1

  21. Kaneko T, Hiramatsu K, Kashino K (2017) Generative attribute controller with conditional filtered generative adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp 7006–7015.

  22. Liu L, Zhang H, Xu X, Zhang Z, Yan S (2020) Collocating clothes with generative adversarial networks cosupervised by categories and attributes: a multidiscriminator framework. IEEE Trans Neural Netw Learn Syst 31(9):3540–3554

    Article  MathSciNet  Google Scholar 

  23. Ma J, Yu W, Liang P, Li C, Jiang J (2019) FusionGAN: a generative adversarial network for infrared and visible image fusion. Information Fusion 48:11–26

    Article  Google Scholar 

  24. Liu X, Wang Y, Liu Q (2018) PSGAN: a generative adversarial network for remote sensing image pan-sharpening. In: Proceedings of the IEEE International Conference on Image Processing, pp 873–877.

  25. Ma J et al. (2020) Pan-GAN: an unsupervised pan-sharpening method for remote sensing image fusion. Inf Fusion 62:110–120.

  26. Wald L, Ranchin T, Mangolini M (1997) Fusion of satellite images of different spatial resolution: assessing the quality of resulting images. Photogramm Eng Remote Sens 63:691–699

    Google Scholar 

  27. He K, Zhang X, Ren S, Sun, J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778.

  28. Johnson J, Alahi A, Li F-F (2016) Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision. Springer, Cham, pp 694–711

  29. Isola P, Zhu J-Y, Zhou T, Efroset AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1125–1134

  30. Alparone L, Wald L, Chanussot J, Thomas C, Gamba P, Bruce LM (2007) Comparison of pansharpening algorithms: outcome of the 2006 GRS-S data-fusion contest. IEEE Trans Geosci Remote Sens 45(10):3012–3021

    Article  Google Scholar 

  31. Zhou J, Civco DL, Silander JA (1998) A wavelet transform method to merge Landsat TM and SPOT panchromatic data. Int J Remote Sens 19(4):743–757

    Article  Google Scholar 

  32. Thomas C, Ranchin T, Wald L, Chanussot J (2008) Synthesis of multispectral images to high spatial resolution: a critical review of fusion methods based on remote sensing physics. IEEE Trans Geosci Remote Sens 46(5):1301–1312

    Article  Google Scholar 

  33. Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9(3):81–84

  34. Alparone L, Aiazzi B, Baronti S et al (2008) Multispectral and panchromatic data fusion assessment without reference. Photogramm Eng Remote Sens 74(2):193–200

    Article  Google Scholar 

  35. Rahmani S, Strait M, Merkurjev D, Moeller M, Wittman T (2010) An adaptive IHS pan-sharpening method. IEEE Geosci Remote Sens Lett 7(4):746–750

    Article  Google Scholar 

  36. Garzelli A, Nencini F, Capobianco L (2008) Optimal MMSE Pan sharpening of very high resolution multispectral images. IEEE Trans Geosci Remote Sens 46(1):228–236

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, 210000, China

    Yue Pan, Dechang Pi & Junfu Chen

  2. School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, 100000, China

    Han Meng

Authors
  1. Yue Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dechang Pi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junfu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Han Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dechang Pi.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest with any person(s) or organization(s).

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pan, Y., Pi, D., Chen, J. et al. FDPPGAN: remote sensing image fusion based on deep perceptual patchGAN. Neural Comput & Applic 33, 9589–9605 (2021). https://doi.org/10.1007/s00521-021-05724-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-05724-1

Keywords

Search

Navigation