Skip to main content
SpringerLink
Log in

Pyramid-attention based multi-scale feature fusion network for multispectral pan-sharpening

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Remote sensing images with high spatial resolution and high spectral resolution have important applications in human society. In general, due to the limitations faced by the optical sensors’, we are limited to obtain only low spatial resolution multispectral images (MS) and high spatial resolution panchromatic images (PAN). To address this limitation, this study proposes a pyramid-attention based multi-scale feature fusion network (PAMF-Net) that combines the pyramid attention mechanism and feature aggregation. Initially, the MS and PAN images are input to the network, and the PAN images pass through the input pyramid branch to generate a multi-level receiving domain. Then, the result is combined with the features of the MS image as the input of the encoder, and these composite features are input to the pyramid attention mechanism module to capture multi-scale corresponding features. Next, the result of the input pyramid branch is input to the feature aggregation module to seamlessly merge with the features of the pyramid attention mechanism. Finally, in the encoding stage, multiple levels of features are multiplexed as encoding secondary lines by skipping connections to obtain high-quality HRMS images. After quantitative and qualitative experiments, the results show that our method is superior to other advanced methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Aiazzi B, Alparone L, Baronti S, Garzelli A, Selva M (2006) Mtf-tailored multiscale fusion of high-resolution ms and pan imagery. Photogrammetric Engineering & Remote Sensing 72:591–596

    Article  Google Scholar 

  2. Al-Wassai FA, Kalyankar N, Al-Zuky AA (2011) The ihs transformations based image fusion. arXiv:1107.4396

  3. Ballester C, Caselles V, Igual L, Verdera J, Rougé B (2006) A variational model for p+ xs image fusion. Int J Comput Vis 69:43–58

    Article  Google Scholar 

  4. Cai W, Xu Y, Wu Z, Liu H, Qian L, Wei Z (2018) Pan-sharpening based on multilevel coupled deep network. In: IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, IEEE pp 7046–7049

  5. Eghbalian S, Ghassemian H (2018) Multi Spectral image fusion with deep convolutional network. In: 2018 9Th international symposium on telecommunications (IST), IEEE, pp 173–177

  6. Garzelli A, Nencini F, Capobianco L (2007) Optimal mmse pan sharpening of very high resolution multispectral images. IEEE Trans Geosci Remote Sens 46:228–236

    Article  Google Scholar 

  7. He K, Gkioxari G, Dollár P, Girshick R (2017) Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 2961–2969

  8. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  9. Huang W, Xiao L, Wei Z, Liu H, Tang S (2015) A new pan-sharpening method with deep neural networks. IEEE Geosci Remote Sens Lett 12:1037–1041

    Article  Google Scholar 

  10. Jiang C, Zhang H, Shen H, Zhang L (2011) A practical compressed sensing-based pan-sharpening method. IEEE Geosci Remote Sens Lett 9:629–633

    Article  Google Scholar 

  11. Jiang C, Zhang H, Shen H, Zhang L (2013) Two-step sparse coding for the pan-sharpening of remote sensing images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 7:1792–1805

    Article  Google Scholar 

  12. Kang X, Li S, Fang L, Benediktsson JA (2014) Pansharpening based on intrinsic image decomposition. Sensing and Imaging 15:94

    Article  Google Scholar 

  13. Kaplan NH, Erer I (2014) Bilateral filtering-based enhanced pansharpening of multispectral satellite images. IEEE Geosci Remote Sens Lett 11:1941–1945

    Article  Google Scholar 

  14. Kwarteng P, Chavez A (1989) Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis. Photogramm Eng Remote Sens 55:1

    Google Scholar 

  15. Laben CA, Brower BV (2000) Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. US Patent 6,011,875.

  16. Li N, Huang N, Xiao L (2017) Pan-sharpening via residual deep learning. In: 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), IEEE, pp 5133–5136

  17. Li S, Yang B (2010) A new pan-sharpening method using a compressed sensing technique. IEEE Trans Geosci Remote Sens 49:738–746

    Article  Google Scholar 

  18. Li S, Yin H, Fang L (2013) Remote sensing image fusion via sparse representations over learned dictionaries. IEEE Trans Geosci Remote Sens 51:4779–4789

    Article  Google Scholar 

  19. Li W, Wu G, Zhang F, Du Q (2016) Hyperspectral image classification using deep pixel-pair features. IEEE Trans Geosci Remote Sens 55:844–853

    Article  Google Scholar 

  20. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) Ssd: Single shot multibox detector. In: European conference on computer vision, Springer, pp 21–37

  21. Liu X, Wang Y, Liu Q (2018) Psgan: A Generative adversarial network for remote sensing image pan-sharpening. In: 2018 25Th IEEE international conference on image processing (ICIP), IEEE, pp 873–877

  22. Long M, Zeng Y (2019) Detecting iris liveness with batch normalized convolutional neural network. Computers, Materials & Continua 58:493–504

    Article  Google Scholar 

  23. Ma J, Yu W, Chen C, Liang P, Guo X, Jiang J (2020) Pan-gan: an unsupervised pan-sharpening method for remote sensing image fusion. Information Fusion 62:110–120

    Article  Google Scholar 

  24. Ma J, Zhou H, Zhao J, Gao Y, Jiang J, Tian J (2015) Robust feature matching for remote sensing image registration via locally linear transforming. IEEE Trans Geosci Remote Sens 53:6469–6481

    Article  Google Scholar 

  25. Ma Y, Zhang Y, Mei X, Dai X, Ma J (2019) Multifeature-based discriminative label consistent k-svd for hyperspectral image classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12:4995–5008

    Article  Google Scholar 

  26. Masi G, Cozzolino D, Verdoliva L, Scarpa G (2016a) Pansharpening by convolutional neural networks. Remote Sens 8:594

    Article  Google Scholar 

  27. Masi G, Cozzolino D, Verdoliva L, Scarpa G (2016b) Pansharpening by convolutional neural networks. Remote Sens 8:594

    Article  Google Scholar 

  28. Nason GP, Silverman BW (1995) The stationary wavelet transform and some statistical applications. In: Wavelets and statistics. Springer, pp 281–299

  29. PJ B (1983) The laplacian pyramid as a compact image code. IEEE Trans Commun 31:532–540

    Article  Google Scholar 

  30. Rahmani S, Strait M, Merkurjev D, Moeller M, Wittman T (2010) An adaptive ihs pan-sharpening method. IEEE Geosci Remote Sens Lett 7:746–750

    Article  Google Scholar 

  31. Shah VP, Younan NH, King RL (2008) An efficient pan-sharpening method via a combined adaptive pca approach and contourlets. IEEE Trans Geosci Remote Sens 46:1323– 1335

    Article  Google Scholar 

  32. Shao Z, Cai J, Fu P, Hu L, Liu T (2019a) Deep learning-based fusion of landsat-8 and sentinel-2 images for a harmonized surface reflectance product. Remote Sens Environ 235:111425

    Article  Google Scholar 

  33. Shao Z, Fu H, Li D, Altan O, Cheng T (2019b) Remote sensing monitoring of multi-scale watersheds impermeability for urban hydrological evaluation. Remote Sens Environ 232:111338

    Article  Google Scholar 

  34. Shao Z, Zhang L (2014) Sparse dimensionality reduction of hyperspectral image based on semi-supervised local fisher discriminant analysis. International Journal of Applied Earth Observation and Geoinformation 31:122–129

    Article  Google Scholar 

  35. Sun L, Ma C, Chen Y, Shim HJ, Wu Z, Jeon B (2019a) Adjacent superpixel-based multiscale spatial-spectral kernel for hyperspectral classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12:1905–1919

    Article  Google Scholar 

  36. Sun L, Ma C, Chen Y, Zheng Y, Shim HJ, Wu Z, Jeon B (2019b) Low rank component induced spatial-spectral kernel method for hyperspectral image classification. IEEE Transactions on Circuits and Systems for Video Technology

  37. Tu TM, Huang PS, Hung CL, Chang CP (2004) A fast intensity-hue-saturation fusion technique with spectral adjustment for ikonos imagery. IEEE Geosci Remote Sens lett 1:309–312

    Article  Google Scholar 

  38. Vrabel J (2000) Multispectral imagery advanced band sharpening study. Photogrammetric Engineering and Remote Sensing 66:73–80

    Google Scholar 

  39. Wald L, Ranchin T, Mangolini M (1997) Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing 63:691–699

    Google Scholar 

  40. Wang H, Peng J, Chen D, Jiang G, Zhao T, Fu X (2020a) Attribute-guided feature learning network for vehicle reidentification. IEEE MultiMedia 27:112–121

    Article  Google Scholar 

  41. Wang H, Peng J, Jiang G, Xu F, Fu X (2021) Discriminative feature and dictionary learning with part-aware model for vehicle re-identification. Neurocomputing 438:55–62

    Article  Google Scholar 

  42. Wang H, Peng J, Zhao Y, Fu X (2020b) Multi-path deep cnns for fine-grained car recognition. IEEE Trans Veh Technol 69:10484–10493

    Article  Google Scholar 

  43. Wang H, Wang Y, Zhang Z, Fu X, Zhuo L, Xu M, Wang M (2020c) Kernelized multiview subspace analysis by self-weighted learning. IEEE Transactions on Multimedia

  44. Wei Y, Yuan Q, Shen H, Zhang L (2017) Boosting the accuracy of multispectral image pansharpening by learning a deep residual network. IEEE Geosci Remote Sens Lett 14:1795–1799

    Article  Google Scholar 

  45. Xu Y, Wu Z, Chanussot J, Wei Z (2019) Nonlocal patch tensor sparse representation for hyperspectral image super-resolution. IEEE Trans Image Process 28:3034–3047

    Article  MathSciNet  Google Scholar 

  46. Yang J, Fu X, Hu Y, Huang Y, Ding X, Paisley J (2017a) Pannet: A deep network architecture for pan-sharpening. In: Proceedings of the IEEE International Conference on Computer Vision, pp 5449–5457

  47. Yang Y, Wan W, Huang S, Lin P, Que Y (2017b) A novel pan-sharpening framework based on matting model and multiscale transform. Remote Sens 9:391

    Article  Google Scholar 

  48. Zeng D, Dai Y, Li F, Sherratt RS, Wang J (2018) Adversarial learning for distant supervised relation extraction. Computers, Materials & Continua 55:121–136

    Google Scholar 

  49. Zeng D, Dai Y, Li F, Wang J, Sangaiah AK (2019) Aspect based sentiment analysis by a linguistically regularized cnn with gated mechanism. J Intell Fuzzy Syst 36:3971–3980

    Article  Google Scholar 

  50. Zheng S, Shi W. z., Liu J, Tian J (2008) Remote sensing image fusion using multiscale mapped ls-svm. IEEE Trans Geosci Remote Sens 46:1313–1322

    Article  Google Scholar 

  51. Zhou S, Ke M, Luo P (2019) Multi-camera transfer gan for person re-identification. J Vis Commun Image Represent 59:393–400

    Article  Google Scholar 

  52. Zhu X, Bao W (2018) Comparison of remote sensing image fusion strategies adopted in hsv and ihs. J Indian Soc Remote Sens 46:377–385

    Article  Google Scholar 

  53. Zhu XX, Bamler R (2012) A sparse image fusion algorithm with application to pan-sharpening. IEEE Trans Geosci Remote Sens 51:2827–2836

    Article  Google Scholar 

Download references

Acknowledgments

The authors acknowledge the National Natural Science Foundation of China (Grant nos. 61772319, 61976125, 61873177 and 61773244), and Shandong Natural Science Foundation of China (Grant no. ZR2017MF049).

Author information

Author notes

  1. Yang Chi

    Present address: School of Information and Electronic Engineering, Shandong Technology and Business University, Yantai, 264005, China

  2. Jinjiang Li

    Present address: School of Computer Science and Technology, Shandong Technology and Business University, Yantai, 264005, China

Authors and Affiliations

  1. School of Computer Science and Technology, Shandong Technology and Business University, Yantai, 264005, China

    Hui Fan

Authors
  1. Yang Chi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinjiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hui Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinjiang Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chi, Y., Li, J. & Fan, H. Pyramid-attention based multi-scale feature fusion network for multispectral pan-sharpening. Appl Intell 52, 5353–5365 (2022). https://doi.org/10.1007/s10489-021-02732-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02732-5

Keywords

Search

Navigation