Skip to main content
Springer Nature Link
Log in

Dual-branch and triple-attention network for pan-sharpening

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Pan-sharpening is a technique used to generate high-resolution multi-spectral (HRMS) images by merging high-resolution panchromatic (PAN) images with low-resolution multi-spectral (LRMS) images. Many existing methods face challenges in effectively balancing the trade-off between spectral and spatial information, leading to spectral and spatial structural distortion. In order to effectively tackle these issues, we propose a dual-branch and triple attention (DBTA) network. The proposed DBTA network consists of two essential modules: the Channel-spatial Attention (CSA) module and the Spectral Attention (SPA) module. The CSA module effectively captures the spatial structural information of the images by jointly using spatial and channel attention units. Meanwhile, the SPA module improves the expressive capacity of spectral information by dynamically adjusting channel weights. These two modules work in synergy to achieve comprehensive extraction and fusion of spectral and spatial information, thus resulting in more accurate and clearer reconstructed images. Extensive experiments have been conducted on various satellite datasets to evaluate the performance of the proposed DBTA method outperforms the state-of-the-art competitors in both qualitative and quantitative evaluations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data Availability

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

References

  1. Alparone L, Aiazzi B, Baronti S, Garzelli A, Nencini F, Selva M (2008) Multispectral and panchromatic data fusion assessment without reference. Photogramm Eng Remote Sensing 74(2):193–200

    Google Scholar 

  2. Alparone L, Wald L, Chanussot J, Thomas C, Gamba P, Bruce LM (2007) Comparison of pansharpening algorithms: Outcome of the 2006 grs-s data-fusion contest. IEEE Trans Geosci Remote Sens 45(10):3012–3021. https://doi.org/10.1109/TGRS.2007.904923

    Article  Google Scholar 

  3. Bullock EL, Woodcock CE, Olofsson P (2020) Monitoring tropical forest degradation using spectral unmixing and landsat time series analysis. Remote Sens Environ 238:110968

    Google Scholar 

  4. Cao X, Chen Y, Cao W (2022) Proximal pannet: A model-based deep network for pansharpening. In: Proceedings of the AAAI conference on artificial intelligence vol 36, pp 176–184

  5. Chen Y, Peng G, Zhu Z, Li S (2020) A novel deep learning method based on attention mechanism for bearing remaining useful life prediction. Appl Soft Comput 86:105919

    Google Scholar 

  6. Choromanski KM, Likhosherstov V, Dohan D, Song X, Gane A, Sarlos T, Hawkins P, Davis JQ, Mohiuddin A, Kaiser L, Belanger DB, Colwell LJ, Weller A (2021) Rethinking attention with performers. In: International conference on learning representations

  7. Ciotola M, Vitale S, Mazza A, Poggi G, Scarpa G (2022) Pansharpening by convolutional neural networks in the full resolution framework. IEEE Trans Geosci Remote Sens 60:1–17

    Google Scholar 

  8. Deng LJ, Vivone G, Jin C, Chanussot J (2020) Detail injection-based deep convolutional neural networks for pansharpening. IEEE Trans Geosci Remote Sens 59(8):6995–7010

    Google Scholar 

  9. Deng LJ, Vivone G, Paoletti ME, Scarpa G, He J, Zhang Y, Chanussot J, Plaza A (2022) Machine learning in pansharpening: A benchmark, from shallow to deep networks. IEEE Geosci Remote Sens Mag 10(3):279–315

    Google Scholar 

  10. Dong C, Loy CC, He K, Tang X (2015) Image super-resolution using deep convolutional networks. IEEE Geosci Remote Sens Mag 38(2):295–307

    Google Scholar 

  11. Fu X, Lin Z, Huang Y, Ding X (2019) A variational pan-sharpening with local gradient constraints. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10265–10274

  12. Gao H, Li S, Li J, Dian R (2023) Multispectral image pan-sharpening guided by component substitution model. IEEE Trans Geosci Remote Sens

  13. Gillespie AR, Kahle AB, Walker RE (1987) Color enhancement of highly correlated images. ii. channel ratio and “chromaticity” transformation techniques. Remote Sens Environ 22(3):343–365

  14. Gong M, Ma J, Xu H, Tian X, Zhang XP (2022) D2tnet: A convlstm network with dual-direction transfer for pan-sharpening. IEEE Trans Geosci Remote Sens 60:1–14

    Google Scholar 

  15. Gong Y, Xiao Z, Tan X, Sui H, Xu C, Duan H, Li D (2019) Context-aware convolutional neural network for object detection in vhr remote sensing imagery. IEEE Trans Geosci Remote Sens 58(1):34–44

    Google Scholar 

  16. Guo MH, Xu TX, Liu JJ, Liu ZN, Jiang PT, Mu TJ, Zhang SH, Martin RR, Cheng MM, Hu SM (2022) Attention mechanisms in computer vision: A survey. Comput Vis media 8(3):331–368

    Google Scholar 

  17. Hashim F, Dibs H, Jaber HS (2022) Adopting gram-schmidt and brovey methods for estimating land use and land cover using remote sensing and satellite images. Nat Environ Pollut Technol 21(2):867–881

    Google Scholar 

  18. He L, Rao Y, Li J, Chanussot J, Plaza A, Zhu J, Li B (2019) Pansharpening via detail injection based convolutional neural networks. IEEE J Sel Top Appl Earth Obs Remote Sens 12(4):1188–1204. https://doi.org/10.1109/JSTARS.2019.2898574

    Article  Google Scholar 

  19. Huang W, Xiao L, Wei Z, Liu H, Tang S (2015) A new pan-sharpening method with deep neural networks. IEEE Geosci Remote Sens Lett 12(5):1037–1041

    Google Scholar 

  20. Javan FD, Samadzadegan F, Mehravar S, Toosi A, Khatami R, Stein A (2021) A review of image fusion techniques for pan-sharpening of high-resolution satellite imagery. ISPRS J Photogramm Remote Sens 171:101–117

    Google Scholar 

  21. Jian L, Yang X, Liu Z, Jeon G, Gao M, Chisholm D (2020) Sedrfuse: A symmetric encoder-decoder with residual block network for infrared and visible image fusion. IEEE Trans Instrum Meas 70:1–15

    Google Scholar 

  22. Jianwen H, Zeping W, Pei H (2023) A review of pansharpening methods based on deep learning. Remote Sensing for Natural Resources 35(1)

  23. Jin C, Deng LJ, Huang TZ, Vivone G (2022) Laplacian pyramid networks: A new approach for multispectral pansharpening. Inf Fusion 78:158–170

    Google Scholar 

  24. Jin ZR, Zhang TJ, Jiang TX, Vivone G, Deng LJ (2022) Lagconv: Local-context adaptive convolution kernels with global harmonic bias for pansharpening. Proceedings of the AAAI conference on artificial intelligence vol 36, pp 1113–1121

  25. Lee J, Seo S, Kim M (2021) Sipsa-net: Shift-invariant pan sharpening with moving object alignment for satellite imagery. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10166–10174

  26. Leung Y, Liu J, Zhang J (2013) An improved adaptive intensity-hue-saturation method for the fusion of remote sensing images. IEEE Geosci Remote Sens Lett 11(5):985–989

    Google Scholar 

  27. Li X, Xu F, Lyu X, Tong Y, Chen Z, Li S, Liu D (2020) A remote-sensing image pan-sharpening method based on multi-scale channel attention residual network. IEEE Access 8:27163–27177

    Google Scholar 

  28. Liang Y, Zhang P, Mei Y, Wang T (2022) Pmacnet: Parallel multiscale attention constraint network for pan-sharpening. IEEE Geosci Remote Sens Lett 19:1–5

    Google Scholar 

  29. Liu Q, Zhou H, Xu Q, Liu X, Wang Y (2020) Psgan: A generative adversarial network for remote sensing image pan-sharpening. IEEE Trans Geosci Remote Sens 59(12):10227–10242

    Google Scholar 

  30. Liu X, Liu Q, Wang Y (2020) Remote sensing image fusion based on two-stream fusion network. Inf Fusion 55:1–15

    Google Scholar 

  31. Ma J, Yu W, Chen C, Liang P, Guo X, Jiang J (2020) Pan-gan: An unsupervised pan-sharpening method for remote sensing image fusion. Inf Fusion 62:110–120

    Google Scholar 

  32. Masi G, Cozzolino D, Verdoliva L, Scarpa G (2016) Pansharpening by convolutional neural networks. Remote Sensing 8(7):594

    Google Scholar 

  33. Meng X, Shen H, Li H, Zhang L, Fu R (2019) Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis: Practical discussion and challenges. Inf Fusion 46:102–113

    Google Scholar 

  34. Menon AS, Aravinth J, Veni S (2023) Pan-sharpening of multi-spectral remote sensing data using multi-resolution analysis. In: Machine intelligence techniques for data analysis and signal processing: proceedings of the 4th international conference MISP 2022, vol 1, pp 697–705. Springer

  35. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10), pp 807–814

  36. Nencini F, Garzelli A, Baronti S, Alparone L (2007) Remote sensing image fusion using the curvelet transform. Inf Fusion 8(2):143–156

    Google Scholar 

  37. Pan Z, Yu W, Yi X, Khan A, Yuan F, Zheng Y (2019) Recent progress on generative adversarial networks (gans): A survey. IEEE access 7:36322–36333

    Google Scholar 

  38. Qu Y, Baghbaderani RK, Qi H, Kwan C (2020) Unsupervised pansharpening based on self-attention mechanism. IEEE Trans Geosci Remote Sens 59(4):3192–3208

    Google Scholar 

  39. Shao Z, Cai J (2018) Remote sensing image fusion with deep convolutional neural network. IEEE J Sel Top Appl Earth Obs Remote Sens 11(5):1656–1669

    Google Scholar 

  40. Sharma KV, Kumar V, Singh K, Mehta DJ (2023) Landsat 8 lst pan sharpening using novel principal component based downscaling model. Remote Sens Appl: Soc Environ 30:100963

    Google Scholar 

  41. Su X, Li J, Hua Z (2022) Transformer-based regression network for pansharpening remote sensing images. IEEE Trans Geosci Remote Sens 60:1–23

    Google Scholar 

  42. Su Y, Zhu H, Wong KC, Chang Y, Li X (2022) Hyperspectral image denoising via weighted multidirectional low-rank tensor recovery. IEEE Trans Cybern 53(5):2753–2766

    Google Scholar 

  43. Suryanarayana G, Saidulu B, Priya MRH, Likhitha K, Pragathi K, Srikanth K (2022) Fusion of hyperspectral and multispectral images based on principal component analysis and guided bilateral filtering. International Journal of System Assurance Engineering and Management, pp 1–10

  44. Tang A, Quan P, Niu L, Shi Y (2022) A survey for sparse regularization based compression methods. Ann Data Sci 9(4):695–722

    Google Scholar 

  45. Tang X, Ma Q, Zhang X, Liu F, Ma J, Jiao L (2021) Attention consistent network for remote sensing scene classification. IEEE J Sel Top Appl Earth Obs Remote Sens 14:2030–2045

    Google Scholar 

  46. Tu TM, Huang PS, Hung CL, Chang CP (2004) A fast intensity-hue-saturation fusion technique with spectral adjustment for ikonos imagery. IEEE Geosci Remote Sens Lett 1(4):309–312

    Google Scholar 

  47. Wald L, Ranchin T, Mangolini M (1997) Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogramm Eng Remote Sensing 63:691–699

    Google Scholar 

  48. Wang Z, Bovik AC (2002) A universal image quality index. IEEE Signal Process Lett 9(3):81–84

    Google Scholar 

  49. Wang Z, Ma Y, Zhang Y (2023) Review of pixel-level remote sensing image fusion based on deep learning. Inf Fusion 90:36–58

    Google Scholar 

  50. Xing Y, Wang M, Yang S, Zhang K (2018) Pansharpening with multiscale geometric support tensor machine. IEEE Trans Geosci Remote Sens 56(5):2503–2517

    Google Scholar 

  51. Xing Y, Zhang Y, Zhang Y (2022) Wavefusion: Wavelet assistant fusion model for pan-sharpening. In: IGARSS 2022-2022 IEEE international geoscience and remote sensing symposium, pp 1083–1086. IEEE

  52. Xiong Z, Liu N, Wang N, Sun Z, Li W (2023) Unsupervised pansharpening method using residual network with spatial texture attention. IEEE Trans Geosci Remote Sens

  53. Xu Q, Li Y, Nie J, Liu Q, Guo M (2023) Upangan: Unsupervised pansharpening based on the spectral and spatial loss constrained generative adversarial network. Inf Fusion 91:31–46

    Google Scholar 

  54. Yan K, Zhou M, Liu L, Xie C, Hong D (2022) When pansharpening meets graph convolution network and knowledge distillation. IEEE Trans Geosci Remote Sens 60:1–15

    Google Scholar 

  55. Yilmaz CS, Yilmaz V, Gungor O (2022) A theoretical and practical survey of image fusion methods for multispectral pansharpening. Inf Fusion 79:1–43

    Google Scholar 

  56. Yuan Q, Wei Y, Meng X, Shen H, Zhang L (2018) A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening. IEEE J Sel Top Appl Earth Obs Remote Sens 11(3):978–989. https://doi.org/10.1109/JSTARS.2018.2794888

    Article  Google Scholar 

  57. Yuhas RH, Goetz AFH, Boardman JW (1992) Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm. In: JPL, Summaries of the third annual JPL airborne geoscience workshop. vol 1: AVIRIS Workshop

  58. Zhai W, Gao M, Souri A, Li Q, Guo X, Shang J, Zou G (2023) An attentive hierarchy convnet for crowd counting in smart city. Cluster Comput 26(2):1099–1111

    Google Scholar 

  59. Zhang H, Li Y, Jiang Y, Wang P, Shen Q, Shen C (2019) Hyperspectral classification based on lightweight 3-d-cnn with transfer learning. IEEE Trans Geosci Remote Sens 57(8):5813–5828

    Google Scholar 

  60. Zhang H, Ma J (2021) Gtp-pnet: A residual learning network based on gradient transformation prior for pansharpening. ISPRS J Photogramm Remote Sens 172:223–239

    Google Scholar 

  61. Zhang H, Xu H, Tian X, Jiang J, Ma J (2021) Image fusion meets deep learning: A survey and perspective. Inf Fusion 76:323–336

    Google Scholar 

  62. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In: Proceedings of the european conference on computer vision (ECCV), pp 286–301

  63. Zhang Y, Liu C, Sun M, Ou Y (2019) Pan-sharpening using an efficient bidirectional pyramid network. IEEE Trans Geosci Remote Sens 57(8):5549–5563. https://doi.org/10.1109/TGRS.2019.2900419

    Article  Google Scholar 

  64. Zheng Y, Li J, Li Y, Cao K, Wang K (2019) Deep residual learning for boosting the accuracy of hyperspectral pansharpening. IEEE Geosci Remote Sens Lett 17(8):1435–1439

    Google Scholar 

  65. Zhong S, Zhang Y, Chen Y, Wu D (2017) Combining component substitution and multiresolution analysis: A novel generalized bdsd pansharpening algorithm. IEEE J Sel Top Appl Earth Obs Remote Sens 10(6):2867–2875

    Google Scholar 

  66. Zhou M, Huang J, Fu X, Zhao F, Hong D (2022) Effective pan-sharpening by multiscale invertible neural network and heterogeneous task distilling. IEEE Trans Geosci Remote Sens 60:1–14

    Google Scholar 

  67. Zhou M, Huang J, Yan K, Yu H, Fu X, Liu A, Wei X, Zhao F (2022) Spatial-frequency domain information integration for pan-sharpening. In: European conference on computer vision, pp 274–291. Springer (2022)

Download references

Acknowledgements

This work is supported in part by the National Natural Science Foundation of China (No.61601266).

Author information

Authors and Affiliations

  1. School of Electrical and Electronic Engineering, Shandong University of Technology, Zibo, 255000, China

    Wenhao Song, Mingliang Gao, Wenzhe Zhai & Gwanggil Jeon

  2. Department of Mathematics and Computer Science, Royal Military College of Canada, Kingston, Canada

    Abdellah Chehri

  3. School of Electronic Engineering and Computer Science, Queen Mary University of London, London, E1 4NS, United Kingdom

    Qilei Li

  4. Department of Embedded Systems Engineering, Incheon National University, Incheon, 22012, South Korea

    Gwanggil Jeon

Authors
  1. Wenhao Song
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Mingliang Gao
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Abdellah Chehri
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Wenzhe Zhai
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Qilei Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Gwanggil Jeon
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

Wenhao Song: Conceptualization and Original draft. Mingliang Gao: Supervision, Review and Editing. Abdellah Chehri: English polishing. Wenzhe Zhai: Network development. Qilei Li: Data curation and Formal analysis. Gwanggil Jeon: Methodology and English polishing.

Corresponding author

Correspondence to Mingliang Gao.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Competing of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, W., Gao, M., Chehri, A. et al. Dual-branch and triple-attention network for pan-sharpening. Appl Intell 54, 8041–8058 (2024). https://doi.org/10.1007/s10489-024-05580-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05580-1

Keywords