Skip to main content
SpringerLink
Log in

Spatial-Frequency Domain Information Integration for Pan-Sharpening

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13678))

Included in the following conference series:

Abstract

Pan-sharpening aims to generate high-resolution multi-spectral (MS) images by fusing PAN images and low-resolution MS images. Despite its great advances, most existing pan-sharpening methods only work in the spatial domain and rarely explore the potential solutions in the frequency domain. In this paper, we first attempt to address pan-sharpening in both spatial and frequency domains and propose a Spatial-Frequency Information Integration Network, dubbed as SFIIN. To implement SFIIN, we devise a core building module tailored with pan-sharpening, consisting of three key components: spatial-domain information branch, frequency-domain information branch, and dual domain interaction. To be specific, the first employs the standard convolution to integrate the local information of two modalities of PAN and MS images in the spatial domain, while the second adopts deep Fourier transformation to achieve the image-wide receptive field for exploring global contextual information. Followed by, the third is responsible for facilitating the information flow and learning the complementary representation. We conduct extensive experiments to validate the effectiveness of the proposed network and demonstrate the favorable performance against other state-of-the-art methods.

M. Zhou and J. Huang—Co-first authors contributed equally.

F. Zhao—Corresponding author.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Addesso, P., Vivone, G., Restaino, R., Chanussot, J.: A data-driven model-based regression applied to panchromatic sharpening. IEEE Trans. Image Process. 29, 7779–7794 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  2. Aiazzi, B., Baronti, S., Selva, M.: Improving component substitution pansharpening through multivariate regression of ms \(+ \) pan data. IEEE Trans. Geosci. Remote Sens. 45(10), 3230–3239 (2007)

    Article  Google Scholar 

  3. Alparone, L., Wald, L., Chanussot, J., Thomas, C., Gamba, P., Bruce, L.M.: Comparison of pansharpening algorithms: Outcome of the 2006 grs-s data fusion contest. IEEE Trans. Geosci. Remote Sens. 45(10), 3012–3021 (2007)

    Article  Google Scholar 

  4. Ballester, C., Caselles, V., Igual, L., Verdera, J., Rougé, B.: A variational model for p+ xs image fusion. Int. J. Comput. Vision 69(1), 43–58 (2006)

    Article  Google Scholar 

  5. Cai, J., Huang, B.: Super-resolution-guided progressive pansharpening based on a deep convolutional neural network. IEEE Trans. Geosci. Remote Sens. 59(6), 5206–5220 (2021)

    Article  Google Scholar 

  6. Cao, X., Fu, X., Hong, D., Xu, Z., Meng, D.: PanCSC-net: a model-driven deep unfolding method for pansharpening. IEEE Trans. Geosci. Remote Sens. 1–13 (2021)

    Google Scholar 

  7. Cao, X., Zhou, F., Xu, L., Meng, D., Xu, Z., Paisley, J.: Hyperspectral image classification with Markov random fields and a convolutional neural network. IEEE Trans. Image Process. 27(5), 2354–2367 (2018)

    Article  MathSciNet  Google Scholar 

  8. Chen, C., Li, Y., Liu, W., Huang, J.: SIRF: simultaneous satellite image registration and fusion in a unified framework. IEEE Trans. Image Process. 24(11), 4213–4224 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  9. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  10. Frigo, M., Johnson, S.G.: FFTW: an adaptive software architecture for the FFT. In: 1988 International Conference on Acoustics, Speech, and Signal Processing. ICASSP-88, vol. 3 (1998)

    Google Scholar 

  11. Fu, X., Lin, Z., Huang, Y., Ding, X.: A variational pan-sharpening with local gradient constraints. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10265–10274 (2019)

    Google Scholar 

  12. Fu, X., Wang, W., Huang, Y., Ding, X., Paisley, J.: Deep multiscale detail networks for multiband spectral image sharpening. IEEE Trans. Neural Netw. Learn. Syst. 32(5), 2090–2104 (2021)

    Article  Google Scholar 

  13. Fu, Y., Liang, Z., You, S.: Bidirectional 3D quasi-recurrent neural network for hyperspectral image super-resolution. IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 14, 2674–2688 (2021)

    Article  Google Scholar 

  14. Fuoli, D., Gool, L.V., Timofte, R.: Fourier space losses for efficient perceptual image super-resolution (2021)

    Google Scholar 

  15. Ghahremani, M., Ghassemian, H.: Nonlinear IHS: a promising method for pan-sharpening. IEEE Geosci. Remote Sens. Lett. 13(11), 1606–1610 (2016)

    Article  Google Scholar 

  16. Gillespie, A.R., Kahle, A.B., Walker, R.E.: Color enhancement of highly correlated images. ii. channel ratio and “chromaticity” transformation techniques - sciencedirect. Remote Sens. Environ. 22(3), 343–365 (1987)

    Google Scholar 

  17. Haut, J.M., Paoletti, M.E., Plaza, J., Li, J., Plaza, A.: Active learning with convolutional neural networks for hyperspectral image classification using a new Bayesian approach. IEEE Trans. Geosci. Remote Sens. 56(11), 6440–6461 (2018)

    Article  Google Scholar 

  18. Haydn, R., Dalke, G.W., Henkel, J., Bare, J.E.: Application of the IHS color transform to the processing of multisensor data and image enhancement. Natl. Acad. Sci. USA 79(13), 571–577 (1982)

    Google Scholar 

  19. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  20. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)

    Google Scholar 

  21. J. R. H. Yuhas, A.F.G., Boardman, J.M.: Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm. In: Proceedings of Summaries Annual JPL Airborne Geoscience Workshop, pp. 147–149 (1992)

    Google Scholar 

  22. Jiang, J., Ma, J., Liu, X.: Multilayer spectral-spatial graphs for label noisy robust hyperspectral image classification. IEEE Trans. Neural Netw. Learn. Syst. 1–14 (2020)

    Google Scholar 

  23. Jiang, J., Ma, J., Wang, Z., Chen, C., Liu, X.: Hyperspectral image classification in the presence of noisy labels. IEEE Trans. Geosci. Remote Sens. 57(2), 851–865 (2019)

    Article  Google Scholar 

  24. Jiang, J., Sun, H., Liu, X., Ma, J.: Learning spatial-spectral prior for super-resolution of hyperspectral imagery. IEEE Trans. Comput. Imaging 6, 1082–1096 (2020)

    Article  Google Scholar 

  25. Jiang, K., Wang, Z., Yi, P., Jiang, J.: A progressively enhanced network for video satellite imagery superresolution. IEEE Sig. Process. Lett. 25(11), 1630–1634 (2018)

    Article  Google Scholar 

  26. Jiang, K., et al.: GAN-based multi-level mapping network for satellite imagery super-resolution. In: 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 526–531 (2019)

    Google Scholar 

  27. Khan, M.M., Chanussot, J., Condat, L., Montanvert, A.: Indusion: fusion of multispectral and panchromatic images using the induction scaling technique. IEEE Geosci. Remote Sens. Lett. 5(1), 98–102 (2008)

    Article  Google Scholar 

  28. Kwarteng, P., Chavez, A.: Extracting spectral contrast in Landsat thematic mapper image data using selective principal component analysis. Photogramm. Eng. Remote. Sens. 55(339–348), 1 (1989)

    Google Scholar 

  29. Laben, C., Brower, B.: Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. US Patent 6011875A (2000)

    Google Scholar 

  30. Laben, C.A., Brower, B.V.: Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. US Patent 6,011,875 (2000)

    Google Scholar 

  31. Liao, W., Xin, H., Coillie, F.V., Thoonen, G., Philips, W.: Two-stage fusion of thermal hyperspectral and visible RGB image by PCA and guided filter. In: Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (2017)

    Google Scholar 

  32. Liu., J.G.: Smoothing filter-based intensity modulation: a spectral preserve image fusion technique for improving spatial details. Int. J. Remote Sens. 21(18), 3461–3472 (2000)

    Google Scholar 

  33. Lu, X., Zhang, J., Yang, D., Xu, L., Jia, F.: Cascaded convolutional neural network-based hyperspectral image resolution enhancement via an auxiliary panchromatic image. IEEE Trans. Image Process. 30, 6815–6828 (2021)

    Article  Google Scholar 

  34. Ma, J., Xu, H., Jiang, J., Mei, X., Zhang, X.P.: DDCGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Trans. Image Process. 29, 4980–4995 (2020)

    Article  MATH  Google Scholar 

  35. Ma, J., Yu, W., Chen, C., Liang, P., Guo, X., Jiang, J.: Pan-GAN: an unsupervised pan-sharpening method for remote sensing image fusion. Inf. Fusion 62, 110–120 (2020)

    Article  Google Scholar 

  36. Masi, G., Cozzolino, D., Verdoliva, L., Scarpa, G.: Pansharpening by convolutional neural networks. Remote Sens. 8(7) (2016)

    Google Scholar 

  37. Shah, V.P., Younan, N.H., King, R.L.: An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets. IEEE Trans. Geosci. Remote Sens. 46(5), 1323–1335 (2008)

    Article  Google Scholar 

  38. Tian, X., Chen, Y., Yang, C., Gao, X., Ma, J.: A variational pansharpening method based on gradient sparse representation. IEEE Sig. Process. Lett. 27, 1180–1184 (2020)

    Article  Google Scholar 

  39. Tian, X., Chen, Y., Yang, C., Ma, J.: Variational pansharpening by exploiting cartoon-texture similarities. IEEE Trans. Geosci. Remote Sens. 1–16 (2021)

    Google Scholar 

  40. Tian, X., Li, K., Wang, Z., Ma, J.: VP-Net: an interpretable deep network for variational pansharpening. IEEE Trans. Geosci. Remote Sens. 1–16 (2021)

    Google Scholar 

  41. Vivone, G., et al.: A critical comparison among pansharpening algorithms. IEEE Trans. Geosci. Remote Sens. 53(5), 2565–2586 (2014)

    Article  Google Scholar 

  42. Wald, L., Ranchin, T., Mangolini, M.: Fusion of satellite images of different spatial resolutions: assessing the quality of resulting images. Photogram. Eng. Remote Sens. 63, 691–699 (1997)

    Google Scholar 

  43. Wang, X., Ma, J., Jiang, J.: Hyperspectral image super-resolution via recurrent feedback embedding and spatial-spectral consistency regularization. IEEE Trans. Geosci. Remote Sens. 1–13 (2021)

    Google Scholar 

  44. Wu, X., Huang, T.Z., Deng, L.J., Zhang, T.J.: Dynamic cross feature fusion for remote sensing pansharpening. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14687–14696, October 2021

    Google Scholar 

  45. Wu, Z.C., Huang, T.Z., Deng, L.J., Hu, J.F., Vivone, G.: Vo+net: an adaptive approach using variational optimization and deep learning for panchromatic sharpening. IEEE Trans. Geosci. Remote Sens. 1–16 (2021)

    Google Scholar 

  46. Xu, H., Ma, J., Shao, Z., Zhang, H., Jiang, J., Guo, X.: SDPNet: a deep network for pan-sharpening with enhanced information representation. IEEE Trans. Geosci. Remote Sens. 59(5), 4120–4134 (2021)

    Article  Google Scholar 

  47. Xu, S., Zhang, J., Zhao, Z., Sun, K., Liu, J., Zhang, C.: Deep gradient projection networks for pan-sharpening. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1366–1375, June 2021

    Google Scholar 

  48. Yan, K., Zhou, M., Liu, L., Xie, C., Hong, D.: When pansharpening meets graph convolution network and knowledge distillation. IEEE Trans. Geosci. Remote Sens. 60, 1–15 (2022). https://doi.org/10.1109/TGRS.2022.3168192

    Article  Google Scholar 

  49. Yang, G., Zhou, M., Yan, K., Liu, A., Fu, X., Wang, F.: Memory-augmented deep conditional unfolding network for pan-sharpening. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1788–1797, June 2022

    Google Scholar 

  50. Yang, J., Fu, X., Hu, Y., Huang, Y., Ding, X., Paisley, J.: PanNet: a deep network architecture for pan-sharpening. In: IEEE International Conference on Computer Vision, pp. 5449–5457 (2017)

    Google Scholar 

  51. Yuan, Q., Wei, Y., Meng, X., Shen, H., Zhang, L.: A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening. IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 11(3), 978–989 (2018)

    Article  Google Scholar 

  52. Zhang, H., Ma, J.: GTP-PNet: a residual learning network based on gradient transformation prior for pansharpening. ISPRS J. Photogramm. Remote. Sens. 172, 223–239 (2021)

    Article  Google Scholar 

  53. Zhou, M., Fu, X., Huang, J., Zhao, F., Liu, A., Wang, R.: Effective pan-sharpening with transformer and invertible neural network. IEEE Trans. Geosci. Remote Sens. 60, 1–15 (2022). https://doi.org/10.1109/TGRS.2021.3137967

    Article  Google Scholar 

  54. Zhou, M., Huang, J., Fang, Y., Fu, X., Liu, A.: Pan-Sharpening with Customized Transformer and Invertible Neural Network. AAAI Press, Palo Alto (2022)

    Google Scholar 

  55. Zhou, M., Xiao, Z., Fu, X., Liu, A., Yang, G., Xiong, Z.: Unfolding Taylor’s approximations for image restoration. In: NeurIPS (2021)

    Google Scholar 

  56. Zhou, M., Yan, K., Huang, J., Yang, Z., Fu, X., Zhao, F.: Mutual information-driven pan-sharpening. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1798–1808 (June 2022)

    Google Scholar 

  57. Zhou, M., Yan, K., Pan, J., Ren, W., Xie, Q., Cao, X.: Memory-augmented deep unfolding network for guided image super-resolution. arXiv abs/2203.04960 (2022)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.

Author information

Authors and Affiliations

  1. University of Science and Technology of China, Hefei, China

    Man Zhou, Jie Huang, Keyu Yan, Hu Yu, Xueyang Fu, Aiping Liu & Feng Zhao

  2. Hefei Institute of Physical Science, Chinese Academy of Sciences, Hefei, China

    Man Zhou & Keyu Yan

  3. MoE Engineering Research Center of Hardware/Software Co-design Technology and Application, East China Normal University, Shanghai, China

    Xian Wei

Authors
  1. Man Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jie Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Keyu Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hu Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xueyang Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Aiping Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Xian Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Feng Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Feng Zhao .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhou, M. et al. (2022). Spatial-Frequency Domain Information Integration for Pan-Sharpening. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13678. Springer, Cham. https://doi.org/10.1007/978-3-031-19797-0_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19797-0_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19796-3

  • Online ISBN: 978-3-031-19797-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation