Skip to main content
SpringerLink
Log in

STP-SOM: Scale-Transfer Learning for Pansharpening via Estimating Spectral Observation Model

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Pansharpening strives to improve the spatial resolution of multi-spectral images while maintaining spectral fidelity. However, existing methods usually cannot guarantee a balance between spatial and spectral quality, and degrade when handling the desirable scale of full resolution. To address these challenges, this prospective study proposes a scale-transfer learning framework via estimating spectral observation model. Specifically, we design a cross-spectral transfer network to learn an expected spectral observation model by the cycle adversarial between spectral degradation and interpolation, which describes the accurate nonlinear mapping process from multi-spectral to panchromatic images. Having the favorable spectral observation model established, a scale-transfer pansharpening paradigm with co-learning of full and reduced resolutions can be constructed. First, we develop a spectral observation model-based spatial fidelity term at the reduced-resolution scale, which can alleviate the imbalance problem of spectral and spatial information widespread in current supervised paradigms. Second, we explore the reprojection regularization from full to reduced resolution based on the spectral observation model, which facilitates the ability of the pansharpening model to be extended to the scale of full resolution. Extensive experiments demonstrate the advantage of our method over the current state-of-the-arts in terms of information balance and scale transformation. We further apply our method to produce the high-resolution normalized difference vegetation index and achieve vegetation enhancement with competitive performance. Moreover, our method is lightweight and faster than other comparative methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Algorithm 1
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Data Availability

The data or code during the current study are available from the corresponding author on reasonable request.

References

  • Aiazzi, B., Alparone, L., Baronti, S., & Garzelli, A. (2002). Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Transactions on Geoscience and Remote Sensing, 40(10), 2300–2312.

    Google Scholar 

  • Aiazzi, B., Alparone, L., Baronti, S., Garzelli, A., & Selva, M. (2006). MTF-tailored multiscale fusion of high-resolution MS and Pan imagery. Photogrammetric Engineering & Remote Sensing, 72(5), 591–596.

    Google Scholar 

  • Aiazzi, B., Baronti, S., & Selva, M. (2007). Improving component substitution pansharpening through multivariate regression of ms \(+ \) pan data. IEEE Transactions on Geoscience and Remote Sensing, 45(10), 3230–3239.

    Google Scholar 

  • Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A., Nencini, F., & Selva, M. (2008). Multispectral and panchromatic data fusion assessment without reference. Photogrammetric Engineering & Remote Sensing, 74(2), 193–200.

    Google Scholar 

  • Ballester, C., Caselles, V., Igual, L., Verdera, J., & Rougé, B. (2006). A variational model for P+ XS image fusion. International Journal of Computer Vision, 69(1), 43–58.

    Google Scholar 

  • Cai, J., & Huang, B. (2021). Super-resolution-guided progressive pansharpening based on a deep convolutional neural network. IEEE Transactions on Geoscience and Remote Sensing, 59(6), 5206–5220.

    Google Scholar 

  • Carper, W., Lillesand, T., & Kiefer, R. (1990). The use of intensity-hue-saturation transformations for merging spot panchromatic and multispectral image data. Photogrammetric Engineering and Remote Sensing, 56(4), 459–467.

    Google Scholar 

  • Chen, C., Li, Y., Liu, W., & Huang, J. (2014). Image fusion with local spectral consistency and dynamic gradient sparsity. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 2760–2765).

  • Chen, C., Li, Y., Liu, W., & Huang, J. (2015). SIRF: Simultaneous satellite image registration and fusion in a unified framework. IEEE Transactions on Image Processing, 24(11), 4213–4224.

    MathSciNet  MATH  Google Scholar 

  • Choi, J., Yu, K., & Kim, Y. (2010). A new adaptive component-substitution-based satellite image fusion by using partial replacement. IEEE Transactions on Geoscience and Remote Sensing, 49(1), 295–309.

    Google Scholar 

  • Choi, M. (2006). A new intensity-hue-saturation fusion approach to image fusion with a tradeoff parameter. IEEE Transactions on Geoscience and Remote Sensing, 44(6), 1672–1682.

    Google Scholar 

  • Ciotola, M., Vitale, S., Mazza, A., Poggi, G., & Scarpa, G. (2022). Pansharpening by convolutional neural networks in the full resolution framework. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–17.

    Google Scholar 

  • Deng, L. J., Vivone, G., Paoletti, M. E., Scarpa, G., He, J., Zhang, Y., Chanussot, J., & Plaza, A. (2022). Machine learning in pansharpening: A benchmark, from shallow to deep networks. IEEE Geoscience and Remote Sensing Magazine, 10(3), 279–315.

    Google Scholar 

  • Dou, W. (2018). Image degradation for quality assessment of pan-sharpening methods. Remote Sensing, 10(1), 154.

    MathSciNet  Google Scholar 

  • Fu, X., Lin, Z., Huang, Y., & Ding, X. (2019). A variational pan-sharpening with local gradient constraints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 10,265–10,274).

  • Garzelli, A., Aiazzi, B., Alparone, L., Lolli, S., & Vivone, G. (2018). Multispectral pansharpening with radiative transfer-based detail-injection modeling for preserving changes in vegetation cover. Remote Sensing, 10(8), 1308.

    Google Scholar 

  • Garzelli, A., Nencini, F., & Capobianco, L. (2007). Optimal MMSE pan sharpening of very high resolution multispectral images. IEEE Transactions on Geoscience and Remote Sensing, 46(1), 228–236.

    Google Scholar 

  • Ghassemian, H. (2016). A review of remote sensing image fusion methods. Information Fusion, 32, 75–89.

    Google Scholar 

  • Guo, A., Dian, R., & Li, S. (2020). Unsupervised blur kernel learning for pansharpening. In Proceedings of the IEEE International Geoscience and Remote Sensing Symposium, (pp. 633–636).

  • Javan, F. D., Samadzadegan, F., Mehravar, S., Toosi, A., Khatami, R., & Stein, A. (2021). A review of image fusion techniques for pan-sharpening of high-resolution satellite imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 171, 101–117.

    Google Scholar 

  • Jiang, Y., Ding, X., Zeng, D., Huang, Y., & Paisley, J. (2015). Pan-sharpening with a hyper-laplacian penalty. In Proceedings of the IEEE International Conference on Computer Vision, (pp. 540–548).

  • Jin, C., Deng, L. J., Huang, T. Z., & Vivone, G. (2022). Laplacian pyramid networks: A new approach for multispectral pansharpening. Information Fusion, 78, 158–170.

    Google Scholar 

  • Jin, Z. R., Zhang, T. J., Jiang, T. X., Vivone, G., & Deng, L. J. (2022). Lagconv: Local-context adaptive convolution kernels with global harmonic bias for pansharpening. In Proceedings of the AAAI Conference on Artificial Intelligence, (vol. 36, pp. 1113–1121).

  • Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

  • Li, J., Luo, J., Ming, D., & Shen, Z. (2005). A new method for merging ikonos panchromatic and multispectral image data. In: Proceedings of the IEEE International Symposium on Geoscience and Remote Sensing, (pp. 3916–3919).

  • Li, S., Tian, Y., Xia, H., & Liu, Q. (2022). Unmixing based pan guided fusion network for hyperspectral imagery. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–17.

    Google Scholar 

  • Lowe, D. G. (1999). Object recognition from local scale-invariant features. In Proceedings of the IEEE International Conference on Computer Vision, (vol. 2, pp. 1150–1157).

  • Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.

    Google Scholar 

  • Ma, J., Yu, W., Chen, C., Liang, P., Guo, X., & Jiang, J. (2020). Pan-GAN: An unsupervised pan-sharpening method for remote sensing image fusion. Information Fusion, 62, 110–120.

    Google Scholar 

  • Masi, G., Cozzolino, D., Verdoliva, L., & Scarpa, G. (2016). Pansharpening by convolutional neural networks. Remote Sensing, 8(7), 594.

    Google Scholar 

  • Massip, P., Blanc, P., & Wald, L. (2011). A method to better account for modulation transfer functions in arsis-based pansharpening methods. IEEE Transactions on Geoscience and Remote Sensing, 50(3), 800–808.

    Google Scholar 

  • Meng, X., Bao, K., Shu, J., Zhou, B., Shao, F., Sun, W., & Li, S. (2021). A blind full-resolution quality evaluation method for pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–16.

    Google Scholar 

  • Meng, X., Shen, H., Li, H., Zhang, L., & Fu, R. (2019). Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis: Practical discussion and challenges. Information Fusion, 46, 102–113.

    Google Scholar 

  • Palsson, F., Sveinsson, J. R., Ulfarsson, M. O., & Benediktsson, J. A. (2015). Model based pansharpening method based on tv and mtf deblurring. In Proceedings of the IEEE International Symposium on Geoscience and Remote Sensing, (pp. 33–36).

  • Palsson, F., Sveinsson, J. R., Ulfarsson, M. O., & Benediktsson, J. A. (2015). Quantitative quality evaluation of pansharpened imagery: Consistency versus synthesis. IEEE Transactions on Geoscience and Remote Sensing, 54(3), 1247–1259.

    Google Scholar 

  • Restaino, R., Dalla Mura, M., Vivone, G., & Chanussot, J. (2017). Context-adaptive pansharpening based on image segmentation. IEEE Transactions on Geoscience and Remote Sensing, 55(2), 753–766.

    MATH  Google Scholar 

  • Shah, V. P., Younan, N. H., & King, R. L. (2008). An efficient pan-sharpening method via a combined adaptive PCA approach and contourlets. IEEE Transactions on Geoscience and Remote Sensing, 46(5), 1323–1335.

    Google Scholar 

  • Thomas, C., & Wald, L. (2006). Analysis of changes in quality assessment with scale. In Proceedings of the International Conference on Information Fusion, (pp. 1–5).

  • Tian, X., Chen, Y., Yang, C., Gao, X., & Ma, J. (2020). A variational pansharpening method based on gradient sparse representation. IEEE Signal Processing Letters, 27, 1180–1184.

    Google Scholar 

  • Tu, T. M., Su, S. C., Shyu, H. C., & Huang, P. S. (2001). A new look at IHS-like image fusion methods. Information Fusion, 2(3), 177–186.

    Google Scholar 

  • Tucker, C. J. (1979). Red and photographic infrared linear combinations for monitoring vegetation. Remote Sensing of Environment, 8(2), 127–150.

    Google Scholar 

  • Vivone, G. (2019). Robust band-dependent spatial-detail approaches for panchromatic sharpening. IEEE Transactions on Geoscience and Remote Sensing, 57(9), 6421–6433.

    Google Scholar 

  • Vivone, G., Restaino, R., & Chanussot, J. (2018a). A Bayesian procedure for full-resolution quality assessment of pansharpened products. IEEE Transactions on Geoscience and Remote Sensing, 56(8), 4820–4834.

    Google Scholar 

  • Vivone, G., Restaino, R., & Chanussot, J. (2018b). Full scale regression-based injection coefficients for panchromatic sharpening. IEEE Transactions on Image Processing, 27(7), 3418–3431.

    MathSciNet  MATH  Google Scholar 

  • Wald, L. (2002). Data fusion: Definitions and architectures: Fusion of images of different spatial resolutions. Presses des Mines.

  • Wald, L., Ranchin, T., & Mangolini, M. (1997). Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing, 63(6), 691–699.

    Google Scholar 

  • Wang, W., Liu, H., Liang, L., Liu, Q., & Xie, G. (2019). A regularised model-based pan-sharpening method for remote sensing images with local dissimilarities. International Journal of Remote Sensing, 40(8), 3029–3054.

    Google Scholar 

  • Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4), 600–612.

    Google Scholar 

  • Xiao, J. L., Huang, T. Z., Deng, L. J., Wu, Z. C., & Vivone, G. (2022). A new context-aware details injection fidelity with adaptive coefficients estimation for variational pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 60, 5408015.

    Google Scholar 

  • Yang, J., Fu, X., Hu, Y., Huang, Y., Ding, X., & Paisley, J. (2017). PanNet: A deep network architecture for pan-sharpening. In Proceedings of the IEEE International Conference on Computer Vision, (pp. 5449–5457).

  • Yilmaz, C. S., Yilmaz, V., & Gungor, O. (2022). A theoretical and practical survey of image fusion methods for multispectral pansharpening. Information Fusion, 79, 1–43.

    Google Scholar 

  • Zhang, H., & Ma, J. (2021). GTP-PNet: A residual learning network based on gradient transformation prior for pansharpening. ISPRS Journal of Photogrammetry and Remote Sensing, 172, 223–239.

    Google Scholar 

  • Zhang, H., Ma, J., Chen, C., & Tian, X. (2020). NDVI-Net: A fusion network for generating high-resolution normalized difference vegetation index in remote sensing. ISPRS Journal of Photogrammetry and Remote Sensing, 168, 182–196.

    Google Scholar 

  • Zhang, H., Xu, H., Tian, X., Jiang, J., & Ma, J. (2021). Image fusion meets deep learning: A survey and perspective. Information Fusion, 76, 323–336.

    Google Scholar 

  • Zhang, K., Zuo, W., & Zhang, L. (2018). Learning a single convolutional super-resolution network for multiple degradations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (pp. 3262–3271).

  • Zhang, L., Zhang, L., Mou, X., & Zhang, D. (2011). FSIM: A feature similarity index for image quality assessment. IEEE Transactions on Image Processing, 20(8), 2378–2386.

  • Zhang, L., Zhang, L., Tao, D., Huang, X., & Du, B. (2013). Hyperspectral remote sensing image subpixel target detection based on supervised metric learning. IEEE Transactions on Geoscience and Remote Sensing, 52(8), 4955–4965.

    Google Scholar 

  • Zhang, M., & Ling, Q. (2021). Supervised pixel-wise GAN for face super-resolution. IEEE Transactions on Multimedia, 23, 1938–1950.

  • Zhou, M., Huang, J., Fang, Y., Fu, X., & Liu, A. (2022). Pan-sharpening with customized transformer and invertible neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, (vol. 36, pp. 3553–3561).

  • Zhou, M., Yan, K., Huang, J., Yang, Z., Fu, X., & Zhao, F. (2022). Mutual information-driven pan-sharpening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (pp. 1798–1808).

  • Zhou, M., Yu, H., Huang, J., Zhao, F., Gu, J., Loy, C. C., Meng, D., & Li, C. (2022). Deep fourier up-sampling. Advances in Neural Information Processing Systems, 35, 22995–23008.

    Google Scholar 

  • Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE Iinternational Conference on Computer Vision, (pp. 2223–2232).

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant no. 62276192.

Author information

Authors and Affiliations

  1. Electronic Information School, Wuhan University, Wuhan, 430072, China

    Hao Zhang & Jiayi Ma

Authors
  1. Hao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiayi Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiayi Ma.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Communicated by Michael Ying Yang.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, H., Ma, J. STP-SOM: Scale-Transfer Learning for Pansharpening via Estimating Spectral Observation Model. Int J Comput Vis 131, 3226–3251 (2023). https://doi.org/10.1007/s11263-023-01840-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-023-01840-8

Keywords

Search

Navigation