Skip to main content
SpringerLink
Log in

DP-INNet: Dual-Path Implicit Neural Network for Spatial and Spectral Features Fusion in Pan-Sharpening

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14432))

Included in the following conference series:

  • 296 Accesses

Abstract

Pan-sharpening is a technique that fuses a high-resolution panchromatic (PAN) image with its corresponding low-resolution multispectral (MS) image to create a high-resolution multispectral image. Due to the powerful representation ability of Convolutional Neural Networks (CNNs), deep learning-based pan-sharpening methods have rapidly developed in recent years. However, existing methods often ignore the representation of multimodal information from the perspective of continuous physical signals, which inevitably leads to the loss of detailed information during the fusion process. Therefore, this paper proposes a novel pan-sharpening method that integrates spectral information with structural information in a continuous domain by using implicit neural representation (INR). Specifically, an implicit representation function is used to align the spatial information of multimodal images in the continuous domain, which preserves structural details. Additionally, a gated convolutional network is utilized to achieve interaction between different order spectral information in multispectral images. Finally, an MLP is used to fuse the spatial and spectral information in the continuous space to generate the expected high-resolution multispectral image. Extensive experiments on different datasets show that our method outperforms existing methods in terms of both quantitative and qualitative metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ballester, C., Caselles, V., Igual, L., Verdera, J., Rougé, B.: A variational model for P+XS image fusion. Int. J. Comput. Vision 69(1), 43 (2006)

    Article  Google Scholar 

  2. Cai, J., Huang, B.: Super-resolution-guided progressive pansharpening based on a deep convolutional neural network. IEEE Trans. Geosci. Remote Sens. 59(6), 5206–5220 (2020)

    Article  Google Scholar 

  3. Carper, W., Lillesand, T., Kiefer, R., et al.: The use of intensity-hue-saturation transformations for merging spot panchromatic and multispectral image data. Photogramm. Eng. Remote. Sens. 56(4), 459–467 (1990)

    Google Scholar 

  4. Chen, Y., Liu, S., Wang, X.: Learning continuous image representation with local implicit image function. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8628–8638 (2021)

    Google Scholar 

  5. Deng, L.J., Vivone, G., Jin, C., Chanussot, J.: Detail injection-based deep convolutional neural networks for pansharpening. IEEE Trans. Geosci. Remote Sens. 59(8), 6995–7010 (2020)

    Article  Google Scholar 

  6. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)

    Article  Google Scholar 

  7. Fu, X., Wang, W., Huang, Y., Ding, X., Paisley, J.: Deep multiscale detail networks for multiband spectral image sharpening. IEEE Trans. Neural Netw. Learn. Syst. 32(5), 2090–2104 (2020)

    Article  Google Scholar 

  8. Gillespie, A.R., Kahle, A.B., Walker, R.E.: Color enhancement of highly correlated images. ii. channel ratio and “chromaticity” transformation techniques. Remote Sens. Environ. 22(3), 343–365 (1987)

    Google Scholar 

  9. He, L., et al.: Pansharpening via detail injection based convolutional neural networks. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 12(4), 1188–1204 (2019)

    Article  Google Scholar 

  10. Henzler, P., Mitra, N.J., Ritschel, T.: Learning a neural 3D texture space from 2D exemplars. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8356–8364 (2020)

    Google Scholar 

  11. Jiang, C., Sud, A., Makadia, A., Huang, J., Nießner, M., Funkhouser, T., et al.: Local implicit grid representations for 3D scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6001–6010 (2020)

    Google Scholar 

  12. Jiang, Y., Ding, X., Zeng, D., Huang, Y., Paisley, J.: Pan-sharpening with a hyper-laplacian penalty. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 540–548 (2015)

    Google Scholar 

  13. Jin, Z.R., Deng, L.J., Zhang, T.J., Jin, X.X.: Bam: bilateral activation mechanism for image fusion. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 4315–4323 (2021)

    Google Scholar 

  14. Kwarteng, P., Chavez, A.: Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis. Photogramm. Eng. Remote. Sens. 55(1), 339–348 (1989)

    Google Scholar 

  15. Liao, W., et al.: Two-stage fusion of thermal hyperspectral and visible RGB image by PCA and guided filter. In: 2015 7th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), pp. 1–4. IEEE (2015)

    Google Scholar 

  16. Mallat, S.G.: A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans. Pattern Anal. Mach. Intell. 11(7), 674–693 (1989)

    Article  Google Scholar 

  17. Masi, G., Cozzolino, D., Verdoliva, L., Scarpa, G.: Pansharpening by convolutional neural networks. Remote Sens. 8(7), 594 (2016)

    Article  Google Scholar 

  18. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65(1), 99–106 (2021)

    Article  Google Scholar 

  19. Niemeyer, M., Mescheder, L., Oechsle, M., Geiger, A.: Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3504–3515 (2020)

    Google Scholar 

  20. Oechsle, M., Mescheder, L., Niemeyer, M., Strauss, T., Geiger, A.: Texture fields: learning texture representations in function space. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4531–4540 (2019)

    Google Scholar 

  21. Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: Deepsdf: learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165–174 (2019)

    Google Scholar 

  22. Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: PIFu: pixel-aligned implicit function for high-resolution clothed human digitization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2304–2314 (2019)

    Google Scholar 

  23. Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G.: Implicit neural representations with periodic activation functions. Adv. Neural. Inf. Process. Syst. 33, 7462–7473 (2020)

    Google Scholar 

  24. Vivone, G., et al.: A critical comparison among pansharpening algorithms. IEEE Trans. Geosci. Remote Sens. 53(5), 2565–2586 (2014)

    Article  Google Scholar 

  25. Xu, S., Zhang, J., Zhao, Z., Sun, K., Liu, J., Zhang, C.: Deep gradient projection networks for pan-sharpening. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1366–1375 (2021)

    Google Scholar 

  26. Yang, J., Fu, X., Hu, Y., Huang, Y., Ding, X., Paisley, J.: Pannet: a deep network architecture for pan-sharpening. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5449–5457 (2017)

    Google Scholar 

  27. Yang, Y., Tu, W., Huang, S., Lu, H., Wan, W., Gan, L.: Dual-stream convolutional neural network with residual information enhancement for pansharpening. IEEE Trans. Geosci. Remote Sens. 60, 1–16 (2021)

    Google Scholar 

  28. Yuan, Q., Wei, Y., Meng, X., Shen, H., Zhang, L.: A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 11(3), 978–989 (2018)

    Article  Google Scholar 

  29. Yuan, Q., Wei, Y., Zhang, Z., Shen, H., Zhang, L.: A multi-scale and multi-depth convolutional neural network for remote sensing imagery pan-sharpening (2017)

    Google Scholar 

  30. Zhang, H., Ma, J.: GTP-PNet: a residual learning network based on gradient transformation prior for pansharpening. ISPRS J. Photogramm. Remote. Sens. 172, 223–239 (2021)

    Article  Google Scholar 

Download references

Acknowledgements

The work was supported in part by the National Natural Science Foundation of China under Grant 82172033, U19B2031, 61971369, 52105126, 82272071, 62271430, and the Fundamental Research Funds for the Central Universities 20720230104.

Author information

Authors and Affiliations

  1. School of Informatics, Xiamen University, Xiamen, 361001, China

    Jingjia Huang, Ge Meng, Yunlong Lin, Yue Huang & Xinghao Ding

  2. Institute of Artificial Intelligence, Xiamen University, Xiamen, 361001, China

    Yingying Wang & Xinghao Ding

Authors
  1. Jingjia Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ge Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yingying Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yunlong Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yue Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xinghao Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yue Huang .

Editor information

Editors and Affiliations

  1. Nanjing University of Information Science and Technology, Nanjing, China

    Qingshan Liu

  2. Xiamen University, Xiamen, China

    Hanzi Wang

  3. Beijing University of Posts and Telecommunications, Beijing, China

    Zhanyu Ma

  4. Sun Yat-sen University, Guangzhou, China

    Weishi Zheng

  5. Peking University, Beijing, China

    Hongbin Zha

  6. Chinese Academy of Sciences, Beijing, China

    Xilin Chen

  7. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  8. Xiamen University, Xiamen, China

    Rongrong Ji

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Huang, J., Meng, G., Wang, Y., Lin, Y., Huang, Y., Ding, X. (2024). DP-INNet: Dual-Path Implicit Neural Network for Spatial and Spectral Features Fusion in Pan-Sharpening. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14432. Springer, Singapore. https://doi.org/10.1007/978-981-99-8543-2_22

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8543-2_22

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8542-5

  • Online ISBN: 978-981-99-8543-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation