Skip to main content
SpringerLink
Log in
Book cover

International Conference on Image and Graphics

ICIG 2021: Image and Graphics pp 425–436Cite as

Self-supervised Hyperspectral and Multispectral Image Fusion in Deep Neural Network

  • Conference paper
  • First Online:

  • 2348 Accesses

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12890))

Abstract

Hyperspectral and multispectral image fusion is to obtain high-resolution hyperspectral images from low-resolution hyperspectral images and high-resolution multispectral images. In recent years, many studies have applied deep learning methods to complete the fusion task. However, the function of deep learning-based methods is enslaved to the size and quality of training dataset, constraining the application of deep learning to the situation where training dataset is not available. In this paper, we introduce a new fusion algorithm, which operates in a self-supervised manner without training datasets. The proposed method obtains high-resolution hyperspectral images with the constraint of low-resolution hyperspectral image and a traditional fusion method. Several simulation and real-data experiments are conducted with remote sensing hyperspectral data under the condition where training datasets are unavailable. Quantitative and qualitative results indicates that the proposed method outperforms those traditional methods by a large extent.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Abdal, R., Qin, Y., Wonka, P.: Image2StyleGAN: how to embed images into the stylegan latent space? In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432–4441 (2019)

    Google Scholar 

  2. Aiazzi, B., Alparone, L., Baronti, S., Garzelli, A., Selva, M.: MTF-tailored multiscale fusion of high-resolution MS and pan imagery. Photogramm. Eng. Remote Sens. 72(5), 591–596 (2006)

    Article  Google Scholar 

  3. Aiazzi, B., Baronti, S., Selva, M.: Improving component substitution Pansharpening through multivariate regression of MS \(+ \) pan data. IEEE Trans. Geosci. Remote Sens. 45(10), 3230–3239 (2007)

    Article  Google Scholar 

  4. Akhtar, N., Shafait, F., Mian, A.: Sparse spatio-spectral representation for hyperspectral image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 63–78. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10584-0_5

    Chapter  Google Scholar 

  5. Akhtar, N., Shafait, F., Mian, A.: Bayesian sparse representation for hyperspectral image super resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3631–3640 (2015)

    Google Scholar 

  6. Carper, W., Lillesand, T., Kiefer, R.: The use of intensity-hue-saturation transformations for merging spot panchromatic and multispectral image data. Photogramm. Eng. Remote. Sens. 56(4), 459–467 (1990)

    Google Scholar 

  7. Chavez, P., Sides, S.C., Anderson, J.A., et al.: Comparison of three different methods to merge multiresolution and multispectral data- Landsat TM and SPOT panchromatic. Photogramm. Eng. Remote. Sens. 57(3), 295–303 (1991)

    Google Scholar 

  8. Gatys, L.A., Ecker, A.S., Bethge, M.: Texture synthesis using convolutional neural networks. arXiv preprint arXiv:1505.07376 (2015)

  9. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016)

    Google Scholar 

  10. Han, X.H., Zheng, Y., Chen, Y.W.: Multi-level and multi-scale spatial and spectral fusion CNN for hyperspectral image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (2019)

    Google Scholar 

  11. Iordache, M.D., Bioucas-Dias, J.M., Plaza, A.: Sparse Unmixing of hyperspectral data. IEEE Trans. Geosci. Remote Sens. 49(6), 2014–2039 (2011)

    Article  Google Scholar 

  12. Kwarteng, P., Chavez, A.: Extracting spectral contrast in Landsat thematic mapper image data using selective principal component analysis. Photogramm. Eng. Remote. Sens. 55(1), 339–348 (1989)

    Google Scholar 

  13. Laben, C.A., Brower, B.V.: Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. US Patent 6,011,875, 4 Jan 2000

    Google Scholar 

  14. Lanaras, C., Baltsavias, E., Schindler, K.: Hyperspectral super-resolution by coupled spectral unmixing. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3586–3594 (2015)

    Google Scholar 

  15. Liu, J.: Smoothing filter-based intensity modulation: a spectral preserve image fusion technique for improving spatial details. Int. J. Remote Sens. 21(18), 3461–3472 (2000)

    Article  Google Scholar 

  16. Palsson, F., Sveinsson, J.R., Ulfarsson, M.O.: Multispectral and hyperspectral image fusion using a 3-D-convolutional neural network. IEEE Geosci. Remote Sens. Lett. 14(5), 639–643 (2017)

    Article  Google Scholar 

  17. Plaza, A., Du, Q., Bioucas-Dias, J.M., Jia, X., Kruse, F.A.: Foreword to the special issue on spectral unmixing of remotely sensed data. IEEE Trans. Geosci. Remote Sens. 49(11), 4103–4110 (2011)

    Article  Google Scholar 

  18. Selva, M., Aiazzi, B., Butera, F., Chiarantini, L., Baronti, S.: Hyper-sharpening: a first approach on SIM-GA data. IEEE J. Selected Topics Appl. Earth Observations Remote Sens. 8(6), 3008–3024 (2015)

    Article  Google Scholar 

  19. Shaham, T.R., Dekel, T., Michaeli, T.: SinGAN: learning a generative model from a single natural image. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4570–4580 (2019)

    Google Scholar 

  20. Shahdoosti, H.R., Javaheri, N.: Pansharpening of clustered MS and pan images considering mixed pixels. IEEE Geosci. Remote Sens. Lett. 14(6), 826–830 (2017)

    Article  Google Scholar 

  21. Shocher, A., Bagon, S., Isola, P., Irani, M.: InGAN: capturing and retargeting the “DNA” of a natural image. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4492–4501 (2019)

    Google Scholar 

  22. Simoes, M., Bioucas-Dias, J., Almeida, L.B., Chanussot, J.: A convex formulation for hyperspectral image superresolution via subspace-based regularization. IEEE Trans. Geosci. Remote Sens. 53(6), 3373–3388 (2014)

    Article  Google Scholar 

  23. Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446–9454 (2018)

    Google Scholar 

  24. Upchurch, P., et al.: Deep feature interpolation for image content changes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7064–7073 (2017)

    Google Scholar 

  25. Wang, X., Yu, K., Dong, C., Tang, X., Loy, C.C.: Deep network interpolation for continuous imagery effect transition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1692–1701 (2019)

    Google Scholar 

  26. Yang, S., Zhang, K., Wang, M.: Learning low-rank decomposition for pan-sharpening with spatial-spectral offsets. IEEE Trans. Neural Networks Learn. Syst. 29(8), 3647–3657 (2017)

    MathSciNet  Google Scholar 

  27. Yasuma, F., Mitsunaga, T., Iso, D., Nayar, S.K.: Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans. Image Process. 19(9), 2241–2253 (2010)

    Article  MathSciNet  Google Scholar 

  28. Yokoya, N., Grohnfeldt, C., Chanussot, J.: Hyperspectral and multispectral data fusion: a comparative review of the recent literature. IEEE Geosci. Remote Sensing Mag. 5(2), 29–56 (2017)

    Article  Google Scholar 

  29. Yokoya, N., Yairi, T., Iwasaki, A.: Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Trans. Geosci. Remote Sens. 50(2), 528–537 (2011)

    Article  Google Scholar 

  30. Zhang, F., Du, B., Zhang, L.: Scene classification via a gradient boosting random convolutional network framework. IEEE Trans. Geosci. Remote Sens. 54(3), 1793–1802 (2015)

    Article  Google Scholar 

  31. Zhang, L., Shen, H., Gong, W., Zhang, H.: Adjustable model-based fusion method for multispectral and panchromatic images. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 42(6), 1693–1704 (2012)

    Google Scholar 

  32. Zhou, Y., Feng, L., Hou, C., Kung, S.Y.: Hyperspectral and multispectral image fusion based on local low rank and coupled spectral unmixing. IEEE Trans. Geosci. Remote Sens. 55(10), 5997–6009 (2017)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Geodesy and Geomatics, Wuhan University, Wuhan, People’s Republic of China

    Jianhao Gao, Jie Li, Qiangqiang Yuan & Jiang He

  2. School of Remote Sensing and Information Engineering, Wuhan University, Wuhan, People’s Republic of China

    Xin Su

Authors
  1. Jianhao Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jie Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qiangqiang Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiang He
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xin Su
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Li .

Editor information

Editors and Affiliations

  1. Peking University, Beijing, China

    Yuxin Peng

  2. Tsinghua University, Beijing, China

    Shi-Min Hu

  3. Tampere University, Tampere, Finland

    Moncef Gabbouj

  4. Zhejiang University, Hangzhou, China

    Kun Zhou

  5. Technion – Israel Institute of Technology, Haifa, Israel

    Michael Elad

  6. Tsinghua University, Beijing, China

    Kun Xu

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gao, J., Li, J., Yuan, Q., He, J., Su, X. (2021). Self-supervised Hyperspectral and Multispectral Image Fusion in Deep Neural Network. In: Peng, Y., Hu, SM., Gabbouj, M., Zhou, K., Elad, M., Xu, K. (eds) Image and Graphics. ICIG 2021. Lecture Notes in Computer Science(), vol 12890. Springer, Cham. https://doi.org/10.1007/978-3-030-87361-5_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-87361-5_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87360-8

  • Online ISBN: 978-3-030-87361-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation