Skip to main content

Advertisement

Log in

A novel spatial and spectral transformer network for hyperspectral image super-resolution

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Recently, transformer networks based on hyperspectral image super-resolution have achieved significant performance in comparison with most convolution neural networks. However, this is still an open problem of how to efficiently design a lightweight transformer structure to extract long-range spatial and spectral information from hyperspectral images. This paper proposes a novel spatial and spectral transformer network (SSTN) for hyperspectral image super-resolution. Specifically, the proposed transformer framework mainly consists of multiple consecutive alternating global attention layers and regional attention layers. In the global attention layer, a spatial and spectral self-attention module with less complexity is introduced to learn spatial and spectral global interaction, which can enhance the representation ability of the network. In addition, the proposed regional attention layer can extract regional feature information by using a window self-attention based on zero-padding strategy. This alternating architecture can adaptively learn regional and global feature information of hyperspectral images. Extensive experimental results demonstrate that the proposed method can achieve superior performance in comparison with the state-of-the-art hyperspectral image super-resolution methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability statement

The data that support the findings of this study are available on request from the corresponding author.

References

  1. Lu, G., Fei, B.: Medical hyperspectral imaging: a review. J. Biomed. Opt. 19(1), 10901 (2014)

    Article  Google Scholar 

  2. Khan, U., Paheding, S., Elkin, C.P., Devabhaktuni, V.K.: Trends in deep learning for medical hyperspectral image analysis. IEEE Access 9, 79534–79548 (2021)

    Article  Google Scholar 

  3. Calin, M.A., Parasca, S.V., Savastru, D., Manea, D.: Hyperspectral imaging in the medical field: Present and future. Appl. Spectrosc. Rev. 49(6), 435–447 (2014)

    Article  Google Scholar 

  4. Gowen, A., O’Donnell, C., Cullen, P., Downey, G., Frias, J.: Hyperspectral imaging - an emerging process analytical tool for food quality and safety control. Trends Food Sci. Technol. 18, 590–8 (2007)

    Article  Google Scholar 

  5. Feng, Y.Z., Sun, D.W.: Application of hyperspectral imaging in food safety inspection and control: a review. Crit. Rev. Food Sci. Nutr. 52(11), 1039–1058 (2012)

    Article  Google Scholar 

  6. Goetz, A.F.H.: Three decades of hyperspectral remote sensing of the earth: a personal view. Remote Sens. Environ. 113(1), S5–S16 (2009)

    Article  Google Scholar 

  7. Lv, Z., Wang, F., Cui, G., Benediktsson, J.A., Lei, T., Sun, W.: Spatial-spectral attention network guided with change magnitude image for land cover change detection using remote sensing images. IEEE Trans. Geosci. Remote Sens. 60, 1–12 (2022)

    Google Scholar 

  8. Adao, T., Hruska, J., Padua, L., Bessa, J., Peres, E., Morais, R., Sousa, J.J.: Hyperspectral imaging: a review on uav-based sensors, data processing and applications for agriculture and forestry. Remote Sens. 9(11), 1110 (2017)

    Article  Google Scholar 

  9. Feng, B., Liu, Y., Chi, H., Chen, X.: Hyperspectral remote sensing image classification based on residual generative adversarial neural networks. Signal Process. 213, 109202 (2023)

    Article  Google Scholar 

  10. Nasrabadi, M.N.: Hyperspectral target detection: an overview of current and future challenges Signal Processing. Magazine 31, 34–44 (2014)

    Google Scholar 

  11. Ke, C.: Military object detection using multiple information extracted from hyperspectral imagery. In: Int. Conf. Progress Inform. Comput. (PIC) 2017, 124–128 (2017)

  12. Liang, J., Zhou, J., Tong, L., Bai, X., Wang, B.: Material based salient object detection from hyperspectral images. Pattern Recogn. 76, 476–490 (2018)

    Article  Google Scholar 

  13. Yan, L., Zhao, M., Wang, X., Zhang, Y., Chen, J.: Object detection in hyperspectral images. IEEE Signal Process. Lett. 28, 508–512 (2021)

    Article  Google Scholar 

  14. Shi, C., Wu, H., Wang, L.: A positive feedback spatial-spectral correlation network based on spectral slice for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 61, 1–17 (2023)

    Google Scholar 

  15. Shi, C., Wu, H., Wang, L.: Cegat,: a cnn and enhanced-gat based on key sample selection strategy for hyperspectral image classification’’. Neural Netw. 168, 105–122 (2023)

    Article  Google Scholar 

  16. Wu, H., Shi, C., Wang, L., Jin, Z.: A cross-channel dense connection and multi-scale dual aggregated attention network for hyperspectral image classification. Remote Sens. 15(9), 2367 (2023)

    Article  Google Scholar 

  17. Shi, C., Wu, H., Wang, L.: A feature complementary attention network based on adaptive knowledge filtering for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 61, 1–19 (2023)

    Google Scholar 

  18. Yokoya, N., Grohnfeldt, C., Chanussot, J.: Hyperspectral and multispectral data fusion: a comparative review of the recent literature. IEEE Geosci. Remote Sens. Mag. 5(2), 29–56 (2017)

    Article  Google Scholar 

  19. Yao, W., Xi’ai, C., Zhi, H., Shiying, H.: Hyperspectral image super-resolution via nonlocal low-rank tensor approximation and total variation regularization. Remote Sens. 9(12), 1286 (2017)

    Article  Google Scholar 

  20. Huang, H., Yu, J., Sun, W.: “Super-resolution mapping via multi-dictionary based sparse representation,” in IEEE International Conference on Acoustics, 2014

  21. Arun, P.V., Buddhiraju, K.M., Porwal, A., Chanussot, J.: Cnn-based super-resolution of hyperspectral images. IEEE Trans. Geosci. Remote Sens. 99, 1–16 (2020)

    Google Scholar 

  22. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)

    Article  Google Scholar 

  23. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets’’. Neural Inform. Process. Syst. 2, 2672–2680 (2014)

    Google Scholar 

  24. Kim, J., Lee, J.K., Lee, K.M.: “Accurate image super-resolution using very deep convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

  25. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: “Residual dense network for image super-resolution,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  26. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision-ECCV, pp. 184–199. Springer International Publishing, Cham (2014)

    Google Scholar 

  27. Xue, L., Shen, J., Wang, R., Yang, J.: Mffn: multi-path feedback fusion network for lightweight image super resolution. IET Image Process. 17, 4190–201 (2023)

    Article  Google Scholar 

  28. Liu, Y., Yue, M., Yan, H., Zhu, L.: Single-image super-resolution using lightweight transformer-convolutional neural network hybrid model. IET Image Process. 17, 2881–93 (2023)

    Article  Google Scholar 

  29. Liu, H., Guo, H., Liu, X.: Uha-cyclegan: unpaired hybrid attention network based on cyclegan for terahertz image super-resolution’’. IET Image Process. 17, 2547–59 (2023)

    Article  Google Scholar 

  30. Li, Y., Hu, J., Zhao, X., Xie, W., Li, J.: Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing 266, 29–41 (2017)

    Article  Google Scholar 

  31. Li, Y., Zhang, L., Dingl, C., Wei, W., Zhang, Y.: “Single hyperspectral image super-resolution with grouped deep recursive residual network,” in 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), 2018

  32. Shaohui, M., Xin, Y., Jingyu, J., Yifan, Z., Shuai, W., Qian, D.: Hyperspectral image spatial super-resolution via 3d full convolutional neural network. Remote Sens. 9(11), 1139 (2017)

    Article  Google Scholar 

  33. Wang, X., Hu, Q., Jiang, J., Ma, J.: A group-based embedding learning and integration network for hyperspectral image super-resolution. IEEE Trans. Geosci. Remote Sens. 60, 1–16 (2022)

    Google Scholar 

  34. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N.: Kaiser, L.u., Polosukhin, I., “Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30. Minus Curran Associates Inc, Berlin (2017)

    Google Scholar 

  35. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold G., Gelly, S., Uszkoreit, J., Houlsby, N.: “An image is worth 16x16 words: Transformers for image recognition at scale,” CoRR, vol. abs/2010.11929, 2020

  36. Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 568–578 October (2021)

  37. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 10012–10022, October (2021)

  38. Ding, M., Xiao, B., Codella, N., Luo, P., Wang, J., Yuan, L.: “Davit: Dual attention vision transformers,” in Computer Vision - ECCV,: S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner. Eds. minus Cham: Springer Nature Switzerland 2022, 74–92 (2022)

  39. Chen, Q., Wu, Q., Wang, J., Hu, Q., Hu, T., Ding, E., Cheng, J., Wang, J.: Mixformer: Mixing features across windows and dimensions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 5249–5259, June (2022)

  40. Kang, B., Chen, X., Wang, D., Peng, H., Lu, H.: “Exploring lightweight hierarchical vision transformers for efficient visual tracking,” 2023

  41. Choi, H., Na, C., Oh, J., Lee, S., Kim, J., Choe, S., Lee, J., Kim, T., Yang, J.: “Ramit: Reciprocal attention mixing transformer for lightweight image restoration,” 2023

  42. Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., Li, Y.: Maxvit Multi-axis vision transformer. In: Avidan, S., Brostow, G., Cisse, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision-ECCv, pp. 459–479. Springer, Cham (2022)

    Google Scholar 

  43. Wang, W., Yao, L., Chen, L., Cai, D., He, X., Liu, W.: “Crossformer: A versatile vision transformer based on cross-scale attention,” CoRR, vol. abs/2108.00154, 2021

  44. Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., Guo, B.: Cswin transformer: A general vision transformer backbone with cross-shaped windows. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12124–12134,(2022)

  45. Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: “Swinir: Image restoration using swin transformer,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, Oct 2021, 1833–1844

  46. Hu, J.-F., Huang, T.-Z., Deng, L.-J., Dou, H.-X., Hong, D., Vivone, G.: Fusformer: A transformer-based fusion network for hyperspectral image super-resolution. IEEE Geosci. Remote Sens. Lett. 19, 1–5 (2022)

    Google Scholar 

  47. Ma, Q., Jiang, J., Liu, X., Ma, J.: Learning a 3d-cnn and transformer prior for hyperspectral image super-resolution. Inform. Fusion 100, 101907 (2023)

    Article  Google Scholar 

  48. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  49. Yuhas, R.H., Goetz, A.F.H., Boardman, J.W.: “Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm,” JPL, Summaries of the Third Annual JPL Airborne Geoscience Workshop. Volume 1: AVIRIS Workshop, 1992

  50. Loncan, L., de Almeida, L.B., Bioucas-Dias, J.M., Briottet, X., Chanussot, J., Dobigeon, N., Fabre, S., Liao, W., Licciardi, G.A., Simões, M., Tourneret, J.-Y., Veganzones, M.A., Vivone, G., Wei, Q., Yokoya, N.: Hyperspectral pansharpening: a review. IEEE Geosci. Remote Sens. Mag. 3(3), 27–46 (2015)

    Article  Google Scholar 

  51. Jiang, J., Sun, H., Liu, X., Ma, J.: Learning spatial-spectral prior for super-resolution of hyperspectral imagery. IEEE Trans. Computat. Imaging 6, 1082–96 (2020)

    Article  Google Scholar 

  52. Shi, J., Li, H., Liu, T., Liu, Y., Zhang, M., Zhu, J., Zheng, L., Weng, S.: “Image super-resolution using efficient striped window transformer,” (2023)

  53. Chen, X., Wang, X., Zhou, J., Dong, C.: “Activating more pixels in image super-resolution transformer,” arXiv e-prints, 2022

  54. Yasuma, F., Mitsunaga, T., Iso, D., Nayar, S.K.: Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans. Image Process. Publ IEEE Signal Process. Soc. 19(9), 2241 (2010)

    Article  MathSciNet  Google Scholar 

  55. Chakrabarti, A., Zickler, T.: “Statistics of Real-World Hyperspectral Images,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 193–200

  56. Yokoya, N., Iwasaki, A.: “Airborne hyperspectral data over chikusei,” Space Application Laboratory, University of Tokyo, Japan, Tech. Rep. SAL-2016-05-27, (2016)

Download references

Author information

Authors and Affiliations

Authors

Contributions

Huapeng Wu: Conceptualization, Formal analysis, Methodology, Validation, Writing – original draft, Writing – review & editing Hui Xu: Methodology, Validation, Visualization, Writing – review & editing Tianming Zhan: Investigation, Resources, Supervision, Writing – review & editing

Corresponding author

Correspondence to Tianming Zhan.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Communicated by Q. Shen.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported in part by the National Natural Science Foundation of China under Grant 61976117, 62375133, in part by the Qinglan Project, in part by the Key Projects of University Natural Science Fund of Jiangsu Province under Grant 23KJA520009, in part by the Research Project of University Natural Science Fund of Jiangsu Province under Grant 22KJB520002, in part by the Natural Science Foundation of Jiangsu Province under Grant BK20230440, and in part by the Postgraduate Research Practice Innovation Program of Jiangsu Province under Grant KYCX22_2220.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, H., Xu, H. & Zhan, T. A novel spatial and spectral transformer network for hyperspectral image super-resolution. Multimedia Systems 30, 165 (2024). https://doi.org/10.1007/s00530-024-01363-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00530-024-01363-3

Keywords