Abstract
Recently, transformer networks based on hyperspectral image super-resolution have achieved significant performance in comparison with most convolution neural networks. However, this is still an open problem of how to efficiently design a lightweight transformer structure to extract long-range spatial and spectral information from hyperspectral images. This paper proposes a novel spatial and spectral transformer network (SSTN) for hyperspectral image super-resolution. Specifically, the proposed transformer framework mainly consists of multiple consecutive alternating global attention layers and regional attention layers. In the global attention layer, a spatial and spectral self-attention module with less complexity is introduced to learn spatial and spectral global interaction, which can enhance the representation ability of the network. In addition, the proposed regional attention layer can extract regional feature information by using a window self-attention based on zero-padding strategy. This alternating architecture can adaptively learn regional and global feature information of hyperspectral images. Extensive experimental results demonstrate that the proposed method can achieve superior performance in comparison with the state-of-the-art hyperspectral image super-resolution methods.







Similar content being viewed by others
Data availability statement
The data that support the findings of this study are available on request from the corresponding author.
References
Lu, G., Fei, B.: Medical hyperspectral imaging: a review. J. Biomed. Opt. 19(1), 10901 (2014)
Khan, U., Paheding, S., Elkin, C.P., Devabhaktuni, V.K.: Trends in deep learning for medical hyperspectral image analysis. IEEE Access 9, 79534–79548 (2021)
Calin, M.A., Parasca, S.V., Savastru, D., Manea, D.: Hyperspectral imaging in the medical field: Present and future. Appl. Spectrosc. Rev. 49(6), 435–447 (2014)
Gowen, A., O’Donnell, C., Cullen, P., Downey, G., Frias, J.: Hyperspectral imaging - an emerging process analytical tool for food quality and safety control. Trends Food Sci. Technol. 18, 590–8 (2007)
Feng, Y.Z., Sun, D.W.: Application of hyperspectral imaging in food safety inspection and control: a review. Crit. Rev. Food Sci. Nutr. 52(11), 1039–1058 (2012)
Goetz, A.F.H.: Three decades of hyperspectral remote sensing of the earth: a personal view. Remote Sens. Environ. 113(1), S5–S16 (2009)
Lv, Z., Wang, F., Cui, G., Benediktsson, J.A., Lei, T., Sun, W.: Spatial-spectral attention network guided with change magnitude image for land cover change detection using remote sensing images. IEEE Trans. Geosci. Remote Sens. 60, 1–12 (2022)
Adao, T., Hruska, J., Padua, L., Bessa, J., Peres, E., Morais, R., Sousa, J.J.: Hyperspectral imaging: a review on uav-based sensors, data processing and applications for agriculture and forestry. Remote Sens. 9(11), 1110 (2017)
Feng, B., Liu, Y., Chi, H., Chen, X.: Hyperspectral remote sensing image classification based on residual generative adversarial neural networks. Signal Process. 213, 109202 (2023)
Nasrabadi, M.N.: Hyperspectral target detection: an overview of current and future challenges Signal Processing. Magazine 31, 34–44 (2014)
Ke, C.: Military object detection using multiple information extracted from hyperspectral imagery. In: Int. Conf. Progress Inform. Comput. (PIC) 2017, 124–128 (2017)
Liang, J., Zhou, J., Tong, L., Bai, X., Wang, B.: Material based salient object detection from hyperspectral images. Pattern Recogn. 76, 476–490 (2018)
Yan, L., Zhao, M., Wang, X., Zhang, Y., Chen, J.: Object detection in hyperspectral images. IEEE Signal Process. Lett. 28, 508–512 (2021)
Shi, C., Wu, H., Wang, L.: A positive feedback spatial-spectral correlation network based on spectral slice for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 61, 1–17 (2023)
Shi, C., Wu, H., Wang, L.: Cegat,: a cnn and enhanced-gat based on key sample selection strategy for hyperspectral image classification’’. Neural Netw. 168, 105–122 (2023)
Wu, H., Shi, C., Wang, L., Jin, Z.: A cross-channel dense connection and multi-scale dual aggregated attention network for hyperspectral image classification. Remote Sens. 15(9), 2367 (2023)
Shi, C., Wu, H., Wang, L.: A feature complementary attention network based on adaptive knowledge filtering for hyperspectral image classification. IEEE Trans. Geosci. Remote Sens. 61, 1–19 (2023)
Yokoya, N., Grohnfeldt, C., Chanussot, J.: Hyperspectral and multispectral data fusion: a comparative review of the recent literature. IEEE Geosci. Remote Sens. Mag. 5(2), 29–56 (2017)
Yao, W., Xi’ai, C., Zhi, H., Shiying, H.: Hyperspectral image super-resolution via nonlocal low-rank tensor approximation and total variation regularization. Remote Sens. 9(12), 1286 (2017)
Huang, H., Yu, J., Sun, W.: “Super-resolution mapping via multi-dictionary based sparse representation,” in IEEE International Conference on Acoustics, 2014
Arun, P.V., Buddhiraju, K.M., Porwal, A., Chanussot, J.: Cnn-based super-resolution of hyperspectral images. IEEE Trans. Geosci. Remote Sens. 99, 1–16 (2020)
Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets’’. Neural Inform. Process. Syst. 2, 2672–2680 (2014)
Kim, J., Lee, J.K., Lee, K.M.: “Accurate image super-resolution using very deep convolutional networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: “Residual dense network for image super-resolution,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision-ECCV, pp. 184–199. Springer International Publishing, Cham (2014)
Xue, L., Shen, J., Wang, R., Yang, J.: Mffn: multi-path feedback fusion network for lightweight image super resolution. IET Image Process. 17, 4190–201 (2023)
Liu, Y., Yue, M., Yan, H., Zhu, L.: Single-image super-resolution using lightweight transformer-convolutional neural network hybrid model. IET Image Process. 17, 2881–93 (2023)
Liu, H., Guo, H., Liu, X.: Uha-cyclegan: unpaired hybrid attention network based on cyclegan for terahertz image super-resolution’’. IET Image Process. 17, 2547–59 (2023)
Li, Y., Hu, J., Zhao, X., Xie, W., Li, J.: Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing 266, 29–41 (2017)
Li, Y., Zhang, L., Dingl, C., Wei, W., Zhang, Y.: “Single hyperspectral image super-resolution with grouped deep recursive residual network,” in 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), 2018
Shaohui, M., Xin, Y., Jingyu, J., Yifan, Z., Shuai, W., Qian, D.: Hyperspectral image spatial super-resolution via 3d full convolutional neural network. Remote Sens. 9(11), 1139 (2017)
Wang, X., Hu, Q., Jiang, J., Ma, J.: A group-based embedding learning and integration network for hyperspectral image super-resolution. IEEE Trans. Geosci. Remote Sens. 60, 1–16 (2022)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N.: Kaiser, L.u., Polosukhin, I., “Attention is all you need. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30. Minus Curran Associates Inc, Berlin (2017)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold G., Gelly, S., Uszkoreit, J., Houlsby, N.: “An image is worth 16x16 words: Transformers for image recognition at scale,” CoRR, vol. abs/2010.11929, 2020
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 568–578 October (2021)
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 10012–10022, October (2021)
Ding, M., Xiao, B., Codella, N., Luo, P., Wang, J., Yuan, L.: “Davit: Dual attention vision transformers,” in Computer Vision - ECCV,: S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner. Eds. minus Cham: Springer Nature Switzerland 2022, 74–92 (2022)
Chen, Q., Wu, Q., Wang, J., Hu, Q., Hu, T., Ding, E., Cheng, J., Wang, J.: Mixformer: Mixing features across windows and dimensions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 5249–5259, June (2022)
Kang, B., Chen, X., Wang, D., Peng, H., Lu, H.: “Exploring lightweight hierarchical vision transformers for efficient visual tracking,” 2023
Choi, H., Na, C., Oh, J., Lee, S., Kim, J., Choe, S., Lee, J., Kim, T., Yang, J.: “Ramit: Reciprocal attention mixing transformer for lightweight image restoration,” 2023
Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., Li, Y.: Maxvit Multi-axis vision transformer. In: Avidan, S., Brostow, G., Cisse, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision-ECCv, pp. 459–479. Springer, Cham (2022)
Wang, W., Yao, L., Chen, L., Cai, D., He, X., Liu, W.: “Crossformer: A versatile vision transformer based on cross-scale attention,” CoRR, vol. abs/2108.00154, 2021
Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., Guo, B.: Cswin transformer: A general vision transformer backbone with cross-shaped windows. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12124–12134,(2022)
Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: “Swinir: Image restoration using swin transformer,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, Oct 2021, 1833–1844
Hu, J.-F., Huang, T.-Z., Deng, L.-J., Dou, H.-X., Hong, D., Vivone, G.: Fusformer: A transformer-based fusion network for hyperspectral image super-resolution. IEEE Geosci. Remote Sens. Lett. 19, 1–5 (2022)
Ma, Q., Jiang, J., Liu, X., Ma, J.: Learning a 3d-cnn and transformer prior for hyperspectral image super-resolution. Inform. Fusion 100, 101907 (2023)
Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Yuhas, R.H., Goetz, A.F.H., Boardman, J.W.: “Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm,” JPL, Summaries of the Third Annual JPL Airborne Geoscience Workshop. Volume 1: AVIRIS Workshop, 1992
Loncan, L., de Almeida, L.B., Bioucas-Dias, J.M., Briottet, X., Chanussot, J., Dobigeon, N., Fabre, S., Liao, W., Licciardi, G.A., Simões, M., Tourneret, J.-Y., Veganzones, M.A., Vivone, G., Wei, Q., Yokoya, N.: Hyperspectral pansharpening: a review. IEEE Geosci. Remote Sens. Mag. 3(3), 27–46 (2015)
Jiang, J., Sun, H., Liu, X., Ma, J.: Learning spatial-spectral prior for super-resolution of hyperspectral imagery. IEEE Trans. Computat. Imaging 6, 1082–96 (2020)
Shi, J., Li, H., Liu, T., Liu, Y., Zhang, M., Zhu, J., Zheng, L., Weng, S.: “Image super-resolution using efficient striped window transformer,” (2023)
Chen, X., Wang, X., Zhou, J., Dong, C.: “Activating more pixels in image super-resolution transformer,” arXiv e-prints, 2022
Yasuma, F., Mitsunaga, T., Iso, D., Nayar, S.K.: Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans. Image Process. Publ IEEE Signal Process. Soc. 19(9), 2241 (2010)
Chakrabarti, A., Zickler, T.: “Statistics of Real-World Hyperspectral Images,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2011, pp. 193–200
Yokoya, N., Iwasaki, A.: “Airborne hyperspectral data over chikusei,” Space Application Laboratory, University of Tokyo, Japan, Tech. Rep. SAL-2016-05-27, (2016)
Author information
Authors and Affiliations
Contributions
Huapeng Wu: Conceptualization, Formal analysis, Methodology, Validation, Writing – original draft, Writing – review & editing Hui Xu: Methodology, Validation, Visualization, Writing – review & editing Tianming Zhan: Investigation, Resources, Supervision, Writing – review & editing
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Communicated by Q. Shen.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This work was supported in part by the National Natural Science Foundation of China under Grant 61976117, 62375133, in part by the Qinglan Project, in part by the Key Projects of University Natural Science Fund of Jiangsu Province under Grant 23KJA520009, in part by the Research Project of University Natural Science Fund of Jiangsu Province under Grant 22KJB520002, in part by the Natural Science Foundation of Jiangsu Province under Grant BK20230440, and in part by the Postgraduate Research Practice Innovation Program of Jiangsu Province under Grant KYCX22_2220.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wu, H., Xu, H. & Zhan, T. A novel spatial and spectral transformer network for hyperspectral image super-resolution. Multimedia Systems 30, 165 (2024). https://doi.org/10.1007/s00530-024-01363-3
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s00530-024-01363-3