Skip to main content
SpringerLink
Log in

Spatial-Spectral Deep Residual Network for Hyperspectral Image Super-Resolution

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Recently, single hyperspectral image super-resolution (SR) methods based on deep learning have been extensively studied. However, there has been limited technical development focusing on single hyperspectral image super-resolution due to the high-dimensional and complex spectral patterns in hyperspectral image. Besides, most existing methods can not effectively explore spatial information and spectral information of hyperspectral image, obtaining relatively low performance. To address these issues, in this paper, we propose a novel spatial-spectral deep residual network (SSDRN) for hyperspectral image super-resolution. To fully exploit the spectral information and spatial correlation characteristics in hyperspectral data, we use residual group (RG) to extract features effectively, which consists of the proposed spatial-spectral residual attention block (SSRAB). Furthermore, we present a upsampling reconstruction module (UP) to utilize the deep features of previously obtained low-resolution hyperspectral image. Then we could learn the nonlinear mapping relationship between low resolution and high resolution to obtain the desired high-resolution hyperspectral image. Experimental results on two benchmark datasets demonstrate that the proposed approach achieves superior performance in comparison to existing state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

All relevant data are within the paper.

References

  1. Zhou Y, Wei Y. Learning hierarchical spectral-spatial features for hyperspectral image classification. IEEE Trans Cybern. 2015;46(7):1667–78.

    Google Scholar 

  2. Pan Z, Healey G, Prasad M, Tromberg B. Face recognition in hyperspectral images. IEEE Trans Pattern Anal Mach Intell. 2003;25(12):1560.

    Google Scholar 

  3. Bioucas-Dias JM, Plaza A, Camps-Valls G, Scheunders P, Nasrabadi N, Chanussot J. Hyperspectral remote sensing data analysis and future challenges. IEEE Geosci Remote Sens Mag. 2013;1(2):6–36.

    Google Scholar 

  4. Li Q, Wang Q, Li X. An efficient clustering method for hyperspectral optimal band selection via shared nearest neighbor. Remote Sens. 2019;11(3):350.

    Google Scholar 

  5. Wang Q, Member S. GETNET: a general end-to-end two-dimensional CNN framework for hyperspectral image change detection. IEEE Geosci Remote Sens Mag. 2019;57(1):3–13.

    Google Scholar 

  6. Yuan Y, Lin J, Wang Q. Hyperspectral image classification via multitask joint sparse representation and stepwise MRF optimization. IEEE Trans Cybern. 2016;46(12):2966–77.

    Google Scholar 

  7. Xiong F, Zhou J, Qian Y. Material based object tracking in hyperspectral videos. IEEE Trans Image Process. 2020;29:3719–33.

    MATH  Google Scholar 

  8. Xiong F, Zhou J, Tao S, Jianfeng L, Qian Y. SNMF-Net: learning a deep alternating neural network for hyperspectral unmixing. IEEE Trans Geosci Remote Sens. 2022;60:1–16.

    Google Scholar 

  9. Xiong F, Zhou J, Zhao Q, Jianfeng L, Qian Y. MAC-Net: model-aided nonlocal neural network for hyperspectral image denoising. IEEE Trans Geosci Remote Sens. 2022;60:1–14.

    Google Scholar 

  10. Li Z, Xiong F, Zhou J, Wang J, Lu J, Qian Y. BAE-Net: a band attention aware ensemble network for hyperspectral object tracking. In: 2020 IEEE international conference on image processing (ICIP). 2020. p. 2106–2110.

  11. Qian Y, Xiong F, Qian Q, Zhou J. Spectral mixture model inspired network architectures for hyperspectral unmixing. IEEE Trans Geosci Remote Sens. 2020;58(10):7418–34.

    Google Scholar 

  12. Kang X, Duan P, Xiang X, Li S, Benediktsson JA. Detection and correction of mislabeled training samples for hyperspectral image classification. IEEE Trans Geosci Remote Sens. 2018;56(10):5673–86.

    Google Scholar 

  13. Paoletti ME, Haut JM, Plaza J, Plaza A. A new deep convolutional neural network for fast hyperspectral image classification. Isprs J Photogramm Remote Sens. 2018;145PA(NOV.):120–47.

    Google Scholar 

  14. Zhang M, Li W, Du Q. Diverse region-based CNN for hyperspectral image classification. IEEE Trans Image Process. 2018;27(6):2623–34.

    MathSciNet  MATH  Google Scholar 

  15. Park SC, Kang MG, Park MK. Super-resolution image reconstruction: a technical overview. IEEE Signal Process. 2003;20(3):21–36.

    Google Scholar 

  16. Xie W, Jia X, Li Y, Lei J. Hyperspectral image super-resolution using deep feature matrix factorization. IEEE Trans Geosci Remote Sens. 2019;57(9):6055–67.

    Google Scholar 

  17. Dong W, Fazuo F, Shi G, Cao X, Jinjian W, Li G, Li X. Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans Image Process. 2016;25(5):2337–52.

    MathSciNet  MATH  Google Scholar 

  18. Veganzones MA, Simoes M, Licciardi G, Yokoya N, Bioucas-Dias JM, Chanussot J. Hyperspectral super-resolution of locally low rank images from complementary multisource data. IEEE Trans Image Process. 2016;25(1):274–88.

    MathSciNet  MATH  Google Scholar 

  19. Kanatsoulis CI, Xiao F, Sidiropoulos ND, Ma WK. Hyperspectral super-resolution: a coupled tensor factorization approach. IEEE Trans Signal Process. 2018;66(24):6503–17.

    MathSciNet  MATH  Google Scholar 

  20. Wei Q, Dobigeon N, Tourneret J-Y. Bayesian fusion of multi-band images. IEEE J Sel Top Signal Process. 2015;9(6):1117–27.

    MATH  Google Scholar 

  21. Fang L, Zhuo H, Li S. Super-resolution of hyperspectral image via superpixel-based sparse representation. Neurocomputing. 2018;273(17):171–7.

    Google Scholar 

  22. Dian R, Li S, Fang L. Learning a low tensor-train rank representation for hyperspectral image super-resolution. IEEE Trans Neural Netw Learn Syst. 2019;30(9):2672–83.

    MathSciNet  Google Scholar 

  23. Chen C, Li Y, Liu W, Huang J. SIRF: simultaneous satellite image registration and fusion in a unified framework. IEEE Trans Image Process. 2015;24(11):4213–24.

    MathSciNet  MATH  Google Scholar 

  24. Pan Z-W, Shen H-L. Multispectral image super-resolution via RGB image fusion and radiometric calibration. IEEE Trans Image Process. 2018;28(4):1783–97.

    MathSciNet  Google Scholar 

  25. Zhou Y, Rangarajan A, Gader PD. An integrated approach to registration and fusion of hyperspectral and multispectral images. IEEE Trans Geosci Remote Sens. 2020;58(5):3020–33.

    Google Scholar 

  26. Huang H, Jing Y, Sun W. Super-resolution mapping via multi-dictionary based sparse representation. In: 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP). 2014. p. 3523–3527.

  27. He S, Wang Y, Cao W, Zhou H, Han Z. Super-resolution reconstruction of hyperspectral images via low rank tensor modeling and total variation regularization. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). 2016. p. 6962–6965.

  28. Wang Y, Chen XA, Han Z, He S. Hyperspectral image super-resolution via nonlocal low-rank tensor approximation and total variation regularization. Remote Sens. 2017;9(12):1286.

    Google Scholar 

  29. Irmak H, Akar GB, Yuksel SE. A map-based approach for hyperspectral imagery super-resolution. IEEE Trans Image Process. 2018;27(6):2942–51.

    MathSciNet  MATH  Google Scholar 

  30. Hu J-F, Huang T-Z, Deng L-J, Jiang T-X, Vivone G, Chanussot J. Hyperspectral image super-resolution via deep spatiospectral attention convolutional neural networks. IEEE Trans Neural Netw Learn Syst. 2021;33(12):7251–65.

    Google Scholar 

  31. Ren S, He K, Girshick R, Jian S. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell. 2015;39(6).

  32. Chao D, Chen CL, He K, Tang X. Learning a deep convolutional network for image super-resolution. In: European conference on computer vision. 2014. p. 184–199.

  33. Lim B, Sanghyun S, Heewon K, Seungjun N, Kyoung ML. Enhanced deep residual networks for single image super-resolution. In: IEEE conference on computer vision & pattern recognition. 2017. p. 1132–1140.

  34. Ledig C, Theis L, Huszar F, Caballero J, Shi W. Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE conference on computer vision & pattern recognition. 2017. p. 4681–4690.

  35. Ying T, Jian Y, Liu X. Image super-resolution via deep recursive residual network. In: IEEE conference on computer vision & pattern recognition. 2017. p. 3147–3155.

  36. Zhang Y, Li K, Wang L, Zhong B, Li K, Fu Y. Image super-resolution using very deep residual channel attention networks. In: European conference on computer vision. 2018. p. 286–301.

  37. Li Y, Ding C, Wei W, Zhang L, Zhang Y. Single hyperspectral image super-resolution with grouped deep recursive residual network. In: IEEE international conference on multimedia big data. 2018. p. 1–4.

  38. Mei S, Zhang Y, Wan S, Ji J, Yuan X, Du Q. Hyperspectral image spatial super-resolution via 3d full convolutional neural network. Remote Sens. 2017;9:1139.

    Google Scholar 

  39. Yokoya N, Student Member IEEE, Yairi T, Iwasaki A. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Trans Geosci Remote Sens. 2011;50(2):528–37.

    Google Scholar 

  40. Kawakami R, Matsushita Y, Wright J, Ben-Ezra M, Tai Y-W, Ikeuchi K. High-resolution hyperspectral imaging via matrix factorization. In: IEEE conference on computer vision & pattern recognition. 2011. p. 2329–2336.

  41. Lanaras C, Baltsavias E, Schindler K. High-resolution hyperspectral imaging via matrix factorization. In: IEEE conference on computer vision & pattern recognition. 2015. p. 3586–3594.

  42. Wei Q, Bioucas-Dias J, Dobigeon N, Tourneret J-Y, Chen M. Multiband image fusion based on spectral unmixing. IEEE Trans Geosci Remote Sens. 2016;54(12):7236–49.

    Google Scholar 

  43. Akhtar N, Shafait F, Mian A. Sparse spatio-spectral representation for hyperspectral image super-resolution. In: European conference on computer vision. 2014. p. 63–78.

  44. Simoes M, Bioucas-Dias J, Almeida LB, Chanussot J. A convex formulation for hyperspectral image superresolution via subspace-based regularization. IEEE Trans Geosci Remote Sens. 2015;53(6):3373–88.

    Google Scholar 

  45. Wei Q, Bioucas-Dias J, Dobigeon N, Tourneret JY. Hyperspectral and multispectral image fusion based on a sparse representation. IEEE Trans Geosci Remote Sens. 2015;53(7):3658–68.

    Google Scholar 

  46. Wang X, Chen J, Wei Q, Richard C. Hyperspectral image super-resolution via deep prior regularization with parameter estimation. IEEE Trans Circuits Syst Video Technol. 2022;32(4):1708–23.

    Google Scholar 

  47. Dian R, Fang L, Li S. Hyperspectral image super-resolution via non-local sparse tensor factorization. In: IEEE conference on computer vision & pattern recognition. 2017. p. 3862–3871.

  48. Li S, Dian R, Fang L, Bioucas-Dias JM. Fusing hyperspectral and multispectral images via coupled sparse tensor factorization. IEEE Trans Image Process. 2018;27(8):4118–30.

    MathSciNet  MATH  Google Scholar 

  49. Kai Z, Min W, Yang S, Jiao L. Spatial-spectral-graph-regularized low-rank tensor decomposition for multispectral and hyperspectral image fusion. IEEE J Select Top Appl Earth Obs Remote Sens. 2018;11(4):1030–40.

    Google Scholar 

  50. Yang J, Fu X, Hu Y, Yue H, Paisley J. PanNet: a deep network architecture for pan-sharpening. In: IEEE international conference on computer vision (ICCV). 2017. p. 5449–5457.

  51. Dian R, Li S, Guo A, Fang L. Deep hyperspectral image sharpening. IEEE Trans Neural Netw Learn Syst. 2018;29(11):5345–55.

    MathSciNet  Google Scholar 

  52. Qu Y, Qi H, Kwan C. Unsupervised sparse Dirichlet-net for hyperspectral image super-resolution. In: IEEE conference on computer vision & pattern recognition. 2018. p. 2511–2520.

  53. Jin Z-R, Zhang T-J, Jin X, Deng L-J. BAM: bilateral activation mechanism for image fusion. In: Proceedings of the 29th ACM international conference on multimedia. 2021. https://doi.org/10.1145/3474085.3475571.

  54. Akgun T, Altunbasak Y, Mersereau RM. Super-resolution reconstruction of hyperspectral images. IEEE Trans Image Process. 2005;14(11):1860–75.

    Google Scholar 

  55. Bauschke HH, Borwein JM. On projection algorithms for solving convex feasibility problems. SIAM Rev. 1996;38(3):367–426.

    MathSciNet  MATH  Google Scholar 

  56. Yuan Y, Zheng X, Xiaoqiang L. Hyperspectral image superresolution by transfer learning. IEEE J Sel Top Appl Earth Obs Remote Sens. 2017;10(5):1963–74.

    Google Scholar 

  57. Jia J, Zhao Y, Ji L, Geng X. Hyperspectral image super-resolution with spectral spatial network. In: Proc. international journal of remote sensing. 2018. p. 7806–7829.

  58. Chen W, Yun L, Xiao B, Tang W, Peng L, Zhou J. Deep residual convolutional neural network for hyperspectral image super-resolution. In: Proc. international conference on image and graphics. 2017. p. 370–380.

  59. Chen W, Zheng X, Lu X. Hyperspectral image super-resolution with self-supervised spectral-spatial residual network. Remote Sens. 2021;13(7):1260.

    Google Scholar 

  60. Sandler M, Zhu M, Zhmoginov A, Howard A, Chen L-C. Mobilenetv2: inverted residuals and linear bottlenecks. In: IEEE conference on computer vision & pattern recognition. 2018. p. 4510–4520.

  61. Shi W, Caballero J, Huszár F, Totz J, Wang Z. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE conference on computer vision & pattern recognition. 2016. p. 1874–1883.

  62. Chao D, Chen CL, Tang X. Accelerating the super-resolution convolutional neural network. In: European conference on computer vision. 2016. p. 391–407.

  63. Dumoulin V, Shlens J, Kudlur M. A learned representation for artistic style. In: Proc. of ICLR. 2016.

  64. Michal I, Peleg S. Improving resolution by image registration. In: CVGIP: graphical models and image processing. 1991.

  65. Zhao Y, Wang RG, Jia W, Wang WM, Gao W. Iterative projection reconstruction for fast and efficient image upsampling. Neurocomputing. 2017;226(22):200–11.

    Google Scholar 

  66. Li Y, Jing H, Zhao X, Xie W, Li JJ. Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing. 2017;266:29–41.

    Google Scholar 

  67. Anwar S, Khan S, Barnes N. A deep journey into super-resolution: a survey. 2019. arXiv:1904.07523.

  68. Yasuma F, Mitsunaga T, Iso D, Nayar SK. Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans Image Process. 2010;19(9):2241–53.

    MathSciNet  MATH  Google Scholar 

  69. Nascimento SMC, Amano K, Foster DH. Spatial distributions of local illumination color in natural scenes. Vis Res. 2016;120:39–44.

    Google Scholar 

  70. Wang Z. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process. 2004;13:600–12.

    Google Scholar 

  71. Wald L. Data fusion: definitions and architectures: fusion of images of different spatial resolutions. Paris: Presses des MINES; 2002.

    Google Scholar 

Download references

Funding

This work was supported by the Industry University Research Innovation Fund of Chinese Universities (2021ALA02009).

Author information

Authors and Affiliations

  1. Center of Campus Network and Educational Technology, Guangdong University of Finance and Economics, Guangzhou, 510000, China

    WeiFa Zheng

  2. The Department of Electronic Engineering, Shantou University, Shantou, 515063, China

    ZiXin Xie

Authors
  1. WeiFa Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. ZiXin Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to ZiXin Xie.

Ethics declarations

Conflict of interest

WeiFa Zheng declares that he has no conflict of interest. ZiXin Xie declares that he has no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, W., Xie, Z. Spatial-Spectral Deep Residual Network for Hyperspectral Image Super-Resolution. SN COMPUT. SCI. 4, 424 (2023). https://doi.org/10.1007/s42979-023-01868-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-023-01868-0

Keywords

Search

Navigation