Skip to main content

Advertisement

Log in

Continuous Spatial-Spectral Reconstruction via Implicit Neural Representation

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Existing methods for spectral image reconstruction from low spatial/spectral resolution inputs are typically in discrete manners, only producing results with fixed spatial/spectral resolutions. However, these discrete methods neglect the continuous nature of three-dimensional spectral signals, limiting their applicability and performance. To address this limitation, we propose a novel method leveraging implicit neural representation, which allows for spectral image reconstruction with arbitrary resolutions in both spatial and spectral dimensions for the first time. Specifically, we design neural spatial-spectral representation (NeSSR), which projects the deep features extracted from low-resolution inputs to the corresponding intensity values under target 3D coordinates (including 2D spatial positions and 1D spectral wavelengths). To achieve continuous reconstruction, within NeSSR we devise: a spectral profile interpolation module, which efficiently interpolates features to the desired resolution, and a coordinate-aware neural attention mapping module, which aggregates the coordinate and content information for the final reconstruction. Before NeSSR, we design the spatial-spectral encoder leveraging large-kernel 3D attention, which effectively captures the spatial-spectral correlation in the form of deep features for subsequent high-fidelity representation. Extensive experiments demonstrate the superiority of our method over existing methods across three representative spatial-spectral reconstruction tasks, showcasing its ability to reconstruct spectral images with arbitrary and even extreme spatial/spectral resolutions beyond the training scale.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Aeschbacher, J., Wu, J., & Timofte, R. (2017). In defense of shallow learned spectral reconstruction from RGB images. In ICCVW.

  • Akhtar, N., & Mian, A. (2018). Hyperspectral recovery from RGB images using gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(1), 100–113.

    Article  MATH  Google Scholar 

  • Arad, B., & Ben-Shahar, O. (2016). Sparse recovery of hyperspectral signal from natural RGB images. In ECCV.

  • Arad, B., Timofte, R., Yahel, R., Morag, N., Bernat, A., Cai, Y., Lin, J., Lin, Z., Wang, H., Zhang, Y., et al. (2022). NTIRE 2022 spectral recovery challenge and data set. In CVPRW.

  • Boss, M., Braun, R., Jampani, V., Barron, J. T., Liu, C., & Lensch, H. P. A. (2021). Nerd: Neural reflectance decomposition from image collections. In ICCV.

  • Cai, Y., Lin, J., Hu, X., Wang, H., Yuan, X., Zhang, Y., Timofte, R., & Van Gool, L. (2022a). Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction. In CVPR.

  • Cai, Y., Lin, J., Lin, Z., Wang, H., Zhang, Y., Pfister, H., Timofte, R., & Van Gool, L. (2022b). MST++: Multi-stage spectral-wise transformer for efficient spectral reconstruction. In CVPRW.

  • Cao, G., Bachega, L. R., & Bouman, C. A. (2010). The sparse matrix transform for covariance estimation and analysis of high dimensional signals. IEEE Transactions on Image Processing, 20(3), 625–640.

    Article  MathSciNet  MATH  Google Scholar 

  • Chabra, R., Lenssen, J. E., Ilg, E., Schmidt, T., Straub, J., Lovegrove, S., & Newcombe, R. (2020). Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In ECCV.

  • Chan, E. R., Monteiro, M., Kellnhofer, P., Wu, J., & Wetzstein, G. (2021). pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. In CVPR.

  • Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021a). Pre-trained image processing transformer. In CVPR.

  • Chen, Y., Liu, S., & Wang, X. (2021b). Learning continuous image representation with local implicit image function. In CVPR.

  • Chen, Y., & Wang, X. (2022). Transformers as meta-learners for implicit neural representations. In ECCV.

  • Chen, Z., & Zhang, H. (2019). Learning implicit fields for generative shape modeling. In CVPR.

  • Dao, P. D., Mantripragada, K., He, Y., & Qureshi, F. Z. (2021). Improving hyperspectral image segmentation by applying inverse noise weighting and outlier removal for optimal scale selection. ISPRS Journal of Photogrammetry and Remote Sensing, 171, 348–366.

    Article  Google Scholar 

  • Dian, R., Fang, L., & Li, S. (2017). Hyperspectral image super-resolution via non-local sparse tensor factorization. In CVPR.

  • Dian, R., Li, S., & Fang, L. (2019). Learning a low tensor-train rank representation for hyperspectral image super-resolution. IEEE Transactions on Neural Networks and Learning Systems, 30(9), 2672–2683.

    Article  MathSciNet  MATH  Google Scholar 

  • Dong, W., Zhou, C., Fangfang, W., Jinjian, W., Shi, G., & Li, X. (2021). Model-guided deep hyperspectral image super-resolution. IEEE Transactions on Image Processing, 30, 5754–5768.

    Article  MATH  Google Scholar 

  • Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR.

  • Gao, S., Liu, X., Zeng, B., Xu, S., Li, Y., Luo, X., Liu, J., Zhen, X., & Zhang, B. (2023). Implicit diffusion models for continuous super-resolution. In CVPR.

  • Gaochang, W., Masia, B., Jarabo, A., Zhang, Y., Wang, L., Dai, Q., Chai, T., & Liu, Y. (2017). Light field image processing: An overview. IEEE Journal of Selected Topics in Signal Processing, 11(7), 926–954.

    Article  MATH  Google Scholar 

  • Goetz, A. F. H., Vane, G., Solomon, J. E., & Rock, B. N. (1985). Imaging spectrometry for earth remote sensing. Science, 228(4704), 1147–1153.

    Article  Google Scholar 

  • Gowen, A. A., O’Donnell, C. P., Cullen, P. J., Downey, G., & Frias, J. M. (2007). Hyperspectral imaging-an emerging process analytical tool for food quality and safety control. Trends in Food Science & Technology, 18(12), 590–598.

    Article  Google Scholar 

  • Haboudane, D., Miller, J. R., Pattey, E., Zarco-Tejada, P. J., & Strachan, I. B. (2004). Hyperspectral vegetation indices and novel algorithms for predicting green LAI of crop canopies: Modeling and validation in the context of precision agriculture. Remote Sensing of Environment, 90(3), 337–352.

    Article  Google Scholar 

  • Hu, X., Cai, Y., Lin, J., Wang, H., Yuan, X., Zhang, Y., Timofte, R., & Van Gool, L. (2022). HDNet: Highresolution dual-domain learning for spectral compressive imaging. In CVPR.

  • Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017). Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR.

  • Jiang, C., Sud, A., Makadia, A., Huang, J., Nießner, M., Funkhouser, T., et al. (2020). Local implicit grid representations for 3D scenes. In CVPR.

  • Jiang, K., Xie, W., Lei, J., Jiang, T., & Li, Y. (2021). LREN: Low-rank embedded network for sample-free hyperspectral anomaly detection. In AAAI.

  • Jin-Fan, H., Huang, T.-Z., Deng, L.-J., Dou, H.-X., Hong, D., & Vivone, G. (2022). Fusformer: A transformer-based fusion network for hyperspectral image super-resolution. IEEE Geoscience and Remote Sensing Letters, 19, 1–5.

    MATH  Google Scholar 

  • Kang, X., Zhang, X., Li, S., Li, K., Li, J., & Benediktsson, J. A. (2017). Hyperspectral anomaly detection with attribute and edge-preserving filters. IEEE Transactions on Geoscience and Remote Sensing, 55(10), 5600–5611.

    Article  MATH  Google Scholar 

  • Kawakami, R., Matsushita, Y., Wright, J., Ben-Ezra, M., Tai, Y.-W., & Ikeuchi, K. (2011). High-resolution hyperspectral imaging via matrix factorization. In CVPR.

  • Kim, M. H., Harvey, T. A., Kittle, D. S., Rushmeier, H., Dorsey, J., Prum, R. O., & Brady, D. J. (2012). 3D imaging spectroscopy for measuring hyperspectral patterns on solid objects. ACM Transactions on Graphics (TOG), 31(4), 1–11.

    Google Scholar 

  • Kuybeda, O., Malah, D., & Barzohar, M. (2007). Rank estimation and redundancy reduction of high-dimensional noisy signals with preservation of rare vectors. IEEE Transactions on Signal Processing, 55(12), 5579–5592.

    Article  MathSciNet  MATH  Google Scholar 

  • Lanaras, C., Baltsavias, E., & Schindler, K. (2015). Hyperspectral super-resolution by coupled spectral unmixing. In ICCV.

  • Lee, J., & Jin, K. H. (2022). Local texture estimator for implicit representation function. In CVPR.

  • Li, J., Wu, C., Song, R., Li, Y., & Liu, F. (2020) Adaptive weighted attention network with camera spectral sensitivity prior for spectral reconstruction from RGB images. In CVPRW.

  • Li, Q., Gong, M., Yuan, Y., & Wang, Q. (2022). Symmetrical feature propagation network for hyperspectral image super-resolution. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–12.

    MATH  Google Scholar 

  • Lim, B., Son, S., Kim, H., Nah, S., & Mu Lee, K. (2017). Enhanced deep residual networks for single image super-resolution. In CVPRW.

  • Liu, X., Liu, Q., & Wang, Y. (2020). Remote sensing image fusion based on two-stream fusion network. Information Fusion, 55, 1–15.

    Article  MATH  Google Scholar 

  • Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 10012–10022).

  • Ma, Q., Jiang, J., Liu, X., & Ma, J. (2021). Deep unfolding network for spatiospectral image super-resolution. IEEE Transactions on Computational Imaging, 8, 28–40.

    Article  MathSciNet  MATH  Google Scholar 

  • Ma, Q., Jiang, J., Liu, X., & Ma, J. (2022). Multi-task interaction learning for spatiospectral image super-resolution. IEEE Transactions on Image Processing, 31, 2950–2961.

    Article  MATH  Google Scholar 

  • Mei, S., Jiang, R., Xu, L., & Qian, D. (2020). Spatial and spectral joint super-resolution using convolutional neural network. IEEE Transactions on Geoscience and Remote Sensing, 58(7), 4590–4603.

    Article  MATH  Google Scholar 

  • Melgani, F., & Bruzzone, L. (2004). Classification of hyperspectral remote sensing images with support vector machines. IEEE Transactions on Geoscience and Remote Sensing, 42(8), 1778–1790.

    Article  MATH  Google Scholar 

  • Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., & Geiger, A. (2019). Occupancy networks: Learning 3D reconstruction in function space. In CVPR.

  • Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). NERF: Representing scenes as neural radiance fields for view synthesis. In ECCV.

  • Niemeyer, M., & Geiger, A. (2021). Giraffe: Representing scenes as compositional generative neural feature fields. In CVPR.

  • Niemeyer, M., Mescheder, L., Oechsle, M., & Geiger, A. (2020). Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In CVPR.

  • Oechsle, M., Mescheder, L., Niemeyer, M., Strauss, T., & Geiger, A. (2019). Texture fields: Learning texture representations in function space. In CVPR.

  • Ost, J., Mannan, F., Thuerey, N., Knodt, J., & Heide, F. (2021). Neural scene graphs for dynamic scenes. In CVPR.

  • Robles-Kelly, A. (2015). Single image spectral reconstruction for multimedia applications. In ACM MM.

  • Shi, Z., Chen, C., Xiong, Z., Liu, D., & Wu, F. (2018a). HSCNN+: Advanced CNN-based hyperspectral recovery from RGB images. In CVPRW.

  • Shi, Z., Chen, C., Xiong, Z., Liu, D., Zha, Z.-J., & Wu, F. (2018b). Deep residual attention network for spectral image super-resolution. In ECCVW.

  • Shoeiby, M., Robles-Kelly, A., Timofte, R., Zhou, R., Lahoud, F., Susstrunk, S., Xiong, Z., Shi, Z., Chen, C., Liu, D., et al. (2018). Pirm2018 challenge on spectral image super-resolution: methods and results. In ECCVW.

  • Sitzmann, V., Chan, E. R., Tucker, R, Snavely, N., & Wetzstein, G. MetaSDF: Meta-learning signed distance functions. In NIPS.

  • Sitzmann, V., Martel, J., Bergman, A., Lindell, D., & Wetzstein, G. (2020). Implicit neural representations with periodic activation functions. In NIPS.

  • Sitzmann, V., Zollhöfer, M., & Wetzstein, G. (2019). Scene representation networks: continuous 3D-structure-aware neural scene representations. In NIPS.

  • Srinivasan, P. P., Deng, B., Zhang, X., Tancik, M., Mildenhall, B., & Barron. J. T. (2021). NERV: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR.

  • Su, S.-Y., Yu, F., Zollhoefer, M., & Rhodin, H. (2021). A-NERF: Surface-free human 3D pose refinement via neural rendering. arXiv:2102.06199

  • Sun, B., Yan, J., Zhou, X., & Zheng, Y. (2021). Tuning IR-cut filter for illumination-aware spectral reconstruction from RGB. In CVPR.

  • Tancik, M., Mildenhall, B., Wang, T., Schmidt, D., Srinivasan, P. P., Barron, J. T., & Ng, R. (2021). Learned initializations for optimizing coordinate-based neural representations. In CVPR.

  • Timofte, R., De Smet, V., & Van Gool, L. (2014). A+: Adjusted anchored neighborhood regression for fast super-resolution. In ACCV.

  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) Attention is all you need. In NIPS.

  • Wang, X., Chen, J., Wei, Q., & Richard, C. (2021). Hyperspectral image super-resolution via deep prior regularization with parameter estimation. IEEE Transactions on Circuits and Systems for Video Technology, 32(4), 1708–1723.

    Article  MATH  Google Scholar 

  • Xiao, Z., Xiong, Z., Fu, X., Liu, D., & Zha, Z.-J. (2020). Space–time video super-resolution using temporal profiles. In ACM MM.

  • Xie, Q., Zhou, M., Zhao, Q., Meng, D., Zuo, W., & Xu, Z. (2019). Multispectral and hyperspectral image fusion by MS/HS fusion net. In CVPR.

  • Xiong, F., Zhou, J., & Qian, Y. (2020). Material based object tracking in hyperspectral videos. IEEE Transactions on Image Processing, 29, 3719–3733.

    Article  MATH  Google Scholar 

  • Xiong, Z., Shi, Z., Li, H., Wang, L., Liu, D., & Wu, F. (2017). HSCNN: CNN-based hyperspectral image recovery from spectrally undersampled projections. In CVPRW.

  • Xu, R., Yao, M., Chen, C., Wang, L., & Xiong, Z. (2022). Continuous spectral reconstruction from RGB images via implicit neural representation. In ECCVW.

  • Yang, J., Shen, S., Yue, H., & Li, K. (2021). Implicit transformer network for screen content image continuous super-resolution. NIPS.

  • Yao, M., Xiong, Z., Wang, L., Liu, D., & Chen, X. (2019). Spectral-depth imaging with deep learning based reconstruction. Optics Express, 27(26), 38312–38325.

    Article  Google Scholar 

  • Yasuma, F., Mitsunaga, T., Iso, D., & Nayar, S. K. (2010). Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum. IEEE Transactions on Image Processing, 19(9), 2241–2253.

    Article  MathSciNet  MATH  Google Scholar 

  • Yen-Chen, L., Florence, P., Barron, J. T., Rodriguez, A., Isola, P., & Lin, T.-Y. (2020). INERF: Inverting neural radiance fields for pose estimation. arXiv:2012.05877

  • Yokoya, N., Yairi, T., & Iwasaki, A. (2011). Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Transactions on Geoscience and Remote Sensing, 50(2), 528–537.

    Article  Google Scholar 

  • Zamir, S., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M.-H., & Shao, L. (2021) Multistage progressive image restoration. In CVPR.

  • Zhang, L., Lang, Z., Wang, P., Wei, W., Liao, S., Shao, L., & Zhang, Y. (2020a). Pixel-aware deep function-mixture network for spectral super-resolution. In AAAI.

  • Zhang, L., Nie, J., Wei, W., Zhang, Y., Liao, S., Shao, L. (2020b). Unsupervised adaptation learning for hyperspectral imagery super-resolution. In CVPR.

  • Zhang, X., Huang, W., Wang, Q., & Li, X. (2020c). SSR-Net: Spatial-spectral reconstruction network for hyperspectral and multispectral image fusion. IEEE Transactions on Geoscience and Remote Sensing, 59(7), 5953–5965.

  • Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., & Fu, Y. (2018). Image super-resolution using very deep residual channel attention networks. In ECCV.

  • Zhao, Y., Po, L.-M., Yan, Q., Liu, W., & Lin, T. (2020). Hierarchical regression network for spectral reconstruction from RGB images. In CVPRW.

  • Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020a) Deformable DETR: Deformable transformers for end-to-end object detection. In ICLR.

  • Zhu, Z., Hou, J., Chen, J., Zeng, H., & Zhou, J. (2020b). Hyperspectral image super-resolution via deep progressive zero-centric residual learning. IEEE Transactions on Image Processing, 30, 1423–1438.

  • Zhu, Z., Liu, H., Hou, J., Zeng, H., & Zhang, Q. (2021). Semantic-embedded unsupervised spectral reconstruction from single RGB images in the wild. In ICCV.

  • Zuckerman, L P., Naor, E., Pisha, G., Bagon, S., & Irani, M. (2020) Across scales and across dimensions: Temporal super-resolution using deep internal learning. In ECCV.

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grants 62131003 and 62021001.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Zhiwei Xiong.

Additional information

Communicated by Boxin Shi.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, R., Yao, M., Chen, C. et al. Continuous Spatial-Spectral Reconstruction via Implicit Neural Representation. Int J Comput Vis 133, 106–128 (2025). https://doi.org/10.1007/s11263-024-02150-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-024-02150-3

Keywords

Navigation