Abstract
Existing methods for spectral image reconstruction from low spatial/spectral resolution inputs are typically in discrete manners, only producing results with fixed spatial/spectral resolutions. However, these discrete methods neglect the continuous nature of three-dimensional spectral signals, limiting their applicability and performance. To address this limitation, we propose a novel method leveraging implicit neural representation, which allows for spectral image reconstruction with arbitrary resolutions in both spatial and spectral dimensions for the first time. Specifically, we design neural spatial-spectral representation (NeSSR), which projects the deep features extracted from low-resolution inputs to the corresponding intensity values under target 3D coordinates (including 2D spatial positions and 1D spectral wavelengths). To achieve continuous reconstruction, within NeSSR we devise: a spectral profile interpolation module, which efficiently interpolates features to the desired resolution, and a coordinate-aware neural attention mapping module, which aggregates the coordinate and content information for the final reconstruction. Before NeSSR, we design the spatial-spectral encoder leveraging large-kernel 3D attention, which effectively captures the spatial-spectral correlation in the form of deep features for subsequent high-fidelity representation. Extensive experiments demonstrate the superiority of our method over existing methods across three representative spatial-spectral reconstruction tasks, showcasing its ability to reconstruct spectral images with arbitrary and even extreme spatial/spectral resolutions beyond the training scale.











Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Aeschbacher, J., Wu, J., & Timofte, R. (2017). In defense of shallow learned spectral reconstruction from RGB images. In ICCVW.
Akhtar, N., & Mian, A. (2018). Hyperspectral recovery from RGB images using gaussian processes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(1), 100–113.
Arad, B., & Ben-Shahar, O. (2016). Sparse recovery of hyperspectral signal from natural RGB images. In ECCV.
Arad, B., Timofte, R., Yahel, R., Morag, N., Bernat, A., Cai, Y., Lin, J., Lin, Z., Wang, H., Zhang, Y., et al. (2022). NTIRE 2022 spectral recovery challenge and data set. In CVPRW.
Boss, M., Braun, R., Jampani, V., Barron, J. T., Liu, C., & Lensch, H. P. A. (2021). Nerd: Neural reflectance decomposition from image collections. In ICCV.
Cai, Y., Lin, J., Hu, X., Wang, H., Yuan, X., Zhang, Y., Timofte, R., & Van Gool, L. (2022a). Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction. In CVPR.
Cai, Y., Lin, J., Lin, Z., Wang, H., Zhang, Y., Pfister, H., Timofte, R., & Van Gool, L. (2022b). MST++: Multi-stage spectral-wise transformer for efficient spectral reconstruction. In CVPRW.
Cao, G., Bachega, L. R., & Bouman, C. A. (2010). The sparse matrix transform for covariance estimation and analysis of high dimensional signals. IEEE Transactions on Image Processing, 20(3), 625–640.
Chabra, R., Lenssen, J. E., Ilg, E., Schmidt, T., Straub, J., Lovegrove, S., & Newcombe, R. (2020). Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In ECCV.
Chan, E. R., Monteiro, M., Kellnhofer, P., Wu, J., & Wetzstein, G. (2021). pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. In CVPR.
Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021a). Pre-trained image processing transformer. In CVPR.
Chen, Y., Liu, S., & Wang, X. (2021b). Learning continuous image representation with local implicit image function. In CVPR.
Chen, Y., & Wang, X. (2022). Transformers as meta-learners for implicit neural representations. In ECCV.
Chen, Z., & Zhang, H. (2019). Learning implicit fields for generative shape modeling. In CVPR.
Dao, P. D., Mantripragada, K., He, Y., & Qureshi, F. Z. (2021). Improving hyperspectral image segmentation by applying inverse noise weighting and outlier removal for optimal scale selection. ISPRS Journal of Photogrammetry and Remote Sensing, 171, 348–366.
Dian, R., Fang, L., & Li, S. (2017). Hyperspectral image super-resolution via non-local sparse tensor factorization. In CVPR.
Dian, R., Li, S., & Fang, L. (2019). Learning a low tensor-train rank representation for hyperspectral image super-resolution. IEEE Transactions on Neural Networks and Learning Systems, 30(9), 2672–2683.
Dong, W., Zhou, C., Fangfang, W., Jinjian, W., Shi, G., & Li, X. (2021). Model-guided deep hyperspectral image super-resolution. IEEE Transactions on Image Processing, 30, 5754–5768.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR.
Gao, S., Liu, X., Zeng, B., Xu, S., Li, Y., Luo, X., Liu, J., Zhen, X., & Zhang, B. (2023). Implicit diffusion models for continuous super-resolution. In CVPR.
Gaochang, W., Masia, B., Jarabo, A., Zhang, Y., Wang, L., Dai, Q., Chai, T., & Liu, Y. (2017). Light field image processing: An overview. IEEE Journal of Selected Topics in Signal Processing, 11(7), 926–954.
Goetz, A. F. H., Vane, G., Solomon, J. E., & Rock, B. N. (1985). Imaging spectrometry for earth remote sensing. Science, 228(4704), 1147–1153.
Gowen, A. A., O’Donnell, C. P., Cullen, P. J., Downey, G., & Frias, J. M. (2007). Hyperspectral imaging-an emerging process analytical tool for food quality and safety control. Trends in Food Science & Technology, 18(12), 590–598.
Haboudane, D., Miller, J. R., Pattey, E., Zarco-Tejada, P. J., & Strachan, I. B. (2004). Hyperspectral vegetation indices and novel algorithms for predicting green LAI of crop canopies: Modeling and validation in the context of precision agriculture. Remote Sensing of Environment, 90(3), 337–352.
Hu, X., Cai, Y., Lin, J., Wang, H., Yuan, X., Zhang, Y., Timofte, R., & Van Gool, L. (2022). HDNet: Highresolution dual-domain learning for spectral compressive imaging. In CVPR.
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017). Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR.
Jiang, C., Sud, A., Makadia, A., Huang, J., Nießner, M., Funkhouser, T., et al. (2020). Local implicit grid representations for 3D scenes. In CVPR.
Jiang, K., Xie, W., Lei, J., Jiang, T., & Li, Y. (2021). LREN: Low-rank embedded network for sample-free hyperspectral anomaly detection. In AAAI.
Jin-Fan, H., Huang, T.-Z., Deng, L.-J., Dou, H.-X., Hong, D., & Vivone, G. (2022). Fusformer: A transformer-based fusion network for hyperspectral image super-resolution. IEEE Geoscience and Remote Sensing Letters, 19, 1–5.
Kang, X., Zhang, X., Li, S., Li, K., Li, J., & Benediktsson, J. A. (2017). Hyperspectral anomaly detection with attribute and edge-preserving filters. IEEE Transactions on Geoscience and Remote Sensing, 55(10), 5600–5611.
Kawakami, R., Matsushita, Y., Wright, J., Ben-Ezra, M., Tai, Y.-W., & Ikeuchi, K. (2011). High-resolution hyperspectral imaging via matrix factorization. In CVPR.
Kim, M. H., Harvey, T. A., Kittle, D. S., Rushmeier, H., Dorsey, J., Prum, R. O., & Brady, D. J. (2012). 3D imaging spectroscopy for measuring hyperspectral patterns on solid objects. ACM Transactions on Graphics (TOG), 31(4), 1–11.
Kuybeda, O., Malah, D., & Barzohar, M. (2007). Rank estimation and redundancy reduction of high-dimensional noisy signals with preservation of rare vectors. IEEE Transactions on Signal Processing, 55(12), 5579–5592.
Lanaras, C., Baltsavias, E., & Schindler, K. (2015). Hyperspectral super-resolution by coupled spectral unmixing. In ICCV.
Lee, J., & Jin, K. H. (2022). Local texture estimator for implicit representation function. In CVPR.
Li, J., Wu, C., Song, R., Li, Y., & Liu, F. (2020) Adaptive weighted attention network with camera spectral sensitivity prior for spectral reconstruction from RGB images. In CVPRW.
Li, Q., Gong, M., Yuan, Y., & Wang, Q. (2022). Symmetrical feature propagation network for hyperspectral image super-resolution. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–12.
Lim, B., Son, S., Kim, H., Nah, S., & Mu Lee, K. (2017). Enhanced deep residual networks for single image super-resolution. In CVPRW.
Liu, X., Liu, Q., & Wang, Y. (2020). Remote sensing image fusion based on two-stream fusion network. Information Fusion, 55, 1–15.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021). Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 10012–10022).
Ma, Q., Jiang, J., Liu, X., & Ma, J. (2021). Deep unfolding network for spatiospectral image super-resolution. IEEE Transactions on Computational Imaging, 8, 28–40.
Ma, Q., Jiang, J., Liu, X., & Ma, J. (2022). Multi-task interaction learning for spatiospectral image super-resolution. IEEE Transactions on Image Processing, 31, 2950–2961.
Mei, S., Jiang, R., Xu, L., & Qian, D. (2020). Spatial and spectral joint super-resolution using convolutional neural network. IEEE Transactions on Geoscience and Remote Sensing, 58(7), 4590–4603.
Melgani, F., & Bruzzone, L. (2004). Classification of hyperspectral remote sensing images with support vector machines. IEEE Transactions on Geoscience and Remote Sensing, 42(8), 1778–1790.
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., & Geiger, A. (2019). Occupancy networks: Learning 3D reconstruction in function space. In CVPR.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). NERF: Representing scenes as neural radiance fields for view synthesis. In ECCV.
Niemeyer, M., & Geiger, A. (2021). Giraffe: Representing scenes as compositional generative neural feature fields. In CVPR.
Niemeyer, M., Mescheder, L., Oechsle, M., & Geiger, A. (2020). Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. In CVPR.
Oechsle, M., Mescheder, L., Niemeyer, M., Strauss, T., & Geiger, A. (2019). Texture fields: Learning texture representations in function space. In CVPR.
Ost, J., Mannan, F., Thuerey, N., Knodt, J., & Heide, F. (2021). Neural scene graphs for dynamic scenes. In CVPR.
Robles-Kelly, A. (2015). Single image spectral reconstruction for multimedia applications. In ACM MM.
Shi, Z., Chen, C., Xiong, Z., Liu, D., & Wu, F. (2018a). HSCNN+: Advanced CNN-based hyperspectral recovery from RGB images. In CVPRW.
Shi, Z., Chen, C., Xiong, Z., Liu, D., Zha, Z.-J., & Wu, F. (2018b). Deep residual attention network for spectral image super-resolution. In ECCVW.
Shoeiby, M., Robles-Kelly, A., Timofte, R., Zhou, R., Lahoud, F., Susstrunk, S., Xiong, Z., Shi, Z., Chen, C., Liu, D., et al. (2018). Pirm2018 challenge on spectral image super-resolution: methods and results. In ECCVW.
Sitzmann, V., Chan, E. R., Tucker, R, Snavely, N., & Wetzstein, G. MetaSDF: Meta-learning signed distance functions. In NIPS.
Sitzmann, V., Martel, J., Bergman, A., Lindell, D., & Wetzstein, G. (2020). Implicit neural representations with periodic activation functions. In NIPS.
Sitzmann, V., Zollhöfer, M., & Wetzstein, G. (2019). Scene representation networks: continuous 3D-structure-aware neural scene representations. In NIPS.
Srinivasan, P. P., Deng, B., Zhang, X., Tancik, M., Mildenhall, B., & Barron. J. T. (2021). NERV: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR.
Su, S.-Y., Yu, F., Zollhoefer, M., & Rhodin, H. (2021). A-NERF: Surface-free human 3D pose refinement via neural rendering. arXiv:2102.06199
Sun, B., Yan, J., Zhou, X., & Zheng, Y. (2021). Tuning IR-cut filter for illumination-aware spectral reconstruction from RGB. In CVPR.
Tancik, M., Mildenhall, B., Wang, T., Schmidt, D., Srinivasan, P. P., Barron, J. T., & Ng, R. (2021). Learned initializations for optimizing coordinate-based neural representations. In CVPR.
Timofte, R., De Smet, V., & Van Gool, L. (2014). A+: Adjusted anchored neighborhood regression for fast super-resolution. In ACCV.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) Attention is all you need. In NIPS.
Wang, X., Chen, J., Wei, Q., & Richard, C. (2021). Hyperspectral image super-resolution via deep prior regularization with parameter estimation. IEEE Transactions on Circuits and Systems for Video Technology, 32(4), 1708–1723.
Xiao, Z., Xiong, Z., Fu, X., Liu, D., & Zha, Z.-J. (2020). Space–time video super-resolution using temporal profiles. In ACM MM.
Xie, Q., Zhou, M., Zhao, Q., Meng, D., Zuo, W., & Xu, Z. (2019). Multispectral and hyperspectral image fusion by MS/HS fusion net. In CVPR.
Xiong, F., Zhou, J., & Qian, Y. (2020). Material based object tracking in hyperspectral videos. IEEE Transactions on Image Processing, 29, 3719–3733.
Xiong, Z., Shi, Z., Li, H., Wang, L., Liu, D., & Wu, F. (2017). HSCNN: CNN-based hyperspectral image recovery from spectrally undersampled projections. In CVPRW.
Xu, R., Yao, M., Chen, C., Wang, L., & Xiong, Z. (2022). Continuous spectral reconstruction from RGB images via implicit neural representation. In ECCVW.
Yang, J., Shen, S., Yue, H., & Li, K. (2021). Implicit transformer network for screen content image continuous super-resolution. NIPS.
Yao, M., Xiong, Z., Wang, L., Liu, D., & Chen, X. (2019). Spectral-depth imaging with deep learning based reconstruction. Optics Express, 27(26), 38312–38325.
Yasuma, F., Mitsunaga, T., Iso, D., & Nayar, S. K. (2010). Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum. IEEE Transactions on Image Processing, 19(9), 2241–2253.
Yen-Chen, L., Florence, P., Barron, J. T., Rodriguez, A., Isola, P., & Lin, T.-Y. (2020). INERF: Inverting neural radiance fields for pose estimation. arXiv:2012.05877
Yokoya, N., Yairi, T., & Iwasaki, A. (2011). Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Transactions on Geoscience and Remote Sensing, 50(2), 528–537.
Zamir, S., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M.-H., & Shao, L. (2021) Multistage progressive image restoration. In CVPR.
Zhang, L., Lang, Z., Wang, P., Wei, W., Liao, S., Shao, L., & Zhang, Y. (2020a). Pixel-aware deep function-mixture network for spectral super-resolution. In AAAI.
Zhang, L., Nie, J., Wei, W., Zhang, Y., Liao, S., Shao, L. (2020b). Unsupervised adaptation learning for hyperspectral imagery super-resolution. In CVPR.
Zhang, X., Huang, W., Wang, Q., & Li, X. (2020c). SSR-Net: Spatial-spectral reconstruction network for hyperspectral and multispectral image fusion. IEEE Transactions on Geoscience and Remote Sensing, 59(7), 5953–5965.
Zhang, Y., Li, K., Li, K., Wang, L., Zhong, B., & Fu, Y. (2018). Image super-resolution using very deep residual channel attention networks. In ECCV.
Zhao, Y., Po, L.-M., Yan, Q., Liu, W., & Lin, T. (2020). Hierarchical regression network for spectral reconstruction from RGB images. In CVPRW.
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020a) Deformable DETR: Deformable transformers for end-to-end object detection. In ICLR.
Zhu, Z., Hou, J., Chen, J., Zeng, H., & Zhou, J. (2020b). Hyperspectral image super-resolution via deep progressive zero-centric residual learning. IEEE Transactions on Image Processing, 30, 1423–1438.
Zhu, Z., Liu, H., Hou, J., Zeng, H., & Zhang, Q. (2021). Semantic-embedded unsupervised spectral reconstruction from single RGB images in the wild. In ICCV.
Zuckerman, L P., Naor, E., Pisha, G., Bagon, S., & Irani, M. (2020) Across scales and across dimensions: Temporal super-resolution using deep internal learning. In ECCV.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grants 62131003 and 62021001.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Boxin Shi.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Xu, R., Yao, M., Chen, C. et al. Continuous Spatial-Spectral Reconstruction via Implicit Neural Representation. Int J Comput Vis 133, 106–128 (2025). https://doi.org/10.1007/s11263-024-02150-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-024-02150-3