Zhe Liu
Skip Abstract Section
Abstract
Reconstructing a high-resolution hyperspectral (HR-HS) image via merging a low-resolution hyperspectral (LR-HS) image and a high-resolution RGB (HR-RGB) image has become a hot research topic, and can greatly benefit for different subsequent high-level vision tasks. Recently, deep learning–based approaches have evolved for HS image reconstruction and validated impressive performance. However, to learn a good reconstruction model in the deep learning–based methods, it is mandatory to previously collect large-scale training triplets consisting of the LR-HS, HR-RGB, and HR-HS images, which is difficult to be collected in real applications. This study proposes a deep self-supervised HS image reconstruction framework (DSSH), which does not have to depend on any handcrafted prior and previously collected training triplets at all. The proposed DSSH method leverages the designed network architecture itself for capturing the prior of the underlying structure in the latent HR-HS image and employs the observed LR-HS and HR-RGB images only for network parameter learning. Experiments on two benchmark HS image datasets validated that the proposed DSSH method manifests very impressive reconstruction performance, and is even better than some state-of-the-art supervised learning approaches.
- [1] . 2014. Sparse spatio-spectral representation for hyperspectral image super-resolution. In European Conference on Computer Vision. Springer, 63–78.Google Scholar
Cross Ref
- [2] . 2015. Bayesian sparse representation for hyperspectral image super resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3631–3640.Google ScholarCross Ref
- [3] . 2016. Hierarchical beta process with Gaussian process prior for hyperspectral image super resolution. In European Conference on Computer Vision. Springer, 103–120.Google ScholarCross Ref
- [4] . 2018. Systems and methods for hyperspectral medical imaging using real-time projection of spectral information. (
Feb. 2018).US Patent No. 9,883,833. Filed May 13, 2009. Issued Feb. 6, 2018.Google Scholar - [5] . 2013. Hyperspectral remote sensing data analysis and future challenges. IEEE Geoscience and Remote Sensing Magazine 1, 2 (2013), 6–36.Google ScholarCross Ref
- [6] . 2011. Hyperspectral remote sensing for mineral exploration in Pulang, Yunnan Province, China. International Journal of Remote Sensing 32, 9 (2011), 2409–2426.Google ScholarDigital Library
- [7] . 2011. Statistics of real-world hyperspectral images. In CVPR 2011. IEEE, 193–200.Google ScholarDigital Library
- [8] . 2006. Self-supervised monocular road detection in desert terrain. In Robotics: Science and Systems, Vol. 38.Google Scholar
- [9] . 2018. Deep hyperspectral image sharpening. IEEE Transactions on Neural Networks and Learning Systems99 (2018), 1–11.Google Scholar
- [10] . 2016. Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Transactions on Image Processing 25, 5 (2016), 2337–2352.Google ScholarDigital Library
- [11] . 2019. Hyperspectral image super-resolution with optimized RGB guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11661–11670.Google ScholarCross Ref
- [12] . 2019. Deep residual network of spectral and spatial fusion for hyperspectral image super-resolution. In 2019 IEEE 5th International Conference on Multimedia Big Data (BigMM’19). IEEE, 266–270.Google ScholarCross Ref
- [13] . 2018. Self-similarity constrained sparse representation for hyperspectral image super-resolution. IEEE Transactions on Image Processing 27, 11 (2018), 5625–5637.Google ScholarDigital Library
- [14] . 2018. SSF-CNN: Spatial and spectral fusion with CNN for hyperspectral image super-resolution. In 2018 25th IEEE International Conference on Image Processing (ICIP’18). IEEE, 2506–2510.Google ScholarCross Ref
- [15] . 2019. Residual component estimating CNN for image super-resolution. In 2019 IEEE 5th International Conference on Multimedia Big Data (BigMM’19). IEEE, 443–447.Google ScholarCross Ref
- [16] . 2019. Multi-level and multi-scale spatial and spectral fusion CNN for hyperspectral image super-resolution. In Proceedings of the IEEE International Conference on Computer Vision Workshops.Google ScholarCross Ref
- [17] . 2015. Total-variation-regularized low-rank matrix factorization for hyperspectral image restoration. IEEE Transactions on Geoscience and Remote Sensing 54, 1 (2015), 178–188.Google ScholarCross Ref
- [18] . 2017. Generative adversarial networks-based semi-supervised learning for hyperspectral image classification. Remote Sensing 9, 10 (2017), 1042.Google ScholarCross Ref
- [19] . 2019. Self-supervised hyperspectral image restoration using separable image prior. arXiv:1907.00651. Google ScholarCross Ref
- [20] . 2019. Zero-shot hyperspectral image denoising with separable image prior. In Proceedings of the IEEE International Conference on Computer Vision Workshops.Google ScholarCross Ref
- [21] . 2011. High-resolution hyperspectral imaging via matrix factorization. In CVPR 2011. IEEE, 2329–2336.Google ScholarDigital Library
- [22] . 2007. Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis. Bioinformatics 23, 12 (2007), 1495–1502.Google ScholarCross Ref
- [23] . 2017. Resolution enhancement for hyperspectral images: A super-resolution and fusion approach. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’17). IEEE, 6180–6184.Google ScholarDigital Library
- [24] . 2000. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. (
Jan. 2000).US Patent 6,011,875. Filed Apr. 29, 1998. Issued Jan. 4, 2000.Google Scholar - [25] . 2015. Hyperspectral super-resolution by coupled spectral unmixing. In Proceedings of the IEEE International Conference on Computer Vision. 3586–3594.Google ScholarDigital Library
- [26] . 2017. Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing 266 (2017), 29–41.Google ScholarCross Ref
- [27] . 2020. Unsupervised multispectral and hyperspectral image fusion with deep spatial and spectral priors. In Proceedings of the Asian Conference on Computer Vision.Google Scholar
- [28] . 2004. Classification of hyperspectral remote sensing images with support vector machines. IEEE Transactions on Geoscience and Remote Sensing 42, 8 (2004), 1778–1790.Google ScholarCross Ref
- [29] . 2018. Unsupervised sparse Dirichlet-Net for hyperspectral image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2511–2520.Google ScholarCross Ref
- [30] . 2019. Deep hyperspectral prior: Denoising, inpainting, super-resolution. arXiv:1902.00301. Google ScholarCross Ref
- [31] . 2019. Deep hyperspectral prior: Denoising, inpainting, super-resolution. CoRR abs/1902.00301 (2019). arXiv:1902.00301. http://arxiv.org/abs/1902.00301.Google Scholar
- [32] . 2015. The effect of dictionary learning algorithms on super-resolution hyperspectral reconstruction. In 2015 XXV International Conference on Information, Communication and Automation Technologies (ICAT’15). IEEE, 1–5.Google ScholarDigital Library
- [33] . 2020. Guided deep decoder: Unsupervised image pair fusion. In European Conference on Computer Vision. Springer, 87–102.Google ScholarDigital Library
- [34] . 2018. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9446–9454.Google Scholar
- [35] . 2004. A hyperspectral imaging system for in vivo optical diagnostics. IEEE Engineering in Medicine and Biology Magazine 23, 5 (2004), 40–49.Google ScholarCross Ref
- [36] . 2019. Hyperspectral image reconstruction using a deep spatial-spectral prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8032–8041.Google ScholarCross Ref
- [37] . 2013. A non-negative sparse promoting algorithm for high resolution hyperspectral imaging. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 1409–1413.Google ScholarCross Ref
- [38] . 2019. Multispectral and hyperspectral image fusion by MS/HS fusion net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1585–1594.Google ScholarCross Ref
- [39] . 2017. Comparison of hyperspectral imaging and computer vision for automatic differentiation of organically and conventionally farmed salmon. Journal of Food Engineering 196 (2017), 170–182.Google ScholarCross Ref
- [40] . 2010. Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum. IEEE Transactions on Image Processing 19, 9 (2010), 2241–2253.Google ScholarDigital Library
- [41] . 2011. Coupled non-negative matrix factorization (CNMF) for hyperspectral and multispectral data fusion: Application to pasture classification. In 2011 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 1779–1782.Google ScholarCross Ref
- [42] . 2017. Multisensor coupled spectral unmixing for time-series analysis. IEEE Transactions on Geoscience and Remote Sensing 55, 5 (2017), 2842–2857.Google ScholarCross Ref
- [43] . 2012. A super-resolution reconstruction algorithm for hyperspectral images. Signal Processing 92, 9 (2012), 2082–2096.Google ScholarDigital Library
- [44] . 2020. Unsupervised adaptation learning for hyperspectral imagery super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3073–3082.Google ScholarCross Ref
- [45] . 2011. Hyperspectral imagery super-resolution by sparse representation and spectral regularization. EURASIP Journal on Advances in Signal Processing 2011, 1 (2011), 1–10.Google ScholarCross Ref
- [46] . 2020. Residual component estimating CNN for image super-resolution. IEEE Transactions on Image Processing 30 (2020), 1423–1428.Google Scholar
- [47] . 2020. Hyperspectral image super-resolution combining with deep learning and spectral unmixing. Signal Processing: Image Communication 84 (2020), 115833.Google ScholarCross Ref
Index Terms
- Deep Self-Supervised Hyperspectral Image Reconstruction
Recommendations
Self-supervised deep learning for joint 3D low-dose PET/CT image denoising
AbstractDeep learning (DL)-based denoising of low-dose positron emission tomography (LDPET) and low-dose computed tomography (LDCT) has been widely explored. However, previous methods have focused only on single modality denoising, neglecting the ...
Graphical abstractDisplay Omitted
Highlights- A two-stage self-supervised framework named MAsk-then-Cycle (MAC) is proposed to enable self-supervised denoising of LDPET/LDCT images. MAC includes a masked autoencoder-based pre-training stage and a self-supervised denoising training ...
Lightweight Hyperspectral Image Reconstruction Network with Deep Feature Hallucination
Computer Vision – ACCV 2022 WorkshopsAbstractHyperspectral image reconstruction from a compressive snapshot is an dispensable step in the advanced hyperspectral imaging systems to solve the low spatial and/or temporal resolution issue. Most existing methods extensively exploit various hand-...
Hyperspectral Image Reconstruction for SD-CASSI Systems Based on Residual Attention Network
Algorithmic Aspects in Information and ManagementAbstractHyperspectral images contain both spatial and spectral information, which can be utilized to material identification. Therefore, they find significant advantages in object detection. Hyperspectral images are also believed to play an important part ...
Comments