RAFnet: Recurrent attention fusion network of hyperspectral and multispectral images

doi:10.1016/j.sigpro.2020.107737

Signal Processing

Volume 177, December 2020, 107737

https://doi.org/10.1016/j.sigpro.2020.107737 Get rights and content

Highlights

•
A recurrent attention fusion network (RAFnet) under a variational probabilistic framework is proposed for HS-MS fusion in an unsupervised manner.
•
Two autoencoders with shared generative model are designed to explore the spectral characteristic with a spectral extractor and spatial features with a spatial extractor.
•
A hierarchical RNN is designed to extract the abundant spectral characteristics of hyperspectral image.
•
A self-attention mechanism a relation-attention mechanism is utilized to model long dependencies of spectra sequences, which applied in conjunction with RNN.

Abstract

Hyperspectral imaging can facilitate a better understanding of more knowledge under real scenes, compared with traditional image systems. However, due to hardware limitations, only low resolution hyperspectral (LrHs) and high resolution multispectral (HrMs) images can generally be acquired. This paper proposed a recurrent attention fusion network (RAFnet) under a variational probabilistic generative framework, in order to fuse the LrHs and HrMs images together to generate a high resolution hyperspectral (HrHs) image in an unsupervised manner. In specific, two variational autoencoders are designed to preserve both spectral and spatial information of LrHs and HrMs images, coupled through a shared decoder to generate hyperspectral images. Considering the spectra of each hyperspectral pixel is intrinsically a sequence based data structure, we construct a hierarchical recurrent neural network to extract the abundant spectral information. Moreover, self-attention and relation-attention mechanisms are adopted to capture long temporal dependencies through the spectral domain. The effectiveness and efficiency are evaluated based on several publicly available hyperspectral datasets, compared with many state-of-the-art methods for the unsupervised fusion task.

Introduction

Hyperspectral images consist of continuous narrow spectral bands of a real scene captured by hyperspectral remote sensors, which can facilitate a fine delivery of more detailed information of distinct materials [1], [2], such as minerals in rocks, vegetation, synthetic materials and water. Hyperspectral image (HSI) analysis has become a thriving and active research area in computer vision with a wide range of applications [3], including object recognition and classification [4], [5], tracking [6], environmental monitoring [7], and change detection [8]. Usually, for high spatial resolution, hyperspectral sensor must have a small instantaneous field of view (IFOV) which reduces the signal to noise (SNR) ratio of the images. To improve the SNR, one has to widen the bandwidth for more light to enter, which reduces the spectral resolution [9]. Therefore, due to the hardware limitations of imaging sensor, there always exists an intrinsic tradeoff between spatial and spectral resolution in the images. Hyperspectral images collect hundreds of contiguous bands which provide finer spectral details of different materials, but often suffer from significantly low spatial resolution [10], [11]. On the contrary, although the multispectral images have high spatial resolution, their spectral resolution is relatively low. Images with both high spectral and spatial resolution are highly desirable [12] for better recognition and analysis. A natural way to generate such high resolution hyperspectral (HrHs) images is to fuse low resolution hyperspectral (LrHs) images with high resolution multispectral (HrMs) images, often referred as HS-MS fusion [13]. Hereafter, the resolution we mentioned denotes the spatial resolution particularly.

Many HS-MS image fusion algorithms have been proposed in the last decades [14], [15], [16], [17], [18], [19], [20], [21]. HrHs images can be reconstructed by combining endmembers of LrHs images and abundances of HrMs images. Addressing the HS-MS fusion task based on a linear spectral mixture model has drawn considerable attention due to its sound physical description [22], [23]. Popular approaches demonstrate the fusion procedure through linear factorization by the aid of different prior knowledge and regularizer such as sparse constraint [11], [22], [23], [24], [25]. Notwithstanding the good performances achieved, those models are restricted by the assumption of spectral mixture process, which is actually much more intricate and complex than a linear process [26].

Recently, deep learning based fusion methods have attracted considerable research interests and resulted in promising performance owing to its high non-linearity and good representation ability. Hence, it is appropriate for modeling the complex nonlinear relationship between LrHs and HrMs images in both spatial and spectral domains [27]. Among the typical deep learning models, convolutional neural network (CNN) based models draw much attention due to its great success in image processing, thus CNNs are used to extract data characteristics of LrHs and HrMs images [18], [28], [29]. In addition, deep learning based methods are data-driven and can reconstruct HrHs images through a feedforward propagation really fast during inference time. However, the current CNN based HS-MS fusion methods still have evident drawbacks as they usually use general frameworks for image processing, which lack specific interpretability for the HS-MS fusion task [13]. As expound in [5], the convolutional neural networks neglect the sequence-based data structure of hyperspectral images, leading to information loss. Thus, the abundant spectral information of hyperspectral image need to be further explored by a sequential model to improve the HS-MS fusion, since an HrHS image collected from real scenes always has hundreds of bands in the spectral domain. The popular sequential models including Recurrent neural network (RNNs), especially long short-term memory (LSTM) [30] and gated recurrent neural network (GRU) [31], have been firmly established as efficient approaches in sequence modeling. Another species of popular sequential models are based on transformers [32], [33], [34], whose direct connections between long-distance pairs are baked in attention mechanisms and enable the learning of long-term dependency [32]. To process the long contextual information of text, hierarchical recurrent neural networks are employed for sequential modeling [35], [36]. Inspired by those sequential models, we apply the hierarchical RNNs to model spectra sequences and draw global dependencies by the attention mechanism similar to transformers.

In addition, most of the deep learning based approaches are supervised models [18], [27], [28], [29], requiring the availability of the target HrHs image for training, which is not realistic in reality. In uSDN [37], an unsupervised deep learning network was first proposed to solve the HS-MS fusion problem, which used two autoencoders to extract the spectral basis from LrHs and spatial representations from HrMs, and reconstructed target HrHs through a shared decoder. Nonetheless, the uSDN optimized two autoencoders separately, and may not make full use of the interactions between the LrHs and HrMs images during the fusion. In addition, with fully-connected network, the uSDN ignored the spatial correlation in the spatial domain and sequential spectral structures of hyperspectral images, leading to the lack of strong representation capability.

As we know, deep probabilistic generative models are skilled in revealing the underlying data distribution and modeling diverse knowledge naturally, which have shown excellent unsupervised data expressive ability, such as deep belief networks [38], [39], variational autoencoders [40]. From a probabilistic perspective, the models are encouraged to capture the inside characteristics and facilitate the generation of long sequence with more information [35], [36], with stochastic variations injected at latent space. Inspired by this, we propose a novel variational probabilistic recurrent attention fusion network for unsupervised HS-MS fusion in this paper, called RAFnet. We reveal the underlying spectrum representations of LrHs with a spectral extractor, and explore the corresponding neighborhood in HrMs with a spatial extractor. In order to fully utilize the information of LrHs and HrMs images, the spectral and spatial features extracted from two extractors are fused together and then fed into a probabilistic generative model to reconstruct the target HrHs image.The main contributions of this work can be summarized as follows:

(1)
We present a hierarchical recurrent attention neural network for HS-MS image fusion, which can effectively exploit the abundant spectral characteristics of hyperspectral image. To the best of our knowledge, it’s novel to utilize a sequential model to extract the underlying spectral information for the HS-MS fusion task.
(2)
We design an architecture composed of two recurrent variational autoencoders for representation learning of LrHs and HrMs images in an unsupervised manner, where both the spatial and spectral characteristics are fused together in the underlying latent space for reconstructing the HrHs image.
(3)
Beyond the hierarchical recurrent mechanism, a self-attention mechanism and a relation-attention mechanism are applied to model long dependencies regardless of the distance between spectra, which are used in conjunction with a recurrent neural network.
(4)
With principled probabilistic modeling, the variational RAFnet is optimized jointly by maximizing the lower-bound of variational framework, leading to an efficient inference scalable to large scene.

This paper is organized as follows. Section 2 describes the related algorithms for the HS-MS fusion and elementary knowledge of sequential models. Section 3 formulates the observation models. Section 4 presents the proposed RAFnet. Experimental results and discussions are presented in Section 5, and the conclusion is given in Section 6.

Section snippets

Traditional methods

Several HSI-MSI fusion algorithms have been proposed in the last decades [14], [16], [17], [19], [20], [21]. Utilizing spectral unmixing in HS-MS fusion has been attracting considerable attention due to its straight-forward interpretation of the fusion process. Unmixing-based fusion methods aim at obtaining endmembers of LrHs and abundances from the HrMs image, respectively, under the constraints of relative sensor characteristics. Then the fused HrHs image can be reconstructed as the product

Problem formulation

Let $\bar{X_{l}} \in R^{h \times w \times S}$ denote the acquired LrHs image, with h and w as its height and width in the spatial dimension, and S as its band number in the spectral dimension. $\bar{X_{m}} \in R^{H \times W \times s}$ denotes the available HrMs image of the same scene, with H and W as its height and width in the spatial dimension, and s represents the band number. In general, HrMs image has much higher spatial resolution than LrHs image, i.e., H > h, W > w, and LrHs image has much higher spectral resolution than HrMs image, i.e., S > s.

Proposed RAFnet

The overall simplified architecture of RAFnet is shown in Fig. 1. The whole architecture can be recognized as two variational probabilistic autoencoders for representation learning of LrHs and HrMs images, respectively. It is composed of three parts: LrHs encoder, HrMs encoder (both are inference models) and the shared decoder (generative model), and here we sketch three components briefly. Firstly, the LrHs encoder extract the latent spectral representation Z_l of LrHs image X_l through a

Datasets and experimental setup

(1) Indian Pines: The hyperspectral data was taken over Indian Pine by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) sensor in 1996, with 224 spectral bands in the 0.4-2.5 µm region [1]. The original image covers 512 × 614 pixels with 20 m spatial resolution, and we select a 145 × 145-pixel-size image as the reference image. Following [22], the HrMs data were produced with uniform spectral response functions corresponding to Landsat TM bands 1–5 and 7, which cover the 450–520,

Conclusion

In this paper, we have provided a novel recurrent attention fusion network for the task of unsupervised HS-MS fusion in an end-to-end fashion. We apply a variational hierarchical recurrent network in the spectral extractor to model the intrinsically latent spaces, taking each pixel of hyperspectral images as a sequential data. At the same time, a spatial feature extractor composed of three convolutional layers is utilized to explore the spatial correlations in HrMs images. Further more, the

CRediT authorship contribution statement

Ruiying Lu: Conceptualization, Methodology, Software, Writing - original draft. Bo Chen: Supervision, Project administration, Funding acquisition, Resources. Ziheng Cheng: Investigation, Validation, Data curation. Penghui Wang: Supervision, Funding acquisition, Writing - review & editing.

Declaration of Competing Interest

We declare that the named authors have no conflict of interest, financial or otherwise in connection with the work submitted.

Acknowledgment

Bo Chen is partially supported by the 111 Project (No. B18039), NSFC (61771361) and Shaanxi Innovation Team Project; Penghui Wang is supported in part by NSFC (61701379).

References (62)

N. Akhtar et al.
Sparse spatio-spectral representation for hyperspectral image super-resolution
Proceedings of the European Conference on Computer Vision
(2014)
N. Yokoya et al.
Hyperspectral and multispectral data fusion: a comparative review of the recent literature
IEEE Geosci. Remote Sens. Mag.
(2017)
X. Cao et al.
Hyperspectral image classification with Markov random fields and a convolutional neural network
IEEE Trans. Image Process
(2018)
A. Chakrabarti et al.
Statistics of real-world hyperspectral images
Proceedings of the Computer Vision and Pattern Recognition (CVPR)
(2011)
M. Fauvel et al.
Advances in spectral-spatial classification of hyperspectral images
Proc. IEEE
(2013)
L. Mou et al.
Deep recurrent neural networks for hyperspectral image classification
IEEE Trans. Geosci. Remote Sens.
(2017)
A. Plaza et al.
Foreword to the special issue on spectral unmixing of remotely sensed data
IEEE Trans. Geosci. Remote Sens.
(2017)
L.H. Spangler et al.
A shallow subsurface controlled release facility in Bozeman, Montana, USA, for testing near surface CO₂ detection techniques and transport models
Environ. Earth Sci.
(2010)
H. Kwon et al.
Kernel matched signal detectors for hyperspectral target detection
Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
(2005)
R.C. Patel et al.
Super-resolution of hyperspectral images using compressive sensing based approach
Remote Sens. Spatial Inf. Sci.
(2012)

R. Kawakami et al.

High-resolution hyperspectral imaging via matrix factorization

Proceedings of the CVPR

(2011)

C. Lanaras et al.

Hyperspectral super-resolution by coupled spectral unmixing

Proceedings of the IEEE International Conference on Computer Vision (ICCV)

(2015)

G. Vivone et al.

A critical comparison among pansharpening algorithms

IEEE Trans. Geosci. Remote Sens.

(2015)

Q. Xie et al.

Multispectral and Hyperspectral Image Fusion by Ms/Hs Fusion Net

(2019)

X. Li et al.

Hyperspectral and multispectral image fusion based on band simulation

IEEE Geosci. Remote. Sens. Lett.

(2020)

X. Li et al.

Hyperspectral and multispectral image fusion via nonlocal low-rank tensor approximation and sparse representation

IEEE Trans. Geosci. Remote Sens.

(2020)

R. Dian et al.

Deep hyperspectral image sharpening

IEEE Trans. Neural Networks Learn. Syst.

(2018)

Q.W. Y. Yuan et al.

Hyperspectral and multispectral image fusion using non-convex relaxation low rank and total variation regularization

Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS)

(2020)

Q. Li et al.

Mixed 2d/3d convolutional network for hyperspectral image super-resolution

Remote. Sens.

(2020)

R. Dian et al.

Hyperspectral image super-resolution via non-local sparse tensor factorization

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR

(2017)

R. Dian et al.

Nonlocal sparse tensor factorization for semiblind hyperspectral and multispectral image fusion.

IEEE Trans. Cybern.

(2019)

S. Li et al.

Fusing hyperspectral and multispectral images via coupled sparse tensor factorization

IEEE Trans. Image Process.

(2018)

N. Yokoya et al.

Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion

IEEE Trans. Geosci. Remote Sens.

(2012)

X.X. Zhu et al.

Exploiting joint sparsity for pansharpening: the j-sparsefi algorithm

IEEE Trans. Geosci. Remote Sens.

(2016)

N. Akhtar et al.

Bayesian sparse representation for hyperspectral image super resolution

Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR)

(2015)

M. Simoes et al.

A convex formulation for hyperspectral image superresolution via subspace-based regularization

IEEE Trans. Geosci. Remote Sens.

(2015)

R. Heylen et al.

A review of nonlinear hyperspectral unmixing methods

IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens.

(2014)

J. Yang et al.

Hyperspectral and multispectral image fusion via deep two-Branches convolutional neural network

Remote Sens. (Basel)

(2018)

F. Palsson et al.

Multispectral and hyperspectral image fusion using a 3-d-convolutional neural network

IEEE Geosci. Remote Sens. Lett.

(2017)

J. Kim et al.

Accurate image super-resolution using very deep convolutional networks

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

(2016)

S. Hochreiter et al.

Long short-term memory

Neural Comput.

(1997)

Cited by (18)

Multispectral and hyperspectral image fusion based on low-rank unfolding network
2023, Signal Processing
Recently, deep unfolding networks (DUNs) have been applied to the fusion of low spatial resolution hyperspectral (LR HS) and high spatial resolution multispectral (HR MS) images and achieved satisfactory high spatial resolution hyperspectral (HR HS) images. However, the low-rank and sparse priors in these networks are not exploited sufficiently. In this paper, we establish an LR HS and HR MS image fusion model based on robust principal component analysis (RPCA), which simultaneously captures the low-rank and sparse properties in HS images. Then, the fusion model is optimized with the alternating direction method of multipliers (ADMM). To make full use of the representation capacity of DUNs, we unfold the derived ADMM algorithm as a network, named low-rank unfolding network (LRU-Net). Specifically, each iteration in ADMM is unfolded as one stage in LRU-Net, in which the low-rank and sparse priors are learned by singular value thresholding (SVT) and sparse module, respectively. Finally, all features from all stages are integrated to produce the desired HR HS image. Three benchmark datasets were chosen for comparison to demonstrate the effectiveness of the proposed LRU-Net. The experimental results demonstrate that LRU-Net performs better in terms of both qualitative and quantitative results compared to state-of-the-art fusion methods. The source code is publicly available at https://github.com/RSMagneto/LRU-Net.
Quaternion convolutional neural networks for hyperspectral image classification
2023, Engineering Applications of Artificial Intelligence
Quaternion convolutional neural networks (QCNNs) can capture quaternion features, which contain not only the contextual information among quaternion feature units but also utilize the quaternion algebra inside the quaternion feature units to express structural information. However, building efficient QCNNs for hyperspectral image (HSI) classification is a challenge due to the lack of methods to map real features into quaternion features and the missing key quaternion modules. This paper proposes methodologies to build QCNNs specifically for HSI classification and designs several key quaternion modules. Firstly, a novel quaternion feature encoder is presented to map real HSI features into quaternion features. Secondly, a paradigm is designed to conveniently modify classical convolutional neural networks to implement QCNNs for HSI classification. Thirdly, we propose a new separable quaternion convolutional neural network (SQNet) based on presented advanced quaternion modules, such as separable quaternion convolutions and quaternion attention mechanisms. In addition, the computational rules in quaternion neurons are improved to enhance the adaptability of QCNNs. Experiments conducted on four widely used HSI datasets indicate that the implemented QCNNs and the proposed SQNet achieve satisfactory results for HSI classification with limited training samples.
Open-source mobile multispectral imaging system and its applications in biological sample sensing
2022, Spectrochimica Acta - Part A: Molecular and Biomolecular Spectroscopy
Citation Excerpt :
Relying on high-performance photoelectric instruments and highly intelligent and automatic image-processing algorithms, machine vision has been widely used in scientific research and industrial settings. Additionally, there is an increasing interest in studying hyperspectral/multispectral cameras for biological sensing and identification[1–4]. Compared with the monochrome or RGB cameras used in traditional machine vision, hyperspectral/multispectral cameras can synchronously obtain the spatial optical information as well as the spectral data, which serves as an important link between biomolecular chemical properties and quantitative data[5,6].
Visible-near-infrared spectroscopy data can be utilized as an important quantitative indicator of biomolecular quantitative analysis. When acquiring spectral information, hyperspectral/multispectral imaging systems can obtain the spatial information of the object of interest. This allows the complete spatial–spectral information of the object of interest to be acquired and the spatial distribution of biomolecules to be analyzed. In this study, we present an open-source mobile multispectral imaging system, test the influence of the utilization of LEDs on the multispectral image, and design image-processing algorithms to correct this influence. To demonstrate the effectiveness of the system, the system is applied to meat freshness analysis, small-animal tumor in-vivo imaging, and chlorophyll spatial distribution imaging. The experimental results verify that our system has stable performance and is compatible with a wide range of spectral imaging applications.
Multi-scale spatial-spectral attention network for multispectral image compression based on variational autoencoder
2022, Signal Processing
Based upon the fact that multispectral image compression needs to remove both spatial and spectral redundancy, recent learnt models via end-to-end manners have shown promising performance. However, most of them ignore the characteristics of multispectral image, i.e., the non-stationarity of spectral correlation and the scale-diversity of spatial features. Meanwhile, they directly utilize fully factorized entropy model, rendering compression performance suboptimal. This paper proposes a Multi-Scale Spatial-Spectral Attention Network (MSSSA-Net) based on variational autoencoder (VAE). Our MSSSA-Net (1) incorporates a simple neuroscience-based non-local attention module into attention mechanism to capture the tiny features in adjacent pixels and large-scale features in spatial domain simultaneously, (2) proposes a multi-scale spectral attention block to extract non-stationary correlation of adjacent spectra at different scales. We demonstrate that our MSSSA-Net offers the state-of-the-art performance in comparison with classical algorithms, including JPEG2000 and 3D-SPIHT, and recent learnt image compression models, on 7-band and 8-band datasets from Landsat-8 and WorldView-3 satellites, when measured by PSNR, MS-SSIM and Mean Spectral Angle. Extensive ablation experiments have verified the effectiveness of each component, and have demonstrated that, for multispectral image compression, Scale-only Hyperprior can make a better trade-off between compression performance and complexity compared with Mean & Scale Hyperprior and Joint Autoregressive model.
Deep learning in multimodal remote sensing data fusion: A comprehensive review
2022, International Journal of Applied Earth Observation and Geoinformation
With the extremely rapid advances in remote sensing (RS) technology, a great quantity of Earth observation (EO) data featuring considerable and complicated heterogeneity are readily available nowadays, which renders researchers an opportunity to tackle current geoscience applications in a fresh way. With the joint utilization of EO data, much research on multimodal RS data fusion has made tremendous progress in recent years, yet these developed traditional algorithms inevitably meet the performance bottleneck due to the lack of the ability to comprehensively analyze and interpret strongly heterogeneous data. Hence, this non-negligible limitation further arouses an intense demand for an alternative tool with powerful processing competence. Deep learning (DL), as a cutting-edge technology, has witnessed remarkable breakthroughs in numerous computer vision tasks owing to its impressive ability in data representation and reconstruction. Naturally, it has been successfully applied to the field of multimodal RS data fusion, yielding great improvement compared with traditional methods. This survey aims to present a systematic overview in DL-based multimodal RS data fusion. More specifically, some essential knowledge about this topic is first given. Subsequently, a literature survey is conducted to analyze the trends of this field. Some prevalent sub-fields in the multimodal RS data fusion are then reviewed in terms of the to-be-fused data modalities, i.e., spatiospectral, spatiotemporal, light detection and ranging-optical, synthetic aperture radar-optical, and RS-Geospatial Big Data fusion. Furthermore, We collect and summarize some valuable resources for the sake of the development in multimodal RS data fusion. Finally, the remaining challenges and potential future directions are highlighted.
A combination method of stacked autoencoder and 3D deep residual network for hyperspectral image classification
2021, International Journal of Applied Earth Observation and Geoinformation
Citation Excerpt :
The OA of deep learning-based classification methods CNN and VAE was 87.92% and 96.36%, respectively, and it was improved by 11.05% and 2.61%, respectively, based on the SAE-3DDRN method. The SSRN structure is more complex that requires two modules to respectively extract spectral and spatial features (Lu et al., 2020; Zhu et al., 2021). There is a high cost for structural design and poor scalability for different HSIs.
In comparison with conventional machine learning algorithms, deep learning can effectively express the deep features of remote sensing images. Considering the rich spectral and spatial information contained in hyperspectral images (HSIs), a combination method was proposed for HSI classification based on stacked autoencoder (SAE) and 3D deep residual network (3DDRN). Specifically, a SAE neural network was first built to reduce the dimensions of original HSIs. A 3D convolutional neural network (3DCNN) was then designed and the residual network module was introduced to build a 3DDRN. The dimension-reduced 3D HSI cubes were input into the 3DDRN to extract identifiable joint spectral-spatial features. Finally, the deep features continuously identified by the 3DDRN were input to Softmax classification layer to realize the classification. In addition, Batch Normalization (BN) and Dropout were used during the learning process to avoid overfitting on training data. The training and test sets of Indian Pines (IP), Pavia University (PU) and Salinas (SA) hyperspectral data sets were selected as the modeling and verification data sources. Six classical classification algorithms were adopted for comparing our proposed method, specifically including conventional machine learning algorithms of Radial Basis Function-Support Vector Machine (RBF-SVM), Kernel Simultaneous Orthogonal Matching Pursuit (KSOMP) and Local Binary Pattern-K-Nearest Neighbor (LBP-KNN), and mainstream deep learning algorithms of Variational Autoencoder (VAE), Convolutional Neural Network (CNN) and Spectral-Spatial Residual Network (SSRN). The results showed that the overall accuracy (OA) reached 98.97%, 99.69% and 99.24%, respectively, only based on 10%, 5% and 1% of training samples for IP, PU and SA. Consequently, the proposed method shows a better classification performance, even in the case of limited samples.

View all citing articles on Scopus

View full text

RAFnet: Recurrent attention fusion network of hyperspectral and multispectral images

Highlights

Abstract

Introduction

Section snippets

Traditional methods

Problem formulation

Proposed RAFnet

Datasets and experimental setup

Conclusion

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgment

Hyperspectral and multispectral data fusion: a comparative review of the recent literature

IEEE Geosci. Remote Sens. Mag.

Hyperspectral image classification with Markov random fields and a convolutional neural network

IEEE Trans. Image Process

Statistics of real-world hyperspectral images

Proceedings of the Computer Vision and Pattern Recognition (CVPR)

Advances in spectral-spatial classification of hyperspectral images

Proc. IEEE

Deep recurrent neural networks for hyperspectral image classification

IEEE Trans. Geosci. Remote Sens.

Foreword to the special issue on spectral unmixing of remotely sensed data

IEEE Trans. Geosci. Remote Sens.

A shallow subsurface controlled release facility in Bozeman, Montana, USA, for testing near surface CO2 detection techniques and transport models

Environ. Earth Sci.

Kernel matched signal detectors for hyperspectral target detection

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Super-resolution of hyperspectral images using compressive sensing based approach

Remote Sens. Spatial Inf. Sci.

High-resolution hyperspectral imaging via matrix factorization

Proceedings of the CVPR

Hyperspectral super-resolution by coupled spectral unmixing

Proceedings of the IEEE International Conference on Computer Vision (ICCV)

A critical comparison among pansharpening algorithms

IEEE Trans. Geosci. Remote Sens.

Multispectral and Hyperspectral Image Fusion by Ms/Hs Fusion Net

Hyperspectral and multispectral image fusion based on band simulation

IEEE Geosci. Remote. Sens. Lett.

Hyperspectral and multispectral image fusion via nonlocal low-rank tensor approximation and sparse representation

IEEE Trans. Geosci. Remote Sens.

Deep hyperspectral image sharpening

IEEE Trans. Neural Networks Learn. Syst.

Hyperspectral and multispectral image fusion using non-convex relaxation low rank and total variation regularization

Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS)

Mixed 2d/3d convolutional network for hyperspectral image super-resolution

Remote. Sens.

Hyperspectral image super-resolution via non-local sparse tensor factorization

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR

Nonlocal sparse tensor factorization for semiblind hyperspectral and multispectral image fusion.

IEEE Trans. Cybern.

Fusing hyperspectral and multispectral images via coupled sparse tensor factorization

IEEE Trans. Image Process.

Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion

IEEE Trans. Geosci. Remote Sens.

Exploiting joint sparsity for pansharpening: the j-sparsefi algorithm

IEEE Trans. Geosci. Remote Sens.

Bayesian sparse representation for hyperspectral image super resolution

Proceedings of the IEEE Computer Vision and Pattern Recognition (CVPR)

A convex formulation for hyperspectral image superresolution via subspace-based regularization

IEEE Trans. Geosci. Remote Sens.

A review of nonlinear hyperspectral unmixing methods

IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens.

Hyperspectral and multispectral image fusion via deep two-Branches convolutional neural network

Remote Sens. (Basel)

Multispectral and hyperspectral image fusion using a 3-d-convolutional neural network

IEEE Geosci. Remote Sens. Lett.

Accurate image super-resolution using very deep convolutional networks

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Long short-term memory

Neural Comput.

A shallow subsurface controlled release facility in Bozeman, Montana, USA, for testing near surface CO₂ detection techniques and transport models