Nonlinear extended blind end-member and abundance extraction for hyperspectral images
Introduction
Hyperspectral images (HSIs) have become a valuable tool to identify qualitative and quantitative mappings of the constitutive elements of a scene, which are indistinguishable in plain sight [1]. The applications of these imaging modalities are quite diverse: mineral exploration, earth observation, seafloor exploration and monitoring, food inspection, biological tissue characterization, artwork investigation, etc. [2], [3], [4], [5], [6], [7], [8], [9], [10]. However, for a proper evaluation of hyperspectral information, a processing stage is required to estimate the spectral signatures of the constitutive elements or end-members, and their fractional contributions or abundances in each pixel of the image [11], [12]. The physical relation between end-members and their abundances is represented by a mathematical model, which describes the optical paths and interactions of the reflected light by the objects in the scene [13].
The basic mathematical model assumes a linear mixing model (LMM), so each measurement is represented by a linear combination of end-members plus an uncertainty/noise component [14], [15]. However, under some conditions, the light captured by the camera shows multipath scattering and nonuniform reflection patterns, so a nonlinear mixture model (NMM) should be pursued. Since late 2000’s and early 2010’s, diverse research groups have proposed NMM’s to highlight different nonlinear physical phenomena: Fan et al. model [16], generalized bilinear model (GBM) [17], polynomial post-nonlinear model (PPNM) [18], and recently multilinear mixing model (MMM) [19]. In all previous NMMs, the model preserves a linear component and includes a nonlinear term depending on the assumed physical interaction. In particular, the MMM shows an evolution of the GBM and PPNMM frameworks to describe all possible optical interactions in the scene. One advantage of the MMM is the addition of a single parameter to quantify the nonlinear phenomena per pixel.
Quantitative characterization of HSIs involves the estimation of end-members and their abundances, and the specific parameters of the NMM [13]. In some approaches, the end-members are assumed known in advance by early studies of the materials in the scene and their spectral signatures. This scenario is called a supervised strategy. On the other hand, if the end-members are unknown, and they have to be jointly estimated, this condition is denoted as an unsupervised approach, or a blind unmixing methodology (BUM) [20].
Recently, the joint application of artificial intelligence (AI) and non-convex modeling in hyperspectral imaging was reviewed in Hong et al. [21], where the main challenges and open problems in image restoration, dimensionality reduction, spectral unmixing, data-fusion and enhancement, and cross-modality learning were discussed. In this way, a deep learning perspective was considered for spectral unmixing in Hong et al. [22], where an end-member guided unmixing network (EGU-NET) was proposed that relies on a two-stream Siamese deep network for blind unmixing. In this work, the deviations of a LMM are denoted as spectral variability and by the structure of the EGU-NET, a nonlinear unmixing of the studied HSI is achieved. A different perspective is followed in Yao et al. [23], where a tensor-based approach was used for spectral unmixing to have a more adequate modeling in both the spatial and spectral domains. The unmixing methodology in Yao et al. [23] is called sparsity-enhanced convolutional decomposition, where the HSI is modeled as a third-order tensor, but no high-order interactions are taken into account in the end-members. Meanwhile, in Yao et al. [24], sparseness of the abundance maps and non-negative matrix factorization (NNMF) are taken into account for spectral unmixing, but once more without high-order components. Recently, in Hong et al. [25], a general multimodal deep learning (MDL) framework was proposed for classification and spatial information modeling in remote sensing datasets. Finally, a new training scheme was suggested in Hong et al. [26] for graph convolutional networks in HSI classification tasks.
Some recent nonlinear unmixing methodologies are reviewed in what follows [27]. As described in Hong et al. [21], nonlinearity in spectral unmixing is an open and pending problem in hyperspectral imaging. In this sense, a bilinear mixture model was considered in Yang et al. [28] to propose an unsupervised scheme that relies on geometric projection for nonlinear unmixing, and NNMF for end-members estimation. In this work, the concepts of virtual nonlinear vertex and nonlinear hyperplanes are adopted to derive the synthesis procedure. Also based on a bilinear mixture model, in Sigurdsson et al. [29], a blind sparse nonlinear hyperspectral unmixing (BSNHU) is suggested that relies on iterative cyclic descent algorithms and the -regularizer to obtain sparse abundances. In [30], an augmented LMM is proposed to consider high-order interactions by incorporating a spectral variability dictionary based on prior knowledge. A different approach has also been suggested in Licciardi et al. [31], where first a nonlinear dimensionality reduction is applied to consider the subsequent estimation of end-members by the N-finder algorithm (N-FINDR) [32]. Finally, the abundances estimation was achieved by constrained quadratic optimization (CQO). In [33], a supervised unmixing was achieved by using continuum removal and natural logarithmic operations to eliminate the nonlinear effects in the HSIs, followed by a linear unmixing strategy. A nonlinear unmixing based on a version of PPNM was proposed in Dixit and Agarwal [34], where a Bayesian framework was used for abundances, nonlinearity factor, and noise variance estimation.
Meanwhile, the state-of-the-art in non-linear unmixing methodologies focused on the MMM is described next. In [19], a supervised scheme was pursued to estimate the abundances and nonlinear interaction terms in the MMM by numerical optimization. In [35], another supervised unmixing approach with a MMM was formulated by Laplacian graph regularizers and alternating direction method of multipliers. This proposal has two distinctive features: the sparsity of the abundances and the use of superpixels to handle large HSIs. To improve noise robustness, a robust unmixing scheme with MMM was proposed in Li et al. [36] by a supervised perspective, where norm-based loss function is adopted for the synthesis problem and also the sparseness of abundances is taken into account. To our knowledge, Wei et al. suggested the only unsupervised strategy for nonlinear unmixing based on the MMM [37]. In this work, a block coordinate descent methodology was used, where the abundances and end-members were updated by a gradient projection method, and the nonlinear interaction level by constrained least-squares.
In this context, the MMM provides a general framework to quantify the nonlinear optical interactions in the scene, but the research work on unsupervised unmixing strategies departing from this model is limited to the work by Wei et al. [37]. Moreover, new BUMs based on the MMM need to take into account precision, robustness, and computational time, as performance requirements. Our proposed BUM follows the estimation framework of the Extended Blind End-member and Abundance Extraction (EBEAE) methodology [20], but is now adapted to the MMM to address these performance issues. The new methodology is called nonlinear EBEAE (NEBEAE). One advantage of the original formulation of EBEAE was the measurements’ normalization to restrict the search space and to limit the variability of the estimation information. Now, we derive a similar condition for NEBEAE and establish a restriction on the abundances. The synthesis optimization objective function has three components: estimation error, entropy of abundances, and similarity of end-members. Due to the non-linear dependence on decision variables, a cyclic coordinate descent algorithm (CCDA) is used to derive an iterative scheme [38]. The implementation details of NEBEAE with respect to initialization, convergence, and measurements reconstruction are discussed in this work. Finally, we evaluate NEBEAE in three different scenarios: 1) synthetic datasets with different types and noise levels; 2) three experimental benchmarks in the remote sensing literature (Cuprite, Urban and Pavia University scene datasets [39]); 3) a biomedical hyperspectral imaging application. In the validation stage, we compared NEBEAE to Heylen and Scheunders [19] with the estimation of end-members by vertex component analysis (VCA), Sigurdsson et al. [29], Wei et al. [37] to show its advantages in terms of precision, robustness and computational time.
The mathematical notation used along this document is described next. The sets of real numbers, real -dimensional vectors, and real matrices are expressed as and , respectively. Scalars and vectors are described by italic and boldface lower-case letters, respectively. Matrices are denoted by upper-case boldface letters, and sets by an upper-case calligraphy font. and refer to an -dimensional vector and a matrix both with unitary entries, respectively, and denotes to the identity matrix. For vectors and with the same dimension, describes its Euclidean norm, the transpose operation, the inner-product, and the Hadamard or element-wise product of both vectors. For a matrix , refers to its Frobenius norm, where denotes the trace operation of , i.e., sum of diagonal elements in , and to the minimum eigenvalue of . For a vector , represents the squared matrix whose diagonal elements are in . For two sets and , the -elements of are described by , denotes its cardinality, and the set difference, i.e., the elements in that do not belong to .
Section snippets
Model description
In our formulation, we assume measurements of a physical positive multidimensional variable, expressed as real positive -dimensional vectors with . Each measurement can be analyzed under various frameworks, but all depart from a LMM for the physical interaction in the observed information:where refers to the th end-member, denotes its contribution to the th measurement (abundance), and represents a noise/uncertainty vector (zero-mean
Validation and discussion
In this section, we validate the BUM under a MMM model with two types of synthetic HSIs, and with experimental ones from the remote sensing literature and a biomedical imaging application. The resulting performance is compared with two state-of-the-art unmixing methods based on MMM: (i) Multilinear Mixing Model for Nonlinear Spectral Unmixing (MMMNSU) (end-members initialized by VCA) [19], and (ii) Unsupervised Nonlinear Spectral Unmixing Based on MMM (UNSUBMMM) [37]. In addition, for the
Conclusions
In this work, we present the mathematical formulation and validation process of a nonlinear version of a new blind nonlinear unmixing method based on a MMM. The estimation process can be separated into three subproblems, which are solved by CCDA and constrained optimization. At each stage, a convex optimization is computed, so the overall convergence is guaranteed, but it is dependent on the initial end-members matrix. During the problem formulation, we included a normalization step in the
CRediT authorship contribution statement
Daniel U. Campos-Delgado: Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Writing – original draft, Writing – review & editing. Inés A. Cruz-Guerrero: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Validation, Visualization, Writing – review & editing. Juan N. Mendoza-Chavarría: Data curation, Software, Validation, Visualization, Writing – review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work was supported by the CONACYT through a Basic Science grant no. 254637. The work of Ines A. Cruz-Guerrero and Juan N. Mendoza-Chavarria was supported by CONACYT through doctoral grants (# 865747)
References (50)
- et al.
First hyperspectral imaging survey of the deep seafloor: high-resolution mapping of manganese nodules
Remote Sens. Environ.
(2018) - et al.
Predicting liquid loss of frozen and thawed cod from hyperspectral imaging
LWT
(2020) - et al.
Chapter 6 - Linear and nonlinear unmixing in hyperspectral imaging
- et al.
Blind spectral unmixing by local maximization of non-Gaussianity
Signal Process.
(2008) - et al.
Non-linear spectral unmixing of hyperspectral data using modified PPNMM
Appl. Comput. Geosci.
(2021) Hyperspectral Imaging
(2019)- et al.
Survey of hyperspectral earth observation applications from space in the sentinel-2 context
Remote Sens.
(2018) - et al.
Label-free hyperspectral imaging and quantification methods for surgical margin assessment of tissue specimens of cancer patients
2017 39th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc.
(2017) - et al.
Spectroscopic techniques for monitoring thermal treatments in fish and other seafood: a review of recent developments and applications
Foods
(2020) - et al.
Medical hyperspectral imaging: a review
J. Biomed. Opt.
(2014)
Chapter 3.6 - Hyperspectral imaging in medical applications
Hyperspectral imaging techniques for the study, conservation and management of rock art
Appl. Sci.
Chapter 3.8 - Hyperspectral imaging for artworks investigation
Hyperspectral Data Processing: Algorithm Design and Analysis
Hyperspectral unmixing overview: geometrical, statistical, and sparse regression-based approaches
IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.
Nonlinear unmixing of hyperspectral images: models and algorithms
IEEE Signal Process. Mag.
Comparative study between a new nonlinear model and common linear model for analysing laboratory simulated-forest hyperspectral data
Int. J. Remote Sens.
Nonlinear unmixing of hyperspectral images using a generalized bilinear model
IEEE Trans. Geosci. Remote Sens.
Supervised nonlinear spectral unmixing using a postnonlinear mixing model for hyperspectral imagery
IEEE Trans. Image Process.
A multilinear mixing model for nonlinear spectral unmixing
IEEE Trans. Geosci. Remote Sens.
Extended blind end-member and abundance extraction for biomedical imaging applications
IEEE Access
Interpretable hyperspectral artificial intelligence: when nonconvex modeling meets hyperspectral remote sensing
IEEE Geosci. Remote Sens. Mag.
Endmember-guided unmixing network (EGU-Net): ageneral deep learning framework for self-supervised hyperspectral unmixing
IEEE Trans. Neural Netw. Learn. Syst.
Sparsity-enhanced convolutional decomposition: anovel tensor-based paradigm for blind hyperspectral unmixing
IEEE Trans. Geosci. Remote Sens.
Nonconvex-sparsity and nonlocal-smoothness-based blind hyperspectral unmixing
IEEE Trans. Image Process.
Cited by (5)
Multi-stream pyramid collaborative network for spectral unmixing
2024, International Journal of Remote SensingHyperspectral imaging benchmark based on machine learning for intraoperative brain tumour detection
2023, npj Precision OncologyHyperspectral imaging for cancer applications
2023, Diagnosis and Treatment of Cancer using Thermal Therapies: Minimal and Non-invasive Techniques