Elsevier

Knowledge-Based Systems

Volume 149, 1 June 2018, Pages 18-33
Knowledge-Based Systems

Sparse tensor neighbor embedding based pan-sharpening via N-way block pursuit

https://doi.org/10.1016/j.knosys.2018.01.022Get rights and content

Abstract

Most of the available pan-sharpening methods use vector or matrix based detail injection to enhance the resolution of MultiSpectral (MS) image, which may result in unavoidable spectral and spatial distortions. In this paper we explore the intrinsic tensor structure and local sparsity of MS images, to develop a novel Sparse Tensor Neighbor Embedding (STNE) based pan-sharpening method that reduces the distortions in the fused images. First, MS images are formulated as some spectral tensors, and each tensor and its nearest neighbor tensors are assumed to lie in a low-dimensional manifold. Then the tensor is sparsely coded under its neighbor tensors, and a joint sparse coding assumption is cast on bands to develop an N-way Block Pursuit algorithm for solving sparse tensor coefficients. Finally high resolution MS tensor can be obtained by weighting Panchromatic image with the sparse tensor coefficients. Tensors are higher order generalizations of vectors and matrices, and taking advantage of high-order structure of multi-dimensional data can help us understand them. The proposed method first combines a sparse tensor with neighbor embedding, to construct a new high-dimensional sparse tensor embedding for efficient pan-sharpening. Because tensor formulation can exploit the structural correlations in high-dimensional MS data, the proposed method can well preserve spectral correlation among different bands simultaneously and capture the underlying high-order statistical properties of MS image. Some experiments are performed on several real QuickBird and GeoEye datasets, and experimental results show that STNE is superior to its counterparts in reducing spectral and spatial distortions.

Introduction

The High spatial Resolution MultiSpectral (HRMS) image is very desirable in many remote sensing applications, such as environmental monitoring, change detection, object recognition, land-cover classification [1], and so on. However, due to the physical limitation of remote sensors, images captured by some satellites such as QuickBird and GeoEye, are composed of Low spatial Resolution MultiSpectral (LRMS) image (e.g., 2–4 m), and High spatial resolution Panchromatic (HPAN) image (e.g., 0.5–1 m). In order to improve the spatial resolution of the LRMS image, pan-sharpening technology is developed that fuses LRMS image with a corresponding HPAN image [2].

Recently many pan-sharpening methods have been proposed, which can be mainly categorized into three groups: (1) Projection Substitution (PS) -, (2) Amélioration de la Résolution Spatiale par Injection de Structures(ARSIS) -, and (3) Modeling and Recovering (MR)-based methods [3]. PS methods are among the most popular pan-sharpening methods, whose basic idea is to project the LRMS image into a new feature space by some transformation(s), and then substitute some transformed component of LRMS image by HPAN image. Although these methods are easy to implement and have high fidelity in rendering spatial details, the adopted projections, such as Intensity-Hue-Saturation (IHS) [4], and Principal Component Analysis (PCA) [5], may distort the spectrums and thus change the color of the fused HRMS image. Therefore, improved methods have been proposed. For example, Xu presented a data fitting method to extract the substitution component [6], where an image matching model is introduced to reduce the distortions in the pan-sharpening. Kang introduced an image matching model to reduce the distortions in the pan-sharpening [7].

ARSIS methods aim to inject the spatial details extracted from the HPAN image [8] into the LRMS image. Although some MultiResolution Analysis (MRA) tools including Wavelet, Curvelet, Contourlet, Ridgelet, support value transformation [9], have proved to be efficient in extracting high frequency components of HPAN image, distortions may arise from the coarse extraction and brute injection of details from HPAN to the LRMS image. Consequently, a mismatch between details of the HPAN and LRMS image will appear, thus degrading the spatial and spectral resolution of the fused HRMS image. Recently Restaino R investigated a non-linear MRA scheme, implemented with morphological pyramids instead of conventional linear MRA approach [10]. A two-step (coarse and refined steps) approach is proposed [11], where a novel multiscale decomposition based on a normalized non-local means filter is developed to extract spatial details. It proved to be able to preserve the self-similarity structure effectively, and extract accurate details for pan-sharpening. These methods can improve the image resolution by adding the details of PAN image into the LRMS image, however, if spatial details extracted from PAN image do not match that of LRMS image, the aliasing and local dissimilarities will appear in the fused image.

Different from PS- and ARSIS-based methods, the third category, MR-based methods benefit from the development of Compressed Sensing (CS) and Sparse Representation (SR) [12], [13], [14]. Their core idea is to model the relationship among HRMS, LRMS, HPAN images and formulate a recovery optimization problem for pan-sharpening [15]. By reducing the pan-sharpening to a recovery task and avoiding a direct injection, MR-based methods can avoid the potential mismatch brought by a direct injection, so recently they are receiving increasing interests. Considering the high computational complexity caused by a large dictionary, Li proposed a dictionary learning approach for adaptive recovery of HRMS image [16]. Furthermore, a novel strategy was designed to construct a dictionary for unknown HRMS image without training samples. Then Zhu proposed a SparseFI [17] method that explores the similarity between sparse coefficients of HRMS and LRMS image patches. Some improved dictionary learning methods were also investigated [18]. Palsson used a Total Variation (TV) regularizer with the observational models [19]. Based on the method of SparseFI, another method named J-SparseFI method was proposed [22] which exploits the structure correlations between multispectral channels.

Although MR-based methods have proved to be efficient in pan-sharpening, there are still some issues to be addressed:

  • (1)

    Ignoring the band correlation and resulting in color distortion.

Most of the available MR-based pan-sharpening methods represent images as vectors or matrices, and deal with different LRMS bands separately. However, bands of multi-spectral image obtained across the spectrum are often highly correlated and exhibit some intrinsic data structure, which is to be ignored in most of the available MR based pan-sharpening methods. After the injection, proportions of bands of the LRMS image are changed and result in remarkable color-distortion.

  • (2)

    Ignoring high-order statistical properties of data.

The most commonly used statistical properties of LRMS image are usually measured in two-dimensional image, like histogram statistics, block self-similarity, 2-D gradients, low-rank property, and so on. However, there are some high-order statistical characteristics in three dimensional spectral data, such as three order gradients, tensor sparsity, and so on. Although there is an evident correlation in each band which has been explored by some of the works, high-order statistical properties of data are always ignored in the available pan-sharpening methods.

It is well known that spectral images are intrinsically tensors, which are the higher order generalizations of vectors and matrices. So directly processing multispectral cubes as tensors can utilize information along all dimensions, and take full advantage of the high-order structure to better understand data without destroying the data structure. As an effective way to support higher order data processing, tensor analysis is attracting increasing interests [36], [37], [38], [39]. Recently, tensor based higher-dimensional data processing approaches have been developed, to overcome the disadvantage of ignoring the inherent structure of data. Meng presented a Tucker decomposition method based on Noise Power Ratio(NPR) analysis for 3-D hyperspectral image classification [20]. Hao proposed a novel linear Support Higher-order Tensor Machine (SHTM) which integrates the merits of linear C-Support Vector Machine (C-SVM) and tensor rank-one decomposition for more accurate classification [21]. There are also several works in the field of image fusion. For example, Karahan formulated data fusion as a coupled matrix and tensor factorization problem and discussed its extension to a structure-revealing data fusion model [22]. Acar formulated data fusion as a coupled matrix and tensor factorization problem and discussed its extension to a structure-revealing data fusion model [23]. This new data fusion model can identify shared and unshared factors in order to jointly analyze heterogeneous datasets, by formulating source images as tensors. High-Order Singular Value Decomposition (HOSVD) based image fusion algorithm was proposed [24], which is used for fusing multi-focus images. The experimental results show that the proposed algorithm is an alternative image fusion approach.

Tensors are higher order generalizations of vectors and matrices, and taking advantage of high-order structure of multi-dimensional data can help us understand them. Motivated by the success of precedent applications of tensors in image processing, a new MR-based method is proposed in this paper, by establishing a sparse manifold embedding framework for pansharpening and extending it to the tensor case [25]. The correlations among spectral bands are taken into consideration, to advance a novel Sparse Tensor Neighbor Embedding (STNE) method, by formulating the multispectral image as high dimensional tensors and casting a joint sparse coding assumption on the spectral tensors. First the LRMS image is partitioned into some image tensors, and each tensor and its nearest neighbor tensors in a local region are assumed to lie in a low-dimensional manifold. Then an assumption that tensors share the similar manifold structure in a local region, is cast on LRMS and HRMS image, to develop a sparse tensor linear neighbor embedding algorithm. Each LRMS tensor is sparsely coded under its neighbors and the coding coefficients in each band are jointly solved by an N-way Block Pursuit (NBP) algorithm. In NBP, the non-zero coefficients are restricted to be located within a sub-tensor (block), and is derived by calculating a block-sparse representation of a tensor with respect to a Kronecker basis. Finally the HRMS image can be obtained by a weighted combination of HPAN image with the sparse coding coefficients.

Compared with the available pan-sharpening methods, the contribution of our work is twofold:

  • (1)

    It can locate the few neighbors that are most related to formulate the manifold in a local region, which is helpful to find more accurate manifold structure for image recovery.

  • (2)

    The tensor formulation can exploit the structural correlations in high-dimensional MS data. Consequently the proposed method can well preserve spectral correlation among different bands simultaneously and capture the underlying high-order statistical properties of MS image, therefore resulting in fewer distortions in the fused images.

It is demonstrated through simulation examples that our proposed method can obtain fused images with less spectral and spatial distortions, for exploring the multi-dimensional structure of image cube and dealing with whole data in pan-sharpening. Moreover, exploring the Kronecker [26] structure together with the block-sparsity property in NBP can also achieve faster and more precise sparse representations of tensors compared with the vector based Orthogonal Matching Pursuit (OMP) algorithm [27]. Some experiments are performed on several real Quickbird and GeoEye datasets, and experimental results show that STNE is superior to its counterparts in reducing spectral and spatial distortions.

The rest of the paper is organized as follows: In Section 2, a detailed description of our proposed sparse tensor neighbor embedding algorithm is given. In Section 3, the experimental results are demonstrated and discussed. Finally, conclusions are drawn in Section 4.

Section snippets

Sparse tensor neighbor embedding

In order to avoid the vectorized operation on spectral data and making avail of the correlations among spectral bands, we formulate multispectral images as high dimensional tensors and exploit sparse representation of tensors to develop a new model, sparse tensor neighbor embedding. By casting a joint sparse coding assumption on the spectral tensors, we solve sparse coefficient of low-resolution image block via an N-way Block OMP algorithm, which is mathematically formulated in detail.

Experimental datasets

In the experiments, we test the proposed method on several datasets acquired by QuickBird and Geoeye satellites. Before the fusion, the LRMS and HPAN images are coregistered. The characteristics of datasets are described as follows: The QuickBird satellite can provide 2.44 m LRMS and 0.61 m resolution HPAN images. GeoEye satellite can provide 2.0 m spatial resolution LRMS and 0.5 m spatial resolution HPAN images. The LRMS image has four spectral bands: Blue (B), Green (G), Red (R) and Near

Conclusions

In this paper, we propose a novel Sparse Tensor Neighbor Embedding (STNE) based pan-sharpening method, by formulating the multispectral image as high dimensional tensors and casting a joint sparse coding assumption on the spectral cube. It can locate the most related few neighbors to formulate the manifold in a local region, which is helpful to find more accurate manifold structure for image recovery. Moreover, the tensor formulation can exploit the structural correlations in high-dimensional

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Nos. 61771380, U1730109, 91438103, 61771376, 61703328, 91438201, U1701267, 61703328); the Equipment pre-research project of the 13th Five-Years Plan (Nos. 6140137050206, 414120101026, 6140312010103, 6141A020223, 6141B06160301, 6141B07090102), the Major Research Plan in Shaanxi Province of China (Nos. 2017ZDXM-GY-103, 017ZDCXL-GY-03-02), the Foundation of the State Key Laboratory of CEMEE (Nos. 2017K0202B, 2018K0101B),

References (39)

  • M. Choi

    A new intensity-hue-saturation fusion approach to image fusion with a tradeoff parameter

    IEEE Trans. Geosci. Remote Sens.

    (2006)
  • C. Psjr et al.

    Comparison of three different methods to merge multiresolution and multispectral data: landsat TM and SPOT panchromatic

    Photogramm. Eng. Remote Sens.

    (1991)
  • Q. Xu et al.

    High-fidelity component substitution pansharpening by the fitting of substitution data

    IEEE Trans. Geosci. Remote Sens.

    (2014)
  • X. Kang et al.

    Pansharpening with matting model

    IEEE Trans. Geosci. Remote Sens.

    (2014)
  • R. Restaino et al.

    Fusion of multispectral and panchromatic images based on morphological operators

    IEEE Trans. Image Process.

    (2016)
  • M. Elad et al.

    On the role of sparse and redundant representations in image processing

    Proc. IEEE

    (2010)
  • D.L. Donoho

    Compressed sensing

    IEEE Trans. Inf. Theory

    (2006)
  • J. Mairal et al.

    Sparse representation for color image restoration

    IEEE Trans. Image Process.

    (2008)
  • S. Li et al.

    A new pan-sharpening method using a compressed sensing technique

    IEEE Trans. Geosci. Remote Sens.

    (2011)
  • Cited by (0)

    View full text