A novel method for multispectral image pansharpening based on high dimensional model representation

Highlights

• A novel pansharpening method based on Adaptive HDMR is proposed.

• Proposed algorithm is a component substitution-based method and is easy to implement.

• Proposed algorithm provides more spectral accuracy than traditional CS-based methods.

• Proposed algorithm outperforms state-of-the-art image pansharpening methods.

Abstract

Pansharpening methods are used to enhance the spatial resolution of a low resolutional multispectral (MS) image by fusing with a high resolutional panchromatic image (PAN). The main difficulty of pansharpening is avoiding spectral distortion while getting a sharpened MS image with high spatial resolution. Intensity-Hue-Saturation (IHS) based methods are applied to transform from color space to IHS and provide equalization of a PAN component with an MS image to eliminate distortion problems. However, most of the modified IHS methods still cause spectral distortion. To overcome this problem, a novel pansharpening method, based on Adaptive High Dimensional Model Representation is proposed in this article. HDMR is a well-known decomposition method for multivariate functions and data sets. The algorithm we propose includes three stages: the first stage is to obtain HDMR components of the MS image using the HDMR decomposition and then to use scaling factors to optimize the effects of the information the components hold. The second stage requires the calculation of some weighting factors in each band to minimize the spectral distortion. Computing the spatial details obtained from the difference between the PAN image and the Adaptive HDMR expansion of the MS image, and adding the difference to the MS image constitutes the third stage. Our proposed algorithm is easy to implement in pansharpening similar to component substitution (CS) based methods, HDMR terms are calculated once and then used adaptively by employing scaling and weighting factors which are determined through a straightforward methodology. The method also provides greater spectral fidelity than the traditional CS based methods as a result of the scaling factors. The proposed method has been tested on different MS images and compared with state-of-the-art pansharpening methods. The results are given both in terms of visual quality and numerical assessments.

Introduction

There have been great improvements in remote sensing imaginery technologies over the years. However, researchers still encounter difficulties due to lack of high spatial resolution of satellite sensors. To overcome the absence of high resolution image data in imaging instruments, most of the remote sensing imaging systems capture two different images of the same geographical location at the same time: multispectral image (MS) and panchromatic image (PAN). An MS image contains rich spectral information with several bands but also low spatial resolution. Contrarily, PAN image has high spatial resolution with only one spectral band. Pansharpening is an important technique to generate high resolutional multispectral image (HMS) by fusing MS and PAN images of the same scene to enhance different applications such as earth monitoring, object/crop classification, target detection, and segmentation (Loncan et al., 2015, Bhandari and Rahul, 2019).

Pansharpening methods vary according to the fusing process and can be classified into four major categories. The first category covers component substitution (CS) based methods. CS based methods replace the components in an MS image with a PAN image, and therefore the MS image is transformed into a different color space. The different details of the PAN image and the bands of the transformed MS image are extracted allowing us to merge these details into a low resolutional MS image. One of the CS based methods relies on Intensity-Hue-Saturation (IHS) transform which is also named as IHS pansharpening (Carper et al., 1990, Al-Wassai et al., 2011). Another method modifies IHS sharpening by finding the gain coefficients of the bands of the MS image adaptively and transferring the edges of the PAN image to the MS image to increase the spectral enhancement (Rahmani et al., 2010). IHS based methods are easy to apply hence they are widely used; however, they may cause spectral distortion. Another well-known CS based technique uses principal component analysis (PCA) which is a linear transformation that utilizes principal components of the MS image and replacing them with the components of the PAN image (Kwarteng & Chavez, 1989), but it may produce distortion caused by local dissimilarities between the components of the transformed MS image and the PAN image (Thomas, Ranchin, Wald, & Chanussot, 2008). Gram-Schmidt (GS) decomposition (Laben & Brower, 2000) and partial replacement adaptive method (PRACS) (Choi, Yu, & Kim, 2011) are also widely used CS based methods. These methods, which are also based on transformation and substitution similar to the abovementioned CS based methods, improve the spatial resolution of MS images but may not preserve spectral information with the MS image.

Multiresolution analysis (MRA) based pansharpening methods are in the second category and mainly use the separation of low and high frequencies through a multiresolution decomposition (Ranchin & Wald, 2000). These methods obtain spatial information from the high-frequency component of the PAN image to construct integration with the upsampled MS image. MRA based methods vary in terms of extraction of spatial details like Laplacian pyramid (LP) (Aiazzi et al., 1999), and wavelet transform methods (Nunez et al., 1999, King and Wang, 2001). MRA based fusion is combined with some other algorithms such as genetic algorithms and particle swarm optimization (Garzelli and Nencini, 2006, Jamal and Faez, 2011) to minimize missing spatial information. Compared to CS based methods, these MRA based methods provide better spectral representation. Nevertheless, they may suffer from distortion in terms of spatial enhancement.

The third category consists of model based methods. Variational optimization models are included in this category (Ballester et al., 2006, Palsson et al., 2013, Duran et al., 2017). Although spectral distortion is minimized by variational methods, these methods suffer from blurring effects and texture contorts. Bayesian statistics based pansharpening methods have been proposed as a model based approach for making good on spectral/spatial trade-off (Fasbender et al., 2008, Zhang et al., 2012, Wang et al., 2018). These methods need some prior knowledge and tuning of hyperparameters which can affect sharpening, and therefore may lead to failures in performance. Most of the model based pansharpening methods rely on linear models to acquire spatial information.

Deep learning (DL) based models have been proposed after the achievements of many different image analysis problems in recent years (Krizhevsky et al., 2017, Shen et al., 2017, Altan and Karasu, 2020). These techniques are a new generation of pansharpening methods and can be considered as the fourth category. They are mostly used to overcome the limitations of linear models (Masi et al., 2016, Wei et al., 2017). Convolutional neural networks (CNN) is a common tool for pansharpening problem (Dong et al., 2015, Scarpa et al., 2018), but the accuracy highly depends on the network architectures and this may induce the generalization problem. To overcome it, further studies used deeper networks but an increasing number of layers led to worsening training accuracy of the network (Wang et al., 2019). To tackle the previously mentioned problems, various deep learning pansharpening methods were proposed which were combined with traditional approaches such as using detail injections before or after the network (Deng et al., 2020, Liu et al., 2020) and Bayesian theory (Guo, Zhuang, & Guo, 2020). However, these methods may suffer from losing high-frequency details because the pre-interpolated MS image is joined with the PAN image as the training data. Non-negative matrix and tensor decomposition based pansharpening methods differ from the abovementioned four pansharpening categories. Matrix factorization based image fusion is an unmixing based solution and served to separate the signals into a dictionary matrix and a weight matrix (Yokoya, Yairi, & Iwasaki, 2011). Tensor decomposition based methods are the extension to matrix decomposition which takes multilinear correlations into account. Tensor decomposition was proposed for pansharpening after the achievements on the super-resolution problem (Li et al., 2018a, Bu et al., 2020). These methods accomplish improvement on the resolution, on the other hand, Tucker decomposition does not guarantee a unique solution as a tensor decomposition algorithm.

Our research aims to develop a novel and efficient CS based pansharpening algorithm for empowering spatial resolution quality of the fused image. To create an effective pansharpening technique, a new High Dimensional Model Representation (HDMR) based method, called Adaptive HDMR, is developed. HDMR is a well known orthogonal data decomposition technique for multivariate functions which was first proposed by Sobol (Sobol, 1993). It is a divide-and-conquer algorithm that allows us to express a multivariate function through functions with fewer variables (Tunga & Demiralp, 2008). This algorithm is also used as a multivariate data partitioning technique in data processing and modeling problems (Karaca and Tunga, 2018, Ozay and Demiralp, 2014, Tunga and Karahoca, 2015, Karcılı and Tunga, 2017). In another study, HDMR is proposed as a powerful decomposing method for image enhancement. The study demonstrates the physical meanings of HDMR components in terms of image processing (Tunga & Koçanoğulları, 2018).

The success of the HDMR method in image decomposition and enhancement led us to use HDMR as a pansharpening method. We investigated if it is possible to transform an MS image with both high spatial and spectral resolution required for the pansharpening problem with the help of HDMR components. The preliminary experiments showed that the classical HDMR expansion causes deformation on the spectral structure. To cope with this problem and develop an efficient pansharpening method, the HDMR components are multiplied by some scaling factors and a new structure which is called Adaptive HDMR, is built. The development of an HDMR based pansharpening method contributes significantly to spatial enhancement while preserving spectrality. When the above categories of pansharpening methods and the experimental results of this study are analyzed, four main contributions of the proposed method can be observed. (1) Adaptive HDMR preserves spectral structure better than the CS based methods. (2) Adaptive HDMR is a spatially superior method unlike most of MRA based methods. (3) Adaptive HDMR has fewer parameters than model based methods which makes it easier to implement. (4) Adaptive HDMR has low model complexity according to the number of used features when compared with the new deep learning based methods. The statistical results in this paper show that the proposed displays an impressive performance on eliminating the addressed gaps in the state-of-the-art pansharpening methods.

The paper is organized as follows. Section 2 explains the mathematical background of HDMR and its components. This section also provides information about the HDMR components of an MS image. Section 3 covers the details of the Adaptive HDMR and the steps of the algorithm proposed as a pansharpening technique. Section 4 includes information about the data used in experiments and performance metrics. Experimental results on different low resolutional MS images of our method and the results on the same MS image of different state-of-the-art pansharpening techniques are also compared in this section. Section 5 contains discussions about the findings as well as the effects of the model parameters on these results. Conclusion and potential future works are given in Section 6.