The PAN and MS image fusion algorithm based on adaptive guided filtering and gradient information regulation
Introduction
Multi-source image fusion technology refers to the process of merging two or more images from different sensors into one image to obtain more complete information than any single remote sensing image, and it has great significance to the image processing tasks in computer vision area [12], [34]. As an important branch of multi-source image fusion technology, remote sensing image fusion technology aims to obtain more information from remote sensing images with relevant feature information by complementing the information under certain constraints. To make up for the limitations of hardware and environment, it is impossible to obtain remote sensing images with both high spectral resolution and high spatial resolution. Remote sensing image fusion is divided into three levels: pixel level fusion, feature level fusion and decision level fusion.
Traditional pixel-based fusion technology can take into account the characteristics of the remote sensing images which obtained by different sensors and focus on the statistical analysis of the pixel information in order to obtain fusion images. According to the scope of the fusion operation, the method can be divided into two methods: fusion based on the spatial domain and fusion based on the transform domain [29]. The fusion method based on the spatial domain completes the fusion process by directly using the spatial domain pixels of the images, such as the Intensity-Hue-Saturation (IHS) transformation, principal component analysis (PCA), Gram Schmidt (GS) transformation and Brovey transformation. These methods usually map MS images to a feature space, by using PAN image replace its highest correlation component to improve the spatial resolution of the fused image, these types of methods are generally simple algorithms and have fast calculation speed. However, the fused image is usually degraded, such as by contrast reductions, and has a poor fusion effect. Later, the FIHS algorithm was proposed, although the quality of the fused image was improved, the application range was limited due to the limitations of sensor types. The fusion method based on transform domain transforms the image from the spatial domain to the transform domain with multi-scale analysis tools and sparse representation. Then, it uses the characteristics of the high frequency subband and the low frequency subband in the transform domain to determine the corresponding fusion rules. The processing of this method is similar to the process of the human visual system (HVS) when recognize objects. The background information of the image is well preserved in the low frequency sub-band, and the textural edge information of the image is preserved in the high frequency sub-band. Then, it effectively retains the background information and spectral information of the spectral image, and the textural details of the high-resolution image are well integrated into it by processing the low frequency and high frequency sub-bands. Therefore, the fused images have great quality [14], [7], [20], [3]. Traditional multi-scale methods were used in remote sensing image fusion area, such as contourlet transform, shearlet transform, wavelet transform, etc. Subsequently, NSCT and NSST transformation were proposed, and the quality of the fused image has been improved greatly.
In recent years, with the development of remote sensing technology, the textural edge information in high-resolution images is becoming more complex, and it is also affected by the acquisition environment. Remote sensing images are usually polluted by patches, noise and shadows during the acquisition and transmission processes, which affect the final fusion quality in degrees. However, the requirements for the accuracy of the feature extraction, classification and target recognition of the ground objects have become higher in applications, such as land cover and land use mapping. In general, better descriptions of ground objects should be obtained by using the feature classification of fused images and increasing the spatial resolution of remote sensing images [23]. All of these lead to the research of feature level and decision level fusion of remote sensing images becoming a research hotspot [4], [17], [27].
Feature level fusion technique uses a set of image pixels to form a continuous region, from which one can extract different features or classify features, which may be pixel intensities, edges or textural features in different types of images from the same geographic location. It can be extracted using methods such as the chromaticity information transformation, guided filtering. Then, the feature information is fused to improve the spatial information. For example, in remote sensing image fusion based on feature sets, the literature [24] extracts textural features, shape features and spectral information from hyperspectral images and fuses them into feature sets by adopted stacking as the fusion scheme. Further, using the support vector machine (SVM) classifier with a cubic polynomial kernel can improve the classification accuracy. Song et al. [30] proposed a learning-based image fusion method, which combined the band width and spectral characteristics of the Landsat Thematic Mapper (TM)/Enhanced Thematic Mapper Plus (ETM+) and the spatial resolution of the SPOT5 (Système Pour l 'Observation de la Terre 5) to obtain better fusion results than traditional methods. However, this method need to be further improved with respect to the spatial details of the TM. Bai et al. [2] proposed a softmax regression-based feature fusion method that learns distinct weights for different features. This method uses the shape, spectral, and textural features to classify objects and gets good results. Liu et al. [15] proposed a two-stream fusion network (TFNet) to address the problem of pansharpening. Unlike previous CNN (Convolutional Neural Networks) method that consider pansharpening as a super resolution problem and perform pansharpening at the pixel level, the proposed TFNet fuses the PAN and MS images at the feature level and reconstructs the pansharpened image from the fused features.
In decision level image fusion, Luo et al. [19] proposed decision-based fusion for the pansharpening of remote sensing images. This method merged the à trous wavelet pansharpening [25] and Laplacian pyramid adaptive pansharpening [1] methods in order to take advantage of both methods by locally selecting the best one. However, due to the segmentation errors that are caused by the level line methods, the methods cannot always provide the best fusion results. In literature [31], a decision level fusion using multiview very-high resolution (VHR) imagery is used in an object recognition strategy. In order to refine the classification results, the classified objects of all views are fused together at the decision level fusion based on the scenic contextual information. Tuia et al. [33] proposed a probabilistic discriminative graphical model relying on conditional random fields for the fusion of the land-cover and land-use classification results from very high-resolution remote sensing images. The system integrates the pixel-based and region-based strategies with a multiscale approach and it is able to find agreement between the probabilistic decisions with multiple spatial supports.
Although the remote sensing image fusion technology based on the feature level and decision level is addressed here, the fusion accuracy and fusion effect of this method still need to be improved. Many problems still need to be solved, such as the selection and assessment of valid features, the information loss in the feature extraction process, the reasonable selection of the injected features, the simplification of the fusion rules and the evaluation methods of the universal fusion results.
In this paper, a multispectral (MS) and panchromatic (PAN) image fusion algorithm based on gradient domain decision map adaptive guided filtering is proposed. The innovations are mainly reflected in the following aspects. 1) An adaptive guided filter (AGIF) based on the gray-level co-occurrence matrix (GLCM) is proposed in order to extract the textural and detailed feature information of PAN images, which ensures the sufficiency of the information extraction. 2) A decision map based on the gradient domain is proposed to supervise the injection position of the textural and detail feature information and ensure the spectral quality of the fused MS image. 3) A weighted matrix constraint scheme based on the gradient entropy measure was proposed to regulate the injection intensity of the PAN feature information, avoid the interference of noise, patches and other information, and enhance the spatial resolution of the fused images. 4)A new MS and PAN fusion scheme based on gradient domain decision template adaptive guided filtering is proposed. The proposed algorithm has the advantages of the textural features of high-resolution PAN images extracted sufficiently, adaptively adjusting the position and intensity when the feature information is injected into a MS image, and having clear textures in the fused image. The advantage of the method based on detail injection into MS image is that on the premise of effectively maintaining the spectral information quality, the spatial resolution of the fusion image is improved. The injection method is not restricted by the relative spectral influence relation between bands, and the fusion can be carried out by using single band of multispectral image and panchromatic image, it is helpful to improve the universality of the fusion method. A large number of simulation experiments verify the effectiveness of the proposed method.
Section snippets
Analysis of guided filtering
Guided image filtering (GIF) is a smoothing filter for edge preservation that was proposed by He et al. [10]. The filter completes the filtering process through a guide image, and the guide image can either be a single image itself or an input texture map. When the guide image is the latter, GIF can well maintain the edge. Because GIF can avoid ringing effects, it is widely used in image processing fields such as feature extraction, image fusion and super-resolution reconstruction.
Let the input
The proposed adaptive guided filtering (AGIF) based on the GLCM
Based on the above analysis of the traditional GIF model and the discussion of the GLCM characteristics for MS and PAN textural analysis, this section proposes an adaptive guided filtering model for remote sensing images based on the GLCM.
Let be the current pixel to be processed, is the window centered on and is the number of pixels of the guide image; , and correspond to the correlation, entropy and angular second moment of the GLCM in the current window, respectively. The
Algorithm flow chart
The flow chart of the algorithm that is proposed in this paper is shown in Fig. 8.
Algorithm flow steps
Let the MS image that is to be fused after registration be and the PAN image that is to be fused be . The specific implementation process of the fusion between and is as follows:
Step 1. Performing IHS transformation on MS image, and extract its intensity component I and retain its spectral and chrominance components.
Step 2. Let PAN be the input matrix. Calculate the edge guidance function by
Experiment and analysis
In order to verify the effectiveness of the MS and PAN image fusion algorithm that is proposed in this paper, the experiment part is divided into two groups: the degraded data experiments and the real experiments. We sampled 100 sets of test images of different sizes from the Quickbird02 and Worldview02 data sets for testing and the example images are shown in Fig. 9, the images in the data set to be tested cover a variety of scenes, including vegetation areas (such as forests and farmland),
Conclusion
This paper proposes an adaptive guided filtering algorithm based on the gradient domain decision map for fusing MS and PAN images. According to the textural features of PAN images, an AGIF model based on the GLCM is proposed in order to extract the textural features from PAN images. In order to maintain the spectral quality of the fusion image and inject the most possible textural information, this paper sets the injection location of the feature information by determining the decision map from
CRediT authorship contribution statement
Xianghai Wang: Conceptualization, Methodology, Funding acquisition. Shifu Bai: Methodology, Validation, Writing - original draft. Zhi Li: Formal analysis, Writing - original draft. Yuanqi Sui: Writing - review & editing. Jingzhe Tao: Investigation, Methodology, Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This study was funded by the National Natural Science Foundation of China (Grant Nos. 41671439, 41971388) and Innovation Team Support Program of Liaoning Higher Education Department (Grant No. LT2017013).
References (36)
A review of remote sensing image fusion methods
Information Fusion.
(2016)- et al.
Multifocus image fusion by combining with mixed-order structure tensors and multiscale neighbourhood
Inf. Sci.
(2016) - et al.
Fusing synergistic information from multi-sensor images: an overview from implementation to performance assessment
Information Fusion.
(2018) Remote sensing for urban planning and management: the useof window-independent context segmentation to extract urban features in Stockholm
Comput. Environ. Urban Syst.
(2015)- et al.
Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis
IEEE Trans. Geosci. Remote Sens.
(2002) - et al.
Object classification via feature fusion based marginalized kernels
IEEE Geosci. Remote Sens. Lett.
(2015) - et al.
Remote sensing image fusion using statistical univariate finite mixture model in shearlet domain
- et al.
A survey of multisensor fusion techniques, architectures and methodologies
- et al.
Comparing cooccurrence probabilities and Markov random fields for texture analysis of SAR sea ice imagery
IEEE Trans. Geosci. Remote Sensing.
(2004) - et al.
Multiscale principal components analysis for image local orientation estimation